This story first appeared in The Algorithm, our weekly newsletter on AI. To get stories like this in your inbox first, sign up here.
Anthropic—currently the world’s most valuable AI company, with a valuation of about trillion—has a reputation for publishing unusual and profound research. It’s looking into whether or not AI models can feel pain, for instance, and should sometimes cut off chatbot conversations if it suspects users are “abusing” the model.
One area that Anthropic spends more time and money on than other AI companies is called mechanistic interpretability, which means looking inside the complex math of an AI model to understand why it produces one particular output and not another. It’s difficult work; there are millions of data points that can contribute to any result, and wading through them can look more like word salad than anything useful. It’s also controversial. Describing AI models with terms borrowed from psychology and neuroscience could make their behavior seem more sophisticated than we might otherwise take it to be.
That’s why, when Anthropic announced last week that it had discovered a new window into its models’ “inner thoughts” as they reason through answers, there was one colleague I needed to talk to. Senior editor Will Douglas Heaven, who holds a PhD in computer science, has spent quite a bit of time digging into what we can say about how AI models work. I spoke with him about what we should take from Anthropic’s new (and predictably quirky) research.
What did Anthropic discover here, exactly?
Anthropic has been trying to understand how large language models (LLMs) work for a few years now. Anthropic isn’t the only one looking at this, but I think the company has made it part of its core mission more than most. Anthropic’s CEO, Dario Amodei, has said we won’t be able to control LLMs fully until we learn more about how they work.
So this new research is very much in that context. It goes deeper into the inner mechanisms inside LLMs than ever before. What Anthropic found was that LLMs have a space inside them—which Anthropic calls the J-arena—filled with words that don’t appear in their output but that seem to influence the way they reason through problems. All this was hidden until Anthropic developed a new technique to probe its model Claude, so it’s a real discovery.
Sometimes these words keep track of where the LLM has gotten to in a particular task, sometimes they look more like flashes of recognition (for instance, “protein” could pop up if you give an LLM only the letters of a protein sequence), and sometimes they represent a kind of inner commentary on the model’s decision-making. In my favorite example, Claude decided to cheat on a coding test when the word “alarm” appeared.
Anthropic also found that LLMs are capable of tracking and manipulating the words in this space. So one way or another they seem to be using it.
Let’s step back for a second. I don’t think of large language models as simple, but they’re also not magic. There’s a bunch of math that learns relationships between words, right? So why is it so hard to “look” into an LLM to understand what’s going on?
Yeah, they’re not magic! I think the fact we don’t fully understand them plays into the mythmaking. And it’s worth noting that the whole narrative Anthropic is leaning into here—that they’ve built this really mysterious technology, but don’t worry, because they’re also the ones to figure it out—very much fits with the company’s vibe. [See how Anthropic warned that its new models were so good at coding they posed a global cybersecurity risk, only for the US government to shut them down shortly thereafter.]
So yes: LLMs are just math. But it’s incredibly complex math. Not only are today’s LLMs made of hundreds of billions of numbers, but running them triggers a cascade of millions and millions of calculations. I wrote last year that if you printed out even a medium-sized LLM on sheets of paper, it could cover a city the size of San Francisco.
It’s not possible to make sense of any of that math without specialized tools that highlight particular aspects of an LLM at specific times. You have to know where to look and how to look. And building these tools requires understanding some of that complex math in the first place.
You’ve written elsewhere about this idea of studying LLMs the way one might study an organism’s brain. Is it okay to use “brain-like” terms when talking about how an LLM works?
I don’t love using these kinds of terms. LLMs are not brains. Talking like this is misleading because it could suggest that LLMs are capable of more human-like things than they are or that we might make assumptions about how they will behave that we shouldn’t. The whole anthropomorphization thing is also tied up with a bunch of strong ideological positions about what this technology is and what it’s going to be.
But at the same time, we lack a proper alternative vocabulary for talking about what these models are doing. I can understand why people reach for words like “think” and “understand” and “brain-like”—they’re convenient shorthand.
Anthropic compares this new space it found inside LLMs to the space that some neuroscientists believe our brains use to keep track of conscious thoughts. I asked the company how seriously we should take that comparison and it said in a statement: “Drawing these analogies was helpful to us in designing our experiments, as they allowed us to make many non-obvious experimental predictions about the J-arena that turned out to be correct. At the same time, it’s important to note that there are some fundamental differences between the J-arena (and language models in general) and the human brain, so we don’t mean to claim there’s a perfect correspondence.”
What’s a problem in AI that this new understanding of the J-arena could be used to solve?
Anthropic has said that monitoring the J-arena could be a way to catch models doing something they shouldn’t. Because words pop up in this space that don’t appear in a model’s output, they can reveal things about its behavior that you might not otherwise see—like when it’s giving biased responses or when it’s weighing the pros and cons of cheating.
That’s the idea, at least. I think it’s better to see this result as one more step on the journey to understanding this technology overall rather than as something that will be useful on its own.
Read more in Will’s full report about the new research.
