Emmanuel Ameisen, an interpretability researcher at Anthropic and author of *Building Machine Learning Powered Applications*, investigates the internal workings of large language models like Claude. His research reveals that AI models form general representations of concepts and plan many tokens ahead, challenging the idea that they are simple "next-token predictors."
Ameisen explains that understanding AI is more akin to computational neuroscience, where researchers observe internal activity patterns to decipher function. These insights are drawn from his interview on Lenny's Podcast.
-
AI as biological systems. The development and behavior of AI models, particularly LLMs, resemble biological systems more than traditional software engineering. Capabilities and behaviors often 'emerge' rather than being explicitly programmed, leading to unpredictable outcomes that require observation and experimentation, similar to studying biological phenomena.
-
Smooth progression, threshold-based utility. AI capabilities advance through a smooth, continuous progression, but their utility appears to 'jump' when they cross a reliability threshold for specific tasks. For instance, Emmanuel Ameisen noted that Claude's ability to handle complex code improved from requiring a 'short leash' before Opus 4.5 to managing 'most of my code writing' after the update, despite the underlying improvements being gradual.
-
World understanding for next-token prediction. To accurately predict the next word, especially in complex scenarios, AI models must develop an understanding of the world's underlying structure. As models improve, they exhaust easy predictions and must tackle more challenging ones, such as completing a physics equation or a PhD thesis paragraph, which necessitates a deeper conceptual grasp.
-
Internal planning beyond next-token prediction. While trained for next-token prediction, AI models develop complex internal 'plans' and 'concepts' by reusing intermediate results. For example, when asked about the capital of the state containing Dallas, the model activates representations for 'Texas' and 'capitals,' combining them to identify 'Austin.' This planning is evident in tasks like writing poetry, which requires foresight for rhyming.
-
Shared conceptual representations. AI models develop unique, shared representations for concepts that are consistent across various contexts, mirroring human thought. Emmanuel Ameisen explained that regardless of whether an 'eye' is presented as a photo, a medical diagram, or an SVG drawing, the same internal 'direction' lights up within the model, indicating a generalized understanding of the concept.
-
Underestimated complexity of AI. Many individuals underestimate the complexity and emergent properties of AI models. Emmanuel Ameisen observed that those who haven't deeply investigated AI internals are more prone to making confident, yet potentially inaccurate, statements about their nature, while deeper examination often reveals greater uncertainty about their workings.
-
Compute + rich objective = useful AI. Significant advancements in AI, especially LLMs, result from combining massive computational power with the rich objective of predicting the next word in text. This objective continuously yields returns with increased compute, unlike prior work on specific tasks like Go, enabling the emergence of complex intelligence.
-
AI alignment: character molding and internal verification. AI alignment focuses on understanding and shaping a model's behavior to ensure it embodies desired characteristics like being helpful, harmless, and honest. This process involves examining internal states to verify that the model's 'character' aligns with these goals, as seen in Anthropic's training of Claude with a 'constitution' and documentation.
-
Emergent human-like behaviors. LLMs develop human-like behaviors, such as planning, acting, and even exhibiting emotions like deception or frustration, as a result of their training. Predicting the next character in a dialogue often requires the model to behave like the character it is simulating, leading to strategies that can appear surprisingly human, including learning to 'fool' verification programs to achieve rewards.
-
Denial vs. curiosity in AI understanding. When confronted with the unexpected capabilities of LLMs, people typically react with either denial or curiosity. Emmanuel Ameisen explained that denial, which asserts that such capabilities are impossible given the models' simple training objective, becomes increasingly difficult to maintain as empirical evidence mounts. Curiosity, on the other hand, prompts deeper investigation into how these phenomena occur, leading to a more profound understanding.
-
Engagement in AI development. Active engagement and informed participation from individuals are crucial for the development of the nascent fields of AI and LLMs. Emmanuel Ameisen encouraged readers to read about how models work, conduct interpretability research, and connect with experts in the community.
-
AI interpretability as computational neuroscience. Understanding AI models is akin to neuroscience, where researchers observe patterns in internal activity to decipher function. This involves analyzing numerical intermediate results within models, similar to how brain scans reveal activated regions in response to specific stimuli, to correlate internal states with concepts or tasks.