Juan Miguel Santos, Dr. Christopher Bautista, Vijaykumar, 2026. "Mechanistic Interpretability of Transformer Circuits From Features to Algorithms", Journal of Frontiers in Artificial Intelligence and Machine Learning 1(1): 30-46.

Transformers have reshaped the landscape of artificial intelligence with their revolutionary success, outperforming previous state-of-the-art models in fields such as natural language processing, computer vision, multimodal reasoning (multimodal transformer), scientific discovery (scientific transformers) and autonomous decision making. LLMs and foundation models based on transformers, have shown unmatched ability to understand context, generate cohesive text outputs, solve complex problems and achieve tasks that were believed only capable of human intelligence. Nevertheless, very little is known about the internal computation that underpins transformer behavior. Transformers have become the foundation of nearly all current natural language processing tasks (Khan et al., 2021; Lewis et al., 2019), but most modern transformer models are extremely advanced black-boxes with billions or trillions of parameters and hard to understand how they learn, what representations that transfer across downstream tasks and what is required for complex reasoning. This opacity has presented a major obstacle for researchers hoping to make reliable, safe, aligned and trustworthy models. With AI systems increasingly affecting key areas of society, a better understanding of their internal workings has become one of the most pressing research priorities in modern machine learning.
A promising scientific approach to this problem is mechanistic interpretability, which tries at their best to dissect neural networks as hierarchical compositions of local computations and identify computational structures that give rise to model behavior. In contrast to traditional interpretability, which measures various statistical approximations or feature importance measures to explain model outputs, mechanistic interpretability attempts to discover what actual algorithms, circuits, and representations neural networks implement. The primary goal is to localize transformer models using layer-wise information flow, interactions between neurons and attention heads, emergence of complicated parts from distributed representations. It considers neural networks to be computational systems which are analyzable, decomposable and comprehensible in an engineered software sense or biological analogue to a neural circuit.
Much of mechanistic interpretability research centers around understanding the features that emerge in transformer models. Features are important patterns, concepts and abstractions learned in training which are the basic units of cognition of the internal model. Compared with conventional machine learning systems, where representations can be often explicit and interpretable, transformer models are known for developing very disperse and entangled representations. Also present in these models are phenomena such as superposition, where multiple concepts are encoded within a single component of the neural substructure making unravelling model behavior all the more difficult. Newer studies have revealed that transformers tend to structure information in more nuanced representation spaces with individual and take into account groups of neurons as well as attention heads all executing a form of calculated functions. These features are key to discovering how transformers understand language, reason about concepts, and generate outputs.

1. 1. Ashish Vaswani, et al. (2017). Attention Is All You Need. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS).
2. 2. Chris Olah, et al. (2020). Zoom In: An Introduction to Circuits. Distill.
3. 3. Anthropic. (2021). A Mathematical Framework for Transformer Circuits.
4. 4. Nelson Elhage, et al. (2021). Transformer Circuits. Anthropic Research.
5. 5. Tom Brown, et al. (2020). Language Models are Few-Shot Learners. NeurIPS.
6. 6. Jacob Devlin, et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL.
7. 7. Alec Radford, et al. (2019). Language Models are Unsupervised Multitask Learners. OpenAI.
8. 8. Dario Amodei, et al. (2016). Concrete Problems in AI Safety. arXiv.
9. 9. Catherine Olsson, et al. (2022). In-Context Learning and Induction Heads. Anthropic.
10. 10. Samuel Marks, et al. (2022). The Geometry of Truth. arXiv.
11. 11. Arthur Conmy, et al. (2023). Towards Automated Circuit Discovery for Mechanistic Interpretability. NeurIPS Workshop.
12. 12. David Bau, et al. (2019). Identifying and Controlling Important Neurons in Neural Machine Translation. ICLR.
13. 13. Jared Kaplan, et al. (2020). Scaling Laws for Neural Language Models. arXiv.
14. 14. Rishi Bommasani, et al. (2021). On the Opportunities and Risks of Foundation Models. Stanford Center for Research on Foundation Models.
15. 15. Shunyu Yao, et al. (2023). Tree of Thoughts: Deliberate Problem Solving with Large Language Models. NeurIPS.
16. 16. Kenneth Li, et al. (2023). Measuring and Controlling Knowledge in Language Models. ICLR.
17. 17. Leo Gao, et al. (2021). Interpretability in Large Language Models. Anthropic Research.
18. 18. Tom McGrath, et al. (2023). Acdc: Automated Circuit Discovery in Transformers. arXiv.
19. 19. Neel Nanda. (2023). TransformerLens: A Library for Mechanistic Interpretability. GitHub Documentation.
20. 20. Rylan Schaeffer, et al. (2024). A Survey of Mechanistic Interpretability. arXiv.
21. 21. Sarah Wiegreffe and Yuval Pinter. (2019). Attention is not not Explanation. EMNLP.
22. 22. Been Kim, et al. (2018). Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors. ICML.
23. 23. Amirata Ghorbani, et al. (2019). Towards Automatic Concept-Based Explanations. NeurIPS.
24. 24. Finale Doshi-Velez and Been Kim. (2017). Towards A Rigorous Science of Interpretable Machine Learning. arXiv.
25. 25. Ian Goodfellow, et al. (2016). Deep Learning. MIT Press.
26. 26. Yoshua Bengio, et al. (2021). Deep Learning for AI. Communications of the ACM.
27. 27. Richard Sutton and Andrew Barto. (2018). Reinforcement Learning: An Introduction (2nd Edition).
28. 28. Percy Liang, et al. (2022). Holistic Evaluation of Language Models. Transactions on Machine Learning Research.
29. 29. John Hewitt and Christopher Manning. (2019). A Structural Probe for Finding Syntax in Word Representations. NAACL.
30. 30. Murray Shanahan. (2024). Talking About Large Language Models. MIT Press.