Fatemeh Mohammadi, Leila Ebrahimi, 2026. "The Ontogenesis of Machine Knowledge: Mechanisms of Representation Emergence", Journal of Frontiers in Artificial Intelligence and Machine Learning 1(1): 64-78.

The ontogenesis of machine knowledge is the processes whereby artificial intelligence systems develop, organize, and refine internal representations which function to permit intelligent behavior. Machine knowledge ontogenesis is motivated by the biological concept of ontogenesis, which is how an organism develops during its lifetime, describes how neural networks convert raw data into structured animal representations, conceptual understanding and perfect computational knowledge. Recent progress in deep learning suggests that the artificial intelligence (AI) of today is able to learn ever more complex knowledge without explicit symbolic programming. Rather, knowledge is revealed through interactions between neural representations and learning and optimization algorithms, which are shaped by environmental inputs. The mechanisms by which these processes, and cognitive development more generally occur, has quickly become a central problem in modern AI as understanding how we come to behave intelligently sheds light on how intelligent behaviour will arise when incorporating distinctive genetic or environmental components into future Generative Adversarial Network (GAN) architectures for greater levels of independence and flexibility.
This work is a research on the mechanisms that give rise to machine knowledge in neural architectures. Special focus is on representation learning, shape of features, layout of latent space, self-organization process, memory consolidation and reasoning ability emerging. The research investigated how neural systems evolve from straightforward detection of shapes to complex conceptual interpretation over a series of levels in representational growth. It further explores the effects of scaling, information integration and adaptive learning dynamics on internal knowledge structures. The analysis of the relationship between neural representations and higher-order cognitive function, such as reasoning, decision-making, and problem-solving which inform a model that if its machines can construct an abstract model from the world.
The results show that machine knowledge does not consist of a bank account full of discrete facts or a programmed set of nearly finished rules but rather, emerges from distributed representational structures which dynamically form with continual learning. It seems as though knowledge development is driven by self-organizing computing processes that specify experience into increasingly abstract and general forms of understanding. These processes allow neural systems to learn conceptual relationships, transfer knowledge from one task to another and produce novel solutions in new situational contexts. The ontogenesis of machine knowledge holds relevance on the road to building more interpretable, adaptive, and trustable AI systems. In addition, it lays the groundwork for future investigations of artificial cognition, autonomous learning and the SCIENTIFIC laws of intelligence emergence in computational constructs.

1. 1. Geoffrey Hinton, Rumelhart, D. E., & Williams, R. J. (1986). Learning Representations by Back-Propagating Errors. Nature, 323(6088), 533–536.
2. 2. Yoshua Bengio, Courville, A., & Vincent, P. (2013). Representation Learning: A Review and New Perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8), 1798–1828.
3. 3. Ian Goodfellow, Yoshua Bengio, & Courville, A. (2016). Deep Learning. MIT Press.
4. 4. Ashish Vaswani, et al. (2017). Attention Is All You Need. NeurIPS.
5. 5. Jacob Devlin, et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL-HLT.
6. 6. Alec Radford, et al. (2019). Language Models are Unsupervised Multitask Learners. OpenAI.
7. 7. Tom Brown, et al. (2020). Language Models are Few-Shot Learners. NeurIPS.
8. 8. Jared Kaplan, et al. (2020). Scaling Laws for Neural Language Models. arXiv.
9. 9. Jordan Hoffmann, et al. (2022). Training Compute-Optimal Large Language Models. arXiv.
10. 10. Christopher Olah, et al. (2020). Zoom In: An Introduction to Circuits. Distill.
11. 11. Nelson Elhage, et al. (2021). A Mathematical Framework for Transformer Circuits. Anthropic.
12. 12. Catherine Olsson, et al. (2022). In-Context Learning and Induction Heads. Anthropic Research.
13. 13. Anthropic. (2022). Toy Models of Superposition. Transformer Circuits Thread.
14. 14. Kenneth Li, et al. (2023). Measuring and Controlling Knowledge in Language Models. ICLR.
15. 15. John Hewitt & Christopher Manning. (2019). A Structural Probe for Finding Syntax in Word Representations. NAACL.
16. 16. David Bau, et al. (2019). Identifying and Controlling Important Neurons in Neural Machine Translation. ICLR.
17. 17. Been Kim, et al. (2018). Interpretability Beyond Feature Attribution with Concept Activation Vectors (TCAV). ICML.
18. 18. Amirata Ghorbani, et al. (2019). Towards Automatic Concept-Based Explanations. NeurIPS.
19. 19. Samuel Bowman, et al. (2022). Measuring Emergent Abilities of Large Language Models. arXiv.
20. 20. Jason Wei, et al. (2022). Emergent Abilities of Large Language Models. TMLR.
21. 21. Andrew Saxe, et al. (2014). Exact Solutions to the Nonlinear Dynamics of Learning in Deep Linear Neural Networks. ICLR.
22. 22. Surya Ganguli, et al. (2018). Statistical Mechanics of Deep Learning. Annual Review of Condensed Matter Physics.
23. 23. Haim Sompolinsky, Crisanti, A., & Sommers, H. J. (1988). Chaos in Random Neural Networks. Physical Review Letters.
24. 24. Max Tegmark & Henry Lin. (2017). Why Does Deep and Cheap Learning Work So Well? Journal of Statistical Physics.
25. 25. Sebastian Ruder. (2019). Neural Transfer Learning for Natural Language Processing. Springer.
26. 26. Aakanksha Chowdhery, et al. (2022). PaLM: Scaling Language Modeling with Pathways. arXiv.
27. 27. Quoc Le, et al. (2023). PaLM 2 Technical Report. Google Research.
28. 28. Sebastian Borgeaud, et al. (2022). Improving Language Models by Retrieving from Trillions of Tokens. ICML.
29. 29. Rishi Bommasani, et al. (2021). On the Opportunities and Risks of Foundation Models. Stanford CRFM.
30. 30. Arthur Conmy, et al. (2023). Towards Automated Circuit Discovery for Mechanistic Interpretability. NeurIPS Workshop.
31. 31. Neel Nanda. (2023). TransformerLens: A Library for Mechanistic Interpretability. GitHub Documentation.
32. 32. Richard Sutton & Andrew Barto. (2018). Reinforcement Learning: An Introduction (2nd Edition). MIT Press.
33. 33. Percy Liang, et al. (2022). Holistic Evaluation of Language Models. TMLR.
34. 34. Murray Shanahan. (2024). Talking About Large Language Models. MIT Press.
35. 35. Rylan Schaeffer, et al. (2024). Representation Emergence and Knowledge Formation in Large Neural Networks: A Survey. arXiv.