Advances & challenges in foundation agents: Section 2.1.2 – Learning across mental state components

This article is Section 2.1.2 of a series of articles featuring Liu and colleagues’ book Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems.

Learning represents the fundamental process through which intelligent agents transform experiences into knowledge within their mental states. This transformation occurs across different cognitive spaces, from holistic updates across the full mental state to refinement of specific cognitive components. The scope of learning encompasses remarkable capacities that serve different objectives: enhancing perceptual understanding, improving reasoning capabilities, and developing richer world understanding. Recent developments in LLM-based agents demonstrate how different learning strategies update specific components of the mental state. Table 2.2 summarizes a selection of representative methods and highlights the cognitive subsystems they primarily affect—ranging from memory updates in systems like Voyager and Generative Agents to reward modeling in RewardAgent and Text2Reward, or world model construction in WebDreamer and AutoManual. This breakdown helps clarify how learning is distributed across the broader architecture of an agent.

Table 2.2: Summary of learning methods with different state modifications. • indicates primary impact while ◦ indicates secondary or no direct impact (source: adapted from Liu et al., 2025). 

Human learning operates across multiple spaces and objectives through the brain’s adaptable neural networks. The brain coordinates learning across its entire network through integrated systems¹: the hippocampus facilitates rapid encoding of episodic experiences, the cerebellum supports supervised learning for precise motor skills, the basal ganglia enable reinforcement learning through dopaminergic reward signals, and cortical regions facilitate unsupervised pattern extraction. At more focused levels, specific neural circuits can undergo targeted adaptation, allowing for specialized skill development and knowledge acquisition. These systems work together² on different timescales, ranging from immediate responses to lifelong development, while being influenced by factors like attention, emotions, and social environment.

LLM agents, while fundamentally different in architecture, implement analogous learning processes across their mental state spaces. At the comprehensive level, they acquire broad knowledge through pre-training on massive datasets, demonstrating a form of unsupervised learning. At more focused levels, they refine specific capabilities through parameter-updating mechanisms like supervised fine-tuning and reinforcement learning. Uniquely, they also demonstrate in-context learning capabilities, adapting to novel tasks without parameter changes by leveraging context within their attention window: a capability that mirrors aspects of human working memory but operates through fundamentally different mechanisms.

The comparison between human and artificial learning systems provides valuable insights for developing more capable, adaptive agents. Human learning demonstrates notable characteristics in efficiency, contextualization, and integration with emotional systems, while LLM-based approaches show distinct capabilities in processing large datasets, representing formal knowledge, and synthesizing information across domains. These complementary strengths suggest productive directions for research. As we explore the foundations of learning, we first examine the spaces where learning occurs within mental states, followed by an analysis of the specific objectives that drive learning processes.

Next part: Section 2.1.3 – Learning space.

Article source: Liu, B., Li, X., Zhang, J., Wang, J., He, T., Hong, S., … & Wu, C. (2025). Advances and challenges in foundation agents: From brain-inspired intelligence to evolutionary, collaborative, and safe systems. arXiv preprint arXiv:2504.01990. CC BY-NC-SA 4.0.

Header image: AI is Everywhere by Ariyana Ahmad & The Bigger Picture / Better Images of AI, CC BY 4.0.

References: