Advances & challenges in foundation agents: Section 2.1.3 – Learning space

This article is Section 2.1.3 of a series of articles featuring Liu and colleagues’ book Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems.

The learning approaches in LLM agents represent a structured, data-driven paradigm in contrast to the exploratory, emotionally-driven learning observed in humans. While human learning often involves active curiosity, motivation, and emotional reinforcement, LLM-based agents typically learn through more formalized processes, such as parameter updates during training or structured memory formation during exploration. Current agent architectures attempt to bridge this gap by implementing mechanisms that simulate aspects of human learning while leveraging the strengths of computational systems.

Learning within an intelligent agent occurs across different cognitive spaces, encompassing both large-scale model updates and more localized changes to modularized mental states M. In systems where the model is the only trainable component, the model parameters θ can be viewed as constituting or encoding the entire mental state. More generally, the mental state can include a combination of subsystems:

M = { M^θ, M^mem, M^wm, M^emo, M^goal, M^rew }

where M^θ, denotes the core model parameters, M^mem represents memory, M^wm denotes the world model, M^emo indicates emotional state, M^goal represents goals, and M^rew represents reward signals. Modifications to M^θ—the core model—often lead to holistic changes that affect all components of the mental state. In contrast, more targeted updates to memory, world model, or reward components allow the agent to adapt specific subsystems while preserving general capabilities. For instance, learning experiences and skills from the environment primarily influence memory, while leveraging the LLM’s inherent predictive capabilities enhances the world model. The distinction between these two paradigms—full mental state learning and partial component-level adaptation—is illustrated in Figure 2.3.

Full mental state learning. Full mental state learning enhances the capabilities of an agent through comprehensive modifications to M^θ, which in turn influences all components of M. This process begins with pre-training, which establishes the foundation of language models by acquiring vast world knowledge, analogous to how human babies absorb environmental information during development, though in a more structured and extensive manner.

Post-training techniques represent the cornerstone for advancing agent capabilities. Similar to how human
brains are shaped by education, these techniques, while affecting the entire model, can emphasize different aspects of cognitive development. Specifically, various forms of tuning-based learning enable agents to acquire domain-specific knowledge and logical reasoning capabilities. Supervised Fine-Tuning (SFT)¹ serves as the fundamental approach where models learn from human-labeled examples, encoding knowledge directly into the model’s weights. For computational efficiency, Parameter-Efficient Fine-Tuning (PEFT) methods have emerged. Adapter-BERT² introduced modular designs that adapt models to downstream tasks without modifying all parameters, while Low-Rank Adaptation (LoRA)³ achieves similar results by decomposing weight updates into low-rank matrices, adjusting only a small subset of effective parameters.

Some agent capabilities are closely connected to how well they align with human preferences, with alignment-based learning approaches modifying M^θ to reshape aspects of the agent’s underlying representations. Reinforcement learning from human feedback (RLHF)⁴ aligns models with human values by training a reward model on comparative judgments and using this to guide policy optimization. InstructGPT⁵ demonstrated how this approach could dramatically improve consistency with user intent across diverse tasks. Direct Preference Optimization (DPO)⁶ has further simplified this process by reformulating it as direct preference learning without explicit reward modeling, maintaining alignment quality while reducing computational complexity.

Reinforcement learning (RL) presents a promising pathway for specialized learning in specific environments. RL has shown particular promise in enhancing reasoning capabilities, essentially enabling the agent to refine its internal thinking processes. Foundational works such as Reinforcement Fine-Tuning (ReFT)⁷ enhance reasoning through fine-tuning with automatically sampled reasoning paths under online reinforcement learning rewards. DeepSeek-R1⁸ advances this approach through rule-based rewards and Group Relative Policy Optimization (GRPO)⁹, while Kimi k1.5¹⁰ combines contextual reinforcement learning with optimized chain-of-thought techniques to improve both planning processes and inference efficiency. In specific environments, modifying models to enhance agents’ understanding of actions and external environments has proven effective, as demonstrated by DigiRL¹¹, which implements a two-stage reinforcement learning approach enabling agents to perform diverse commands on real-world Android device simulators.

Recent works have attempted¹² to¹³ integrate¹⁴ agent¹⁵ action¹⁶ spaces¹⁷ directly into model training, enabling learning of appropriate actions for different states through RL or SFT methods. This integration fundamentally affects the agent’s memory, reward understanding, and world model comprehension, pointing toward a promising direction for the emergence of agentic models.

Partial mental state learning. While full mental state learning through updates to Mθ provides comprehensive capability updates, learning focused on particular components of M represents another essential and often more efficient approach. Such partial mental state learning can be achieved either through targeted model updates or through in-context adaptation without parameter changes.

In-context learning (ICL) illustrates how agents can effectively modify specific mental state components without modifying the underlying model. This mechanism allows agents to adapt to new tasks by leveraging examples or instructions within their context window, paralleling human working memory’s role in rapid task adaptation. Chain-of-thought (CoT)¹⁸ demonstrates the effectiveness of this approach, showing how agents can enhance specific cognitive capabilities while maintaining their base model parameters unchanged.

The feasibility of partial mental state learning is evidenced through various approaches targeting different components such as memory (M^mem), reward (M^rew), and world model (M^wm). Through normal communication and social interaction, Generative agents¹⁹ demonstrate how agents can accumulate and replay memories, extracting high-level insights to guide dynamic behavior planning. In environmental interaction scenarios, Voyager²⁰ showcases how agents can continuously update their skill library through direct engagement with the Minecraft environment, accumulating procedural knowledge without model retraining. Mem0²¹ provides agents with persistent and efficient long-term memory through scalable dynamic memory management and graph-based memory representations, significantly enhancing their ability to handle complex, multi-session tasks.

Learn-by-Interact²² further extends this approach by synthesizing experiential data through direct environmental interaction, eliminating the need for manual annotation or reinforcement learning frameworks. Additionally, agents can learn from their mistakes and improve through reflection, as demonstrated by Reflexion²³, which guides agents’ future thinking and actions by obtaining textual feedback from repeated trial and error experiences. Further, KnowSelf²⁴ introduces knowledgeable self-awareness, enabling agents to intelligently assess whether external knowledge is needed or to self-correct based on specific contexts, leading to more efficient and strategic planning.

Modifications to reward and world models provide another example of partial mental state learning. ARMAP²⁵ refines environmental reward models by distilling them from agent action trajectories, providing a foundation for further learning. AutoMC²⁶ constructs dense reward models through environmental exploration to support agent behavior. Meanwhile, LLMs are explicitly leveraged as world models²⁷ to predict the impact of future actions, effectively modifying the agent’s world understanding (M^wm). ActRe²⁸ builds upon the language model’s inherent world understanding to construct tasks from trajectories, enhancing the agent’s capabilities as both a world model and reasoning engine through iterative training.

Next part: Section 2.1.4 – Learning objective.

Article source: Liu, B., Li, X., Zhang, J., Wang, J., He, T., Hong, S., … & Wu, C. (2025). Advances and challenges in foundation agents: From brain-inspired intelligence to evolutionary, collaborative, and safe systems. arXiv preprint arXiv:2504.01990. CC BY-NC-SA 4.0.

Header image: AI is Everywhere by Ariyana Ahmad & The Bigger Picture / Better Images of AI, CC BY 4.0.

References: