The benefits of narrow AI, and the challenges in its creation
Currently, AI systems relevant to any particular domain are general systems that are relevant in every domain. When domain-specific models do get created, they are typically general foundation models fine-tuned on a particular task, rather than new models trained from scratch (with some exceptions). This is convenient and powerful, as a single general model can be used for a variety of applications.
However, AI systems used in specific applications can possess large amounts of knowledge which is never needed in those applications. Instead, we could use smaller, specialized narrow AI networks which preserve the coding knowledge of general systems without the same breadth of irrelevant knowledge.
Narrow AI systems may also pose fewer safety risks in sensitive domains than general systems, be easier to understand, or have properties that are easier to verify. Additionally, for systems to operate autonomously in the world requires AI models to have a wide breadth of “skills.” An ecosystem of narrow “tool AI” systems may therefore reduce loss-of-control risks and better support human agency in the long term.
In a recent arXiv preprint1, researchers from MIT and the U.S. National Science Foundation (NSF) AI Institute for Artificial Intelligence and Fundamental Interactions (IAIFI) investigate two challenges in regard to how neural networks learn and represent AI model skills that are relevant to the problem of creating narrow AI systems.
Two challenges
The first challenge relates to when it is possible to train narrow AI models from scratch. Through experiments on a synthetic task, the researchers find that it is sometimes necessary to train networks on a wide distribution of data to learn certain narrow skills within that distribution. This effect arises when skills depend on each other hierarchically, and training on a broad distribution introduces a curriculum which substantially accelerates learning.
The second challenge regards how to transfer particular AI model skills from large general models into small specialized models. The researchers look at two approaches: distillation and pruning. Distillation2 is a technique designed to transfer the knowledge of a large pre-trained model (the “teacher”) into a smaller model (the “student”). Pruning3 is the practice of removing parameters from an existing artificial neural network with the aim of reducing the size of the network while maintaining accuracy.
Skills may not be localizable to a particular set of model components. In this case, pruning of model components won’t precisely retain wanted skills and remove unwanted skills. However, the researchers tentatively find that methods based on pruning outperform distillation and training networks from scratch for the creation of smaller, more narrow systems. Reinforcing this finding, they acknowledge that their “greedy” pruning strategy is quite simple, and they cannot rule out that more sophisticated pruning strategies would be more successful in preserving some skills while unlearning others.
Article source: Michaud, Parker-Sartori, & Tegmark, 2025. CC BY 4.0.
Header image source: Created by Bruce Boyes with Perchance AI Photo Generator.
References:
- Michaud, E. J., Parker-Sartori, A., & Tegmark, M. (2025). On the creation of narrow AI: hierarchy and nonlocality of neural network skills. arXiv preprint arXiv:2505.15811. ↩
- Yadav, V., & Pandey, N. (2024, December 6). Distillation: Turning Smaller Models into High-Performance, Cost-Effective Solutions. Microsoft AI Platform Blog. ↩
- Wikipedia, CC BY-SA 4.0. ↩
