Using GraphRAG to enhance LLM-based information retrieval supporting knowledge management (KM)

This article is part of an ongoing series looking at AI in KM, and KM in AI.

Since the release of OpenAI’s ChatGPT in 2022, artificial intelligence (AI) large language models (LLMs) have revolutionized information processing because of their exceptional capabilities to understand context and generate text that closely resembles human language. However, LLMs face limitations such as knowledge rigidity, reliance on static training data, and susceptibility to generating inaccurate or misleading outputs.

In a recent paper¹ in the journal Electronics, authors Hisatoshi Naganawa, Enna Hirata, and Akira Yamada advise that these challenges highlight the need for hybrid approaches that combine LLMs with external databases to enhance reliability and contextual accuracy.

In response, they introduce a novel information management approach that uses graph retrieval-augmented generation (GraphRAG) to systematically organize and integrate literature related to the physical internet. Retrieval-augmented generation (RAG)² improves LLM outputs by incorporating external information retrieval before generating responses. Unlike LLMs alone that rely on static training data, RAG pulls relevant text from databases, uploaded documents, or web sources.

As shown in Figure 1, GraphRAG has the potential to improve LLM outputs even further by also incorporating knowledge graphs, which were introduced in a previous RealKM Magazine article series. Naganawa and colleagues explore this potential using the case study of the physical internet.

The physical internet and its information challenges

The concept of the physical internet was first introduced³ as a transformative framework for sustainable logistics. By applying the design principles of the digital Internet, the physical internet aims to create an interoperable, collaborative and efficient freight transport system.

The core idea is to route modular containers through transit hubs, called physical internet hubs, to maximize consolidation opportunities and minimize inefficiencies in transportation networks. Over the past decade, the physical internet has received considerable attention from researchers and practitioners alike for its potential to address critical challenges such as greenhouse gas emissions and resource inefficiencies in global logistics systems.

The growing body of literature on the physical internet reflects its increasing importance. The growing academic interest in the the physical internet is can be seen in the increasing volume of peer-reviewed publications and systematic literature reviews over the past decade.

However, the synthesis of current knowledge in the physical internet field is severely hindered by fragmentation, methodological issues, and the rate at which new research is produced. Manual review articles on the physical internet have reviewed less than 700 articles, yet there are more than 2000 scientific papers already published.

In contrast, by building a comprehensive, semantically rich knowledge graph, GraphRAG enables information to be integrated and retrieved from thousands of diverse sources. This scalable, flexible framework for information organization fills a critical gap left by traditional review methods. GraphRAG excels at uncovering relationships among data points, retrieving highly relevant information, and generating contextually rich outputs. These capabilities make it particularly well-suited for addressing complex problems with interconnected data structures, such as those found in physical internet research.

Technologies similar to GraphRAG are HybridRAG and LightRAG. The differences in these technologies are summarized in Table 1.

Overall, hybridRAG combines vector search and keyword search, enabling it to obtain data with strong superficial relevance, but it is difficult to capture indirect relationships. In addition, LightRAG is basically a vector search based on simple similarity and does not consider relationships, so although it is fast in generation, there is a high possibility that fragmentary answers will be mixed in the generated results.

Therefore, Naganawa and colleagues consider GraphRAG to have the best potential for addressing the information challenges of the physical internet.

Building the knowledge graph

As shown in Figure 2, construction of the knowledge graph (KG) began with data from four key sources:

1. Scientific literature, made up of 2537 relevant papers published between 2006 and 2024.
2. Proceedings of the International Physical Internet Conference (IPIC).
3. The European physical internet roadmap developed by the Alliance for Logistics Innovation through Collaboration in Europe (ALICE).
4. The national physical internet roadmap prepared by Japan’s Ministry of Economy, Trade and Industry (METI).

The raw data collected from the four sources was aggregated into a unified dataset. To ensure data quality, noise such as stop words, URLs, punctuation, and other irrelevant character symbols were removed. The cleaned and standardized corpus served as the input for subsequent graph construction.

After preprocessing, the cleaned data was hosted in a Neo4j AuraDB graph database. Using GPT-4o mini, entity and relationship extraction was conducted to form nodes and edges, which together created a structured knowledge graph representing the physical internet domain. The knowledge graph is shown in Figure 3.

Performance evaluation

Four retrieval strategies were implemented and compared to evaluate their performance in generating insights. The four strategies are, I: GPT-only, II: Local Search, III: Global Search, and IV: Hybrid Search. The details are explained in Table 2. Hybrid search is a strategy that combines “details” (Local) and “context” (Global) to generate more accurate and comprehensive information.

To preliminarily evaluate GraphRAG’s usefulness under different user needs, Naganawa and colleagues designed stakeholder-specific prompts for three groups: industry practitioners, policymakers, and academic researchers. Several parameters were employed to optimize GraphRAG’s performance, and metrics were used to quantitatively assess the quality of responses generated by each strategy.

The results are shown in Table 3, demonstrating GraphRAG’s potential as a powerful tool for organizing and synthesizing fragmented knowledge in complex domains like the physical internet.

The tailored prompts also revealed unique insights for each stakeholder group. This enhances GraphRAG’s applicability in real-world decision-making contexts. Industry practitioners can identify new business opportunities; policymakers can design informed strategies; and academics can uncover research gaps. All of these can be performed in a unified platform.

Limitations of the research

In concluding their paper, Naganawa and colleagues alert that wile their study possesses certain strengths, it also has several limitations. These include that while GraphRAG tends to exhibit lower hallucination risk compared to standard LLM generation, this risk is not entirely eliminated. Even with source linking capabilities, the model may still generate inaccurate yet plausible descriptions, potentially undermining user trust in identified research gaps or findings.

To mitigate such risks, it is crucial to provide referable and transparent outputs, such as explicitly indicating which graph nodes or edges support each part of the response. This enables users to verify the basis of the model’s answers and more effectively evaluate the reliability of the presented findings.

Article source: Naganawa et al., 2025. CC BY 4.0.

Header image source: Buzz on Pixabay.

References: