GraphRAG: Community-Summarization Retrieval

A single fact is a lonely thing. Connect it to other facts and you have understanding. Connect enough facts and you have wisdom.

RAG, Fact-Hoarding AI Agent

35.4.2 The Microsoft GraphRAG Pipeline

pip install graphrag
# 1. Initialize a project skeleton (creates ./settings.yaml and ./prompts/).
python -m graphrag init --root ./ragtest
# 2. Drop .txt files into ./ragtest/input/, then build the index:
python -m graphrag index --root ./ragtest
# 3. Query in either mode:
python -m graphrag query --root ./ragtest --method global \
    --query "What are the main themes in this corpus?"
python -m graphrag query --root ./ragtest --method local \
    --query "How is Einstein connected to Bohr?"

Microsoft's GraphRAG (Edge et al., 2024) introduced a four-stage pipeline that transforms an unstructured corpus into a queryable knowledge graph with hierarchical community summaries. The pipeline proceeds as follows:

1. Entity and relation extraction: An LLM processes each text chunk to extract entities (persons, organizations, concepts) and the relationships between them, producing triples like (Einstein, developed, General Relativity).
2. Graph construction and merging: Extracted entities undergo coreference resolution (merging "Einstein" and "Albert Einstein" into a single node) and are assembled into a unified knowledge graph.
3. Community detection: The Leiden algorithm partitions the graph into communities of densely connected entities at multiple hierarchical levels.
4. Community summarization: An LLM generates a natural language summary for each community, capturing the key themes, entities, and relationships within that cluster.

At query time, GraphRAG supports two search modes. Local search starts from entities mentioned in the query, retrieves their graph neighborhood and associated text chunks, and generates an answer grounded in that local context. Global search distributes the query across all community summaries using a map-reduce pattern: each summary is scored for relevance, the most relevant summaries are aggregated, and the LLM synthesizes a final answer from the combined context.

Concretely, suppose you have indexed 5,000 medical research papers about diabetes. A local query like "what drugs interact with metformin?" identifies the Metformin node, walks one or two hops along interacts_with edges, collects the connected nodes (Warfarin, Cimetidine, Furosemide) and the supporting text chunks, and asks the LLM "given this neighborhood, what interacts with metformin?". A global query like "what are the major themes in this corpus?" cannot be answered from any single node's neighborhood, so it instead asks "do you see this theme?" against each of the 52 community summaries in parallel, ranks the summaries that responded affirmatively, and asks the LLM to synthesize across them. Local search uses graph traversal; global search uses map-reduce over precomputed community summaries.

Figure 35.4.1: The Microsoft GraphRAG pipeline. Indexing proceeds through four stages: entity extraction, graph construction with coreference resolution, community detection via the Leiden algorithm, and LLM-generated community summaries. At query time, local search traverses entity neighborhoods while global search distributes queries across all community summaries using a map-reduce pattern.

35.4.2.1 Running the Microsoft GraphRAG Pipeline

This snippet runs the Microsoft GraphRAG indexing and query pipeline on a local document corpus.

import os
import graphrag
from graphrag.config import GraphRagConfig
from graphrag.index import run_pipeline
# Configure the pipeline
config = GraphRagConfig(
root_dir="./ragtest",
input_dir="./ragtest/input",
llm={
"type": "openai_chat",
"model": "gpt-4o",
"api_key": os.environ["OPENAI_API_KEY"],
},
embeddings={
"llm": {
"type": "openai_embedding",
"model": "text-embedding-3-small",
}
},
chunks={"size": 1200, "overlap": 100},
entity_extraction={
"max_gleanings": 1, # additional extraction passes
"prompt": None, # use default extraction prompt
},
community_reports={
"max_length": 2000, # max tokens per community summary
},
claim_extraction={"enabled": False},
)
# Run the full indexing pipeline
# This extracts entities, builds the graph, runs Leiden
# community detection, and generates community summaries
import asyncio
result = asyncio.run(run_pipeline(config))
print(f"Indexed {result.entities} entities, "
f"{result.relationships} relationships, "
f"{result.communities} communities")

from graphrag.query import LocalSearch, GlobalSearch
# Local search: entity-centric queries
local_search = LocalSearch(
    config=config,
    llm=config.llm,
    context_builder_params={
    "text_unit_prop": 0.5, # weight for text chunks
    "community_prop": 0.1, # weight for community info
    "conversation_history_prop": 0.1,
    "top_k_entities": 10,
    "top_k_relationships": 10,
    }
)
local_result = await local_search.asearch(
    "What side effects does metformin cause?"
)
print(local_result.response)
print(f"Sources: {len(local_result.context_data['sources'])}")
# Global search: corpus-level thematic queries
global_search = GlobalSearch(
    config=config,
    llm=config.llm,
    map_reduce_params={
    "map_max_tokens": 1000,
    "reduce_max_tokens": 2000,
    }
)
global_result = await global_search.asearch(
    "What are the major drug safety concerns across all reports?"
)
print(global_result.response)

35.4.3 What Makes GraphRAG Different From Generic KG-RAG

The generic KG-grounded retrieval substrate (property graph storage, LLM-extracted triples, Cypher path queries, hybrid graph + vector retrieval) is the territory of Section 35.3. GraphRAG is what you get when you bolt three additional layers on top of that substrate. First, community detection with the Leiden algorithm partitions the graph into nested clusters of densely-connected entities. Second, LLM-generated community summaries precompute a natural-language description of each cluster's central entities, relationships, and themes. Third, the local / global query router decides whether to fan a query out to entity neighborhoods (local) or to map-reduce it across community summaries (global). The community-summary step is the load-bearing one: it is what makes corpus-level "what are the main themes?" queries tractable at serving time, because the expensive thematic synthesis happened at indexing time.

35.4.3.1 Community Summaries as Pre-Computed Themes

For each community at each level of the Leiden hierarchy, GraphRAG asks an LLM to produce a 1 to 2 page summary describing the entities, relationships, and themes in that community. These summaries are GraphRAG's distinctive artifact. At query time, a global search over "what are the main safety themes across all clinical trial reports?" runs map-reduce: each community summary is scored for relevance, the top-k are concatenated, and the LLM produces a final synthesized answer. None of this is possible with generic KG retrieval alone, because the structured graph holds entities and edges but not theme-level prose.

Formally, let $G = (V, E)$ be the extracted knowledge graph and let $\{C_1^\ell, \ldots, C_{k_\ell}^\ell\}$ be the Leiden partition at hierarchy level $\ell$, where each community $C_j^\ell \subseteq V$ is a connected subset of entities. The community summary $s_j^\ell$ is the LLM-generated description of that subgraph,

$$ \begin{aligned} s_j^\ell \;=\; \mathrm{LLM}_{\text{sum}}\!\Big(\, & \big\{ (v, \mathrm{desc}(v)) : v \in C_j^\ell \big\} \;\cup\; \\ & \big\{ (u, r, v) \in E : u, v \in C_j^\ell \big\} \;\cup\; \\ & \big\{ s_{j'}^{\ell-1} : C_{j'}^{\ell-1} \subseteq C_j^\ell \big\} \,\Big), \end{aligned} $$

which folds in the entity descriptions, the relationships internal to the community, and the summaries of any sub-communities one level down. A global query $q$ is then answered by map-reducing over the level-$\ell$ summaries,

$$ \begin{aligned} \mathrm{Answer}(q) \;=\; \mathrm{LLM}_{\text{reduce}}\!\Big(\, & q,\; \\ & \mathrm{top\text{-}}k\big\{ (s_j^\ell, \mathrm{score}(q, s_j^\ell)) : j = 1, \ldots, k_\ell \big\} \,\Big), \end{aligned} $$

where the relevance score is itself an LLM map step that asks "does this community summary help answer $q$?" The expensive thematic synthesis happens once at indexing time inside $\mathrm{LLM}_{\text{sum}}$; serving-time work is the much cheaper map-reduce over precomputed prose.

35.4.3.2 Quality Bottleneck: Coreference and Extraction

GraphRAG's quality is dominated by the quality of the underlying graph, which means by the quality of entity extraction and coreference resolution. Both topics are covered for general KG construction in Section 35.3; the GraphRAG-specific multiplier is that community detection assumes "one entity = one node". Duplicated entities (Apple Inc. vs Apple vs AAPL) silently split a real community into multiple smaller ones, which produces multiple narrow summaries instead of one cohesive theme. Empirically, a graph with 20% entity duplication produces global-search results that score worse than naive RAG because the community summaries are themselves fragmented. Schema-guided extraction, multi-pass gleaning, and an embedding + LLM coreference pass (also discussed in 35.3) are the standard fixes.

35.4.4 Evaluation: Faithfulness and Completeness

Evaluating GraphRAG requires metrics that capture both the faithfulness of generated answers (are claims supported by retrieved evidence?) and the completeness of retrieval (did the system find all relevant information?). The original GraphRAG paper evaluates on query-focused summarization (QFS) tasks, where the goal is to produce comprehensive summaries that address a specific question across an entire corpus.

35.4.4.1 Evaluation Metrics

35.4.4.2 Benchmarking Datasets

Microsoft released the graphrag-benchmarking-datasets collection, which provides standardized corpora for evaluating GraphRAG systems. These datasets include pre-annotated entities, communities, and reference summaries. The AP News and Podcast Transcripts datasets are commonly used for benchmarking, with queries spanning both local factual questions and global thematic questions.

import json
from openai import OpenAI
client = OpenAI()
def evaluate_comprehensiveness(query, answer_a, answer_b):
 """Pairwise LLM-as-judge evaluation for comprehensiveness."""
prompt = f"""You are evaluating two answers to the following question.
    Question: {query}
    Answer A:
 {answer_a}
    Answer B:
 {answer_b}
    Which answer is more comprehensive? Consider:
    1. Coverage of all relevant aspects of the question
    2. Depth of detail for each aspect
    3. Inclusion of supporting evidence
    Respond with a JSON object:
 {{"winner": "A" or "B" or "tie",
     "explanation": "brief reasoning",
     "score_a": 1-5,
     "score_b": 1-5}}"""
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": prompt}],
response_format={"type": "json_object"},
)
return json.loads(response.choices[0].message.content)
# Compare GraphRAG global search vs. naive RAG
graphrag_answer = global_result.response
naive_rag_answer = naive_rag_pipeline.query(query)
eval_result = evaluate_comprehensiveness(
query="What are the major drug safety themes?",
answer_a=graphrag_answer,
answer_b=naive_rag_answer,
)
print(f"Winner: {eval_result['winner']}")
print(f"GraphRAG: {eval_result['score_a']}/5, "
f"Naive RAG: {eval_result['score_b']}/5")

Exercises

from collections import defaultdict
def hybrid_retrieve(query, k=10):
    vec_hits = vector_index.search(embed(query), 20)
    entities = ner(query)
    graph_hits = []
    for entity in entities:
        graph_hits += graph.one_hop_neighbors(entity)
        bm25_hits = bm25_index.search(query, 10)
        scores = defaultdict(float)
        for hits in (vec_hits, graph_hits, bm25_hits):
            for r, h in enumerate(hits): scores[h.id] += 1 / (60 + r)
            return sorted(scores, key=scores.get, reverse=True)[:k]