Vector RAG is the default way to give a language model your data, and for a lot of questions it is the right tool. You chunk your documents, embed each chunk, and at query time you pull back the handful of chunks whose embeddings sit closest to the question. Ask "what is our refund window" and the chunk that states the refund window comes back.
It breaks on a different shape of question. Ask "what are the main themes across these 400 support tickets" and there is no single chunk that looks like the answer, because the answer is not written down anywhere. Ask "how is our biggest customer connected to the outage last March" and the facts you need are spread across three documents that only relate through a shared name. Similarity search does one hop and has no idea those documents are linked. That gap is what GraphRAG fills.
What vector RAG cannot see
Embeddings capture meaning, not structure. A passage about Acme Corp and a passage about the March outage can both be relevant to your question while sitting far apart in embedding space, because they use different words. Three problems follow from that:
- Connecting facts across documents. When the answer requires joining facts that live in separate passages, linked only through a shared entity, chunks embedded in isolation never surface the link.
- Global questions. "Summarize the whole corpus" or "what are the recurring risks" need everything, but similarity search returns only what superficially matches the query string.
- Multi-hop reasoning. A to B to C means traversing relationships. Vector search does one similarity jump and stops.
You can soften these with a bigger top-k, a reranker, and query rewriting, and it helps at the margin. But the model is still guessing at structure it was never given.
What GraphRAG actually does
GraphRAG, introduced by Microsoft Research in April 2024 and open-sourced that July, builds the structure first. The indexing pipeline runs before any question is asked:
1. Chunk. Split documents into units (Microsoft's default is 1,200 tokens).
2. Extract the graph. An LLM reads each chunk and pulls out entities (a name, a type, a description) and relationships (source, target, description). There is no fixed schema; it extracts whatever is there.
3. Merge. The same entity mentioned in ten chunks becomes one node, and the LLM collapses its ten descriptions into one. That is entity resolution, built in.
4. Find communities. The hierarchical Leiden algorithm partitions the graph into clusters of densely connected nodes, recursively, from small leaf communities up to broad ones.
5. Summarize each community. The LLM writes a report for every community at every level, bottom up. These summaries are the expensive part, and they are what makes the next step fast.
At query time there are two modes, and picking the right one matters.
Global search answers corpus-wide questions. It runs your question against many community summaries in parallel (a map step), scores each partial answer, and combines the best ones (a reduce step). It never reads raw source text at query time; it reads the summaries it precomputed. That is how it answers "what are the themes" without stuffing 400 tickets into a context window.
Local search answers entity-centered questions. It starts from the entities most relevant to your question, then walks outward to their neighbors, their relationships, and the chunks and community summaries around them, assembling that neighborhood as context.
Does it actually beat vector RAG
Sometimes, and it depends heavily on the question.
On global sensemaking, Microsoft's own paper reported GraphRAG beating naive vector RAG on comprehensiveness and answer diversity by wide margins, while using far fewer tokens than summarizing the raw text would. Those are vendor-reported numbers, so read them as the authors' claim, not a neutral verdict.
Independent work lines up on multi-hop. On hard cross-document question-answering sets, graph-guided retrieval lifted recall@5 from the low 70s to the high 80s, with the biggest gains on the questions that span the most documents. Where it does not help: simple single-fact lookup, where graph and vector land in a statistical tie, and small corpora, where the indexing cost buys you nothing. A 2025 study that ran vector RAG against several GraphRAG variants under one fair protocol found no single winner across the board.
One honest caveat on all of this: most of these benchmarks use a language model as the judge, and those judges have documented biases, including a position bias large enough to swing a win rate by tens of points just by reordering the answers. Treat headline win rates as directional.
The cost you have to plan for
GraphRAG moves work to index time, and that work is LLM calls over every chunk plus a summary of every community. It is not cheap. Public estimates for a moderate corpus land in the tens of dollars against GPT-4o, and the figure people quote for a large corpus is that full GraphRAG can cost thousands to index. Global search at query time is heavy too, because it fans out across many summaries.
The ecosystem has spent the last year attacking that cost:
- LazyGraphRAG (Microsoft, late 2024) skips the upfront LLM extraction and summarization. It uses plain noun-phrase extraction to build the graph and defers all LLM work to query time. Microsoft reports indexing cost roughly on par with vector RAG, a small fraction of full GraphRAG, at comparable quality on global questions. Vendor-reported again.
- LightRAG keeps a lighter graph index, supports incremental updates without a full re-index, and offers a hybrid graph-and-vector mode.
- nano-graphrag is a tiny, readable reimplementation of the core idea for teams that want to see the whole pipeline.
The practical levers if you run GraphRAG yourself: use a cheap model for extraction and save the frontier model for the answer, use larger chunks, cache aggressively so re-runs skip processed content, and leave claim extraction off unless you need it.
Where the graph should live
You need somewhere to store nodes, edges, and embeddings. The two common answers are a dedicated graph database like Neo4j, or your existing Postgres.
Neo4j gives you native graph storage and index-free adjacency, which is genuinely faster for deep multi-hop traversal over very large graphs. The cost is another system to run, and the common production shape ends up being two stores anyway: the graph database for the graph, a vector store for the embeddings.
Postgres keeps everything in one place. With pgvector for embeddings and ordinary relational tables for entities and edges, you traverse with recursive queries and get one connection string, one backup, one thing to monitor. It handles vector workloads into the millions of rows, and it avoids a second database and its licensing. The tradeoff is real: very large graphs and deep traversals are less ergonomic in SQL and can be slower than a native graph engine. For most knowledge bases, where the graph is large but not enormous and questions are a few hops deep, that tradeoff is worth it.
That is the route we took for Cruq's knowledge graph. The graph lives in Postgres next to the rest of the workspace data, so an agent's knowledge base is one store to secure and back up, not a graph database bolted onto a vector database.
When to reach for it
The rule that survives contact with real data: use vector RAG when your questions are mostly single-fact and your corpus is small or simple. Reach for a graph when the corpus is richly interconnected, the same entities recur across many documents, and the questions are multi-hop, corpus-wide, or need an answer you can trace back to its sources. Compliance, legal, healthcare, financial filings, and large research corpora are the clearest fits, because in those the relationships are the point.
In production, do not treat it as either-or. The strongest setups route by question: vector search for the lookups, graph traversal for the ones that need structure, often with both feeding the same answer.
