GraphRAG: When Knowledge as a Graph Beats Vectors

A company producing technical documentation for a dozen product lines notices a problem: the RAG assistant cites fragments well but can’t answer the question, “Which components from the older line X are compatible with the new line Y, and what certifications cover this combination?” This is a multi-step question requiring information from three different documents. Semantic search returns the best-matching fragments separately but doesn’t build bridges between them.

This is exactly the niche for GraphRAG.

A knowledge graph is a data structure composed of nodes (entities) and edges (relationships). In a corporate context: nodes are products, processes, departments, regulations, customers; edges are relationships like “requires,” “replaces,” “reports to,” “produced by.”

Building a graph from documents involves three steps.

Entity and relationship extraction. The language model reads each document fragment and extracts pairs (entity, relationship, entity). For example, from the sentence “Component A-7 requires CE certification before deployment in the EU,” the model extracts nodes A-7 and CE and the edge “requires.” This is the most costly part: with 10,000 fragments and a local model, you’re looking at tens to hundreds of minutes of processing.

Deduplication and normalization. The same product may appear as “A-7,” “component A7,” “module A-7 rev.2.” Without normalization, the graph fragments instead of consolidating knowledge. Here, a decision is needed: a synonym dictionary written manually by domain experts or an additional LLM step recognizing aliases. Both have costs.

Index + vector database. In full GraphRAG, you don’t abandon embeddings: you store both a relational graph (e.g., Neo4j, Memgraph, or lighter NetworkX for smaller corpora) and a vector index for entry-point search. The query first hits the vector index to find nearby entities, then the graph to trace relationships.

Vector search measures semantic similarity. Graphs measure connections. These are different questions.

Multi-step (multi-hop) queries. “What regulations apply to product X, which we sell to sector Y?” Vectors return documents thematically close to the query. The graph can traverse the path: product X → CE certification → machinery directive → Article 12 → audit requirement. Each step is an edge in the graph, not necessarily explicitly present in a single document fragment.

Entity-centric queries. “List all suppliers of components used in product line Z.” Vectors may miss suppliers mentioned marginally. The graph with “supplier-component-line” edges answers completely—if the graph is complete.

Knowledge consistency across documents. When the same entity appears in documents from different years and its attributes change, the graph helps track history and versions. Classic RAG may return conflicting fragments without signaling the contradiction.

Explanation and decision trace. An answer generated from a graph can show the literal node path: “This answer follows from the relationship A→B→C.” This is critical for systems classified as high-risk under the AI Act and requiring explainability.

For most corporate RAG deployments, a vector index with hybrid search is sufficient. Graphs make sense when:

• The corpus has a clear entity-relationship-entity structure that can be meaningfully extracted (product documentation, legal registries, domain knowledge bases);
• User questions genuinely require linking information from multiple sources through shared nodes;
• You have resources to maintain the graph, as every document update requires re-extraction or edge patching.

Don’t implement GraphRAG when:

• The corpus consists of loose descriptive documents without clear entities (strategic reports, meeting notes, correspondence);
• Questions are primarily descriptive and conceptual (where semantic search performs well);
• You have fewer than a few thousand documents and classic RAG achieves recall above 0.85 on the golden set;
• Token budget and extraction time are limited, and the deployment timeline is short.

At Cashcrown, we default to recommending starting with classic RAG with hybrid search and measuring recall. GraphRAG comes into play when data shows a specific class of queries that vectors don’t handle.

Extracting a graph from 50,000 fragments with a local model (e.g., llama3.1:70b on Ollama) takes 8-24 hours and requires GPU memory. With a commercial API, it’s tens to hundreds of dollars for a one-time extraction, plus similar costs for major corpus updates. Numbers depend heavily on fragment length and extraction prompt complexity: we provide ranges, not guarantees.

A typical GraphRAG stack in 2026 looks like this:

Step 1: Chunking and extraction. Documents go through chunking to the extraction model. The prompt instructs the model to return structured JSON with (subject, predicate, object) pairs. Use structured output with JSON Schema validation, as raw model text for relationship extraction can be inconsistent.

Step 2: Graph construction. Extracts go to a graph database or simple in-memory structure. For smaller corpora (up to a few thousand nodes), Python’s NetworkX suffices. For larger corpora and Cypher queries, consider Neo4j or Memgraph with self-hosting.

Step 3: Retrieval. The user query hits the vector index (entry point in the graph). From the top-k vector nodes, the system expands the neighborhood in the graph: fetches nodes 1-2 edges away. This set goes into the LLM context.

Step 4: Generation with path citation. The model generates an answer, citing the node path as the source. This isn’t a technical requirement but significantly boosts user trust and meets explainability requirements for AI Act-regulated systems.

Step 5: Observability. Measure vector query recall (hits in top-5) and graph path recall (whether multi-hop questions yield complete answers) separately. Without splitting these metrics, you won’t know which component needs optimization.

We describe a similar pattern in the context of vector database selection in how to choose a vector database and when discussing fragment reranking in reranking as a RAG quality layer.

Graph extraction has three costly pitfalls rarely mentioned in technical articles.

Pitfall 1: Extraction quality drops with low-quality documents. OCR scans, documents with tables, or bulleted lists without sentence context yield chaotic relationships. Before graph extraction, ensure chunking quality, as discussed in preparing data for RAG.

Pitfall 2: The graph becomes outdated with every document update. In classic RAG, you update the embedding of a fragment in seconds. In GraphRAG, you must re-extract relationships from that fragment and update edges in the graph. If documents change frequently, maintenance costs rise quickly. Incremental updates (only changed documents) work but require version bookkeeping.

Pitfall 3: The graph can perpetuate errors. If the extraction model misinterprets a relationship, that relationship stays in the graph and affects answers until manually corrected. Classic RAG may pick the wrong fragment, but the error isn’t structurally embedded. With GraphRAG, periodically audit a sample of edges. This requires human effort, not automation.

No. It’s a complement, not a replacement. In most corporate RAG deployments, vector RAG with hybrid search covers 80-90% of use cases. GraphRAG serves as an additional layer for specific multi-step query classes. Deployments that replaced vectors with graphs alone typically reverted to a hybrid approach after a few weeks.

Microsoft released the open-source GraphRAG library (Python), which automates entity extraction and graph construction from text files. LangChain and LlamaIndex have integrations with graph databases (Neo4j, Memgraph). For self-deployment without external APIs, Ollama with a local model for extraction and NetworkX for graph storage works for corpora under tens of thousands of nodes. For larger datasets, a dedicated graph database with a Cypher interface significantly speeds up queries.

The range is wide and depends on the model and document length. With a local model (llama3.1:70b on Ollama): 8-24 hours of extraction, zero token costs. With a commercial API (GPT-4o or Claude Sonnet): $50-300 for a one-time extraction, plus similar costs for updates. Add the cost of entity normalization (usually 1-2 days of domain expert work for the first extraction) and plan time for auditing a sample of edges. We provide ranges based on typical projects, not a specific quote without data analysis.

The node path as a decision trace helps meet the explainability requirement (Art. 13 AI Act). However, this alone isn’t sufficient: system documentation, test case logs, and human gates for irreversible decisions are also required. GraphRAG improves the explainability of the retrieval mechanism but doesn’t relieve the implementer of compliance obligations.

When the corpus has a complex ontology (many entity and relationship types), documents are of low structural quality (scans, legacy PDFs), or the timeline is short. Relationship extraction requires several iterations of domain-specific prompt engineering, sample validation, and correction. At Cashcrown, we assess technical readiness and data before recommending an architecture, because a corporate RAG built on poor graph extraction performs worse than a simple vector with semantic search.