Your leadership wants a chatbot that answers questions from company knowledge, and someone has suggested GraphRAG. Retrieval-augmented generation (RAG) is the standard way to let a language model answer from your documents. GraphRAG adds a knowledge graph of entities and relationships. This guide compares the two, helps you pick a graph database, and covers the part most GraphRAG projects get wrong: keeping the graph free of hallucinated and duplicate entities.

How does standard RAG work?

Documents are split into chunks and turned into vector embeddings. When someone asks a question, the system finds the most similar chunks and passes them to the language model, which writes an answer from that context. It’s quick to build and works well when the answer sits in a few passages.

Its weak spot is connections. Vector search finds passages that sound like the question. It can’t follow a chain such as person to project to deadline to status report, so multi-step questions get partial or invented answers.

How does GraphRAG work?

GraphRAG stores entities such as people, projects, products, customers and regulations, plus the relationships between them, in a knowledge graph. At question time it combines graph traversal with vector search, so the model receives connected facts instead of isolated passages. In practice the language model translates the user’s question into a graph query (Cypher, for Neo4j), runs it, and writes the answer from the returned facts and the documents attached to them.

How do RAG and GraphRAG compare?

Which questions need GraphRAG?

• Who owns the at-risk deliverables this sprint, and what did they commit to last week?

• Which of our projects used supplier X, and which of them had quality issues?

• Which products are affected if regulation Y changes?

• Who in the company has worked with customer Z, and on what?

Why do automatic LLM graph builders fail in production?

The usual GraphRAG pipeline lets a language model read every document and extract entities and relationships on its own. Automatic graph builders make this fast to try. When we piloted that approach for an enterprise client, the graph wasn’t reliable enough for leadership decisions, for three reasons:

• Duplicate entities. The same person shows up as “Maria Weber”, “M. Weber” and “Maria (Infrastructure Lead)”, which creates three nodes with three partial histories.

• Hallucinated structure. The model invents roles and relationships that don’t exist. Nothing in the graph catches the error, so it spreads into every later answer.

• Token cost. Every document passes through the LLM for extraction, classification and deduplication. For a midsize organization that can cost thousands of dollars per ingestion cycle.

A more reliable pattern: the two-layer fixed entity architecture

The fix is to stop asking the model to guess your company’s structure. In our proof of concept for a multi-team enterprise, we split the graph into two layers:

• Fixed entity layer. A verified skeleton of known people, roles, projects and departments, loaded from authoritative structured sources. No LLM writes to this layer, so it is the ground truth.

• Document layer. Meeting transcripts, status reports and chat logs are chunked, embedded and attached to the fixed entities by similarity. Every chunk must attach to an existing verified node. There are no orphan chunks.

The result was zero entity duplication across the graph, and leadership meeting prep went from hours of reading to a cited answer in seconds. Indexing cost stayed an order of magnitude below LLM-based extraction. We also built a transparency mode that shows the generated Cypher query and raw results, because leadership wouldn’t accept a black box.

The trade-offs are real. A new department or project lead must be added to the fixed layer before the chatbot can reason about them. That suits an enterprise with a stable org chart and slows down a fast-changing startup. We also limited the proof of concept to one primary source document, to prove the reasoning layer before scaling ingestion.

Which graph database should you use for GraphRAG?

Neo4j is the most common choice. It has a mature query language (Cypher), built-in vector indexes so graph and vector search run in one database, and managed hosting with AuraDB. Language models also write Cypher more reliably than most other graph query languages, because there are more public examples of it.

Choose on update frequency, expected graph size, native vector search, access control at node or document level, and where your data already lives. If that’s a lakehouse, our Fabric vs Databricks comparison covers the platform side. At tens of millions of entities, the bottleneck is usually ingestion quality and query design, not storage.

How do you reduce hallucinations in a knowledge graph chatbot?

• Constrain query generation. Give the model the exact graph schema at question time, plus a few examples, so it can’t query node types or relationships that don’t exist.

• Ground every answer. Answer only from returned graph facts and attached documents, and cite them. If nothing is found, the bot should say so.

• Test the edges. Simple questions translate reliably into graph queries. Ambiguous or compound questions are where errors appear, so put most of your test cases there.

• Feed it clean data. A knowledge graph chatbot is only as accurate as its sources. Incomplete meeting notes produce incomplete answers.

When should you choose RAG, GraphRAG or a hybrid?

• Start with standard RAG if your questions are mostly lookups in manuals, policies or FAQs.

• Choose GraphRAG if answers depend on connections between entities spread across documents and systems.

• Use a hybrid in most enterprise cases: vector search for passages, the graph for relationships and filtering.

• Measure before you decide. Build an evaluation set of real user questions and compare accuracy.

For budgets and timelines, see how much an AI agent project costs.

How RUBICON helps with knowledge graph chatbots

We built the two-layer GraphRAG proof of concept described above on Neo4j, and we deliver RAG and agent solutions on Azure and Databricks. We’re a Microsoft Solutions Partner for Cloud & AI Platforms, with about 55 people, 40+ of them engineers, in Sarajevo.

See our AI agents and AI and machine learning services. If you’re deciding between approaches, we can test RAG and GraphRAG against your real questions with you.

Frequently asked questions