RAG
Answers grounded in your own documents, not the model's general training.
RAG is the difference between an AI that guesses about your business and one that quotes your own files. It is the single biggest lever against hallucination, because retrieval — not the model — does the heavy lifting. Get chunking and search right and a cheap model with grounding beats an expensive one without it.
You need this when
- Your knowledge is too large or changes too often to paste into every prompt: procedures, contracts, product docs, support history.
- Answers must be traceable — you need to see which document each claim came from before you act on it.
- Staff keep asking the same questions about the same documents, and the answers move as the documents move.
You can skip it when
- A handful of documents that fit comfortably in the context window — paste them straight into the prompt instead.
- The AI must update records or trigger workflows, not explain — that is function calling/MCP territory, not retrieval.
- You have five FAQs and no document sprawl — write one good system prompt and stop there.
The long version — open when you want the depth
What it is — in one coffee-break
RAG — retrieval-augmented generation — is the pattern that makes an AI answer from YOUR information instead of its general training. The flow: your documents get chunked and embedded into a vector store (lessons 2 and 7); when a question arrives, the system retrieves the most relevant chunks and places them in the model's context (lesson 3) with an instruction to answer from those sources. The model stops guessing and starts citing.
It's the single biggest lever against hallucination, and the backbone of every serious "chat with your knowledge base" product. It is not magic: RAG is only as good as its weakest step, and the weakest step is usually not the model.
When you actually need it (and when you don't)
Need it: whenever answers must come from a body of knowledge too large or too changeable to paste into a prompt — your procedures, contracts, product docs, support history. Our SOP answer machine blueprint is a complete self-hosted RAG system for exactly that.
Don't need it: when the knowledge fits in the context window comfortably — a handful of documents can simply ride along in the prompt, and with today's large windows that covers more cases than most vendors admit. And don't confuse it with acting: RAG retrieves and explains; if the AI must update records or trigger workflows, that's function calling/MCP territory — mature systems combine both (retrieve the policy, then execute the refund).
What actually determines quality, per the production guides: chunking strategy first (300–500 token chunks with modest overlap is the reliable start), hybrid search second (keywords + meaning), reranking third (a cheap second pass that reorders the top results and typically improves answers 10–20%). Model choice comes fourth — a truth vendors selling bigger models rarely lead with.
How to recognize good vs bad implementations
RAG (Rg) produces the single most instructive score on our table: NotebookLM at 9.5 — the best pure RAG product money can currently rent, grounded so hard in your sources that it declines to freelance. Gemini scores 9 with managed RAG infrastructure; n8n 8 for build-it-yourself pipelines with full control. At the bottom, Zapier scores 4 — an action platform where retrieval is an afterthought. The three questions that expose weak RAG in any demo: Does it cite which source each claim came from? What does it say when the answer isn't in the sources (good systems say "I don't know")? And can you update a document and see the answer change?
What this costs
Modest, and mostly one-time: embedding a million document chunks costs about $10 on the default model; storage a few euros a month (or €0 self-hosted); the per-question cost is dominated by the LLM call itself since retrieved chunks ride in as input tokens. A fully self-hosted stack — local embeddings, pgvector, local model — runs a small-business knowledge base for the price of electricity, which is precisely how the tools in our blueprints do it.
Where to see it scored
Rg with reasons: NotebookLM (9.5), Gemini (9), n8n (8), Zapier (4). This lesson closes the retrieval chain: Embeddings → Vector store → Context → RAG.
Flashcards
Check yourself
1. Which tool scores highest on RAG (Rg) in the audit?
2. What determines RAG answer quality most?
3. What chunk size is the recommended starting point?
4. Reranking typically improves answers by roughly how much?
5. The AI must look up a refund policy AND issue the refund. What do you need?
Cheat sheet
- Chunk first: 300-500 tokens with modest overlap is the reliable starting point.
- Hybrid search beats pure vector search — keywords catch what meaning blurs.
- Reranking is a cheap second pass; expect roughly 10-20% better answers.
- Model choice matters fourth, after chunking, hybrid search and reranking.
- Demo test: does it cite sources, and say 'I don't know' when it should?
- Embedding a million chunks costs about $10; the LLM call dominates per-question cost.
Who actually does this well
| Best on this element | Score | Why it scored that |
|---|---|---|
| Glean | 9.5 | The product's entire reason for existing: grounding answers in permission-scoped data across 100+ connected sources, with an internal RAGAS-style relevance/completeness/groundednes |
| NotebookLM | 9.5 | The defining consumer RAG product: strict source grounding with citation hover-through, Deep Research source-finding, and Google reports 50% higher answer satisfaction on large sou |
| OpenEvidence | 9.5 | Retrieval-augmented generation done properly. Licensed corpora — NEJM, JAMA and all eleven specialty journals, NCCN, Cochrane, ASCO, ACOG, AUA, 300+ journals plus FDA and CDC — and |
| Gemini / Google | 9 | RAG Engine (serverless mode, metadata filtering, cross-corpus retrieval), Agent Search and grounding with Google Search cover the full RAG spectrum as managed services. |
| Writer | 9 | Knowledge Graph is the platform's signature RAG layer: multi-hop questions, inline citations, handles spreadsheets/PDFs/slides, and scored over 86 on RobustQA in Writer's own bench |
| Fin (Intercom) | 8.5 | The Fin AI Engine is a bespoke three-phase RAG architecture (refine query, generate grounded response, validate accuracy) — grounding in your content is the whole product and the a |
| Gong | 8.5 | The entire product is retrieval over the customer's own calls, emails and CRM. Grounding in first-party data is the architecture, not a feature. |
| Guru | 8.5 | Grounding in the org's own governed knowledge is the whole product: cited, permission-aware answers pulled from Guru Cards plus 100+ connected sources. |
And the other end of the same column:
| Weakest | Score | Why it scored that |
|---|---|---|
| Higgsfield | 3 | Connectors let the agent read Slack/Drive/Notion content as working context, but there's no dedicated grounding feature over a user's own document corpus. |
| Luma AI | 3 | Image-to-video grounds outputs in user-provided images and keyframes, but no document-level RAG or knowledge-base grounding exists. |
| Meta Llama ecosystem | 3 | Meta itself no longer ships a grounding surface — the Llama Stack RAG/file-search layer now lives in the community-run OGX project, so RAG around Llama is expected but entirely sel |
| Portkey | 2.5 | Portkey does not do retrieval. It routes and observes RAG traffic your application builds elsewhere — retrieval latency and document counts show up in the trace, but grounding your |
Scored on 134 of 144 audited tools. Every score links to the full audit and its reasoning.