What Is RAG? Retrieval-Augmented Generation for Developers
10 min read
Retrieval-augmented generation (RAG) is a way to make a large language model answer from your own data. Before the model writes anything, your system searches a knowledge base for the passages most relevant to the question and adds them to the prompt. The answer is then grounded in those sources rather than in whatever the model memorised during training.
The idea comes from a 2020 paper by Patrick Lewis and colleagues at Facebook AI Research, "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks". Today it is the default way to build chatbots over company documents, support assistants and "chat with your PDF" features, and a regular topic in interviews for developers who work with LLMs.
Key takeaways
- RAG = search first, then generate. The model answers from passages your system retrieves at question time.
- It fixes three LLM weaknesses: no knowledge of your private data, a training cutoff, and confident made-up answers.
- A RAG pipeline has two halves: indexing (chunk, embed, store) ahead of time, and retrieval plus generation when a question arrives.
- Answer quality depends mostly on retrieval. If the right passage isn't found, no prompt can save the answer.
- Choose RAG to give a model knowledge. Choose fine-tuning to change its behaviour or style.
Why do LLMs need RAG?
A language model only knows what was in its training data, frozen at a cutoff date. That causes three practical problems:
- It has never seen your data. Your product docs, internal wiki and last week's tickets were never part of its training.
- It goes out of date. Anything that changed after the cutoff is invisible to it.
- It fills gaps with fluent guesses. Asked something it doesn't know, a model often produces a plausible answer instead of saying so.
RAG addresses all three without retraining anything. Update a document, re-index it, and the next answer reflects the change. Because you know which passages went into the prompt, you can also show the sources, which lets users check the answer instead of trusting it.
How does RAG work, step by step?
A RAG system has two halves.
Indexing, done ahead of time:
- Load your documents: pages, PDFs, tickets, database rows.
- Chunk them into passages of a few hundred words, often with some overlap so a sentence cut at a boundary still appears whole in one chunk.
- Embed each chunk: an embedding model turns the text into a vector of numbers, so passages with similar meaning end up close together.
- Store the vectors in a vector index, with the original text and metadata such as the source and date.
Retrieval and generation, for every question:
- Embed the question with the same embedding model.
- Retrieve the k chunks whose vectors are closest to it, usually by cosine similarity.
- Rerank (optional): a slower, more accurate model reorders those candidates.
- Build the prompt: the instructions, the retrieved chunks and the question.
- Generate: the LLM writes an answer from that context, ideally citing which chunks it used.
Build a tiny RAG pipeline in Python
The clearest way to understand RAG is to build the retrieval half yourself. The script below has no dependencies. It uses TF-IDF vectors (a classic keyword-weighting scheme) where a production system would call an embedding model, and it stops at the prompt, because sending that prompt to an LLM is a single API call.
import math
import re
from collections import Counter
# A tiny "knowledge base" for a made-up product. In a real system these would
# be chunks cut from your docs, tickets or wiki pages.
DOCS = {
"backups": "AcmeCloud takes a backup of every database every 6 hours and keeps each backup for 30 days.",
"regions": "AcmeCloud servers run in Mumbai and Singapore. You pick the region when you create a project.",
"billing": "AcmeCloud bills per hour of compute. Stopped servers are not billed, but their disks are.",
"restore": "To restore a database, open the project, choose a backup by date and click Restore.",
}
STOP_WORDS = {
"a", "an", "and", "are", "but", "by", "can", "do", "does", "for", "how", "i",
"in", "is", "it", "my", "of", "on", "the", "their", "to", "what", "you", "your",
}
def tokenize(text):
words = re.findall(r"[a-z0-9]+", text.lower())
# Drop filler words and fold simple plurals ("backups" -> "backup").
return [w[:-1] if len(w) > 3 and w.endswith("s") else w for w in words if w not in STOP_WORDS]
# Indexing (done once, ahead of time): turn every document into a vector.
# TF-IDF stands in for an embedding model so this runs with no dependencies.
doc_tokens = {doc_id: tokenize(text) for doc_id, text in DOCS.items()}
doc_freq = Counter(term for tokens in doc_tokens.values() for term in set(tokens))
N = len(DOCS)
def vectorize(tokens):
counts = Counter(tokens)
return {t: c * (math.log((1 + N) / (1 + doc_freq.get(t, 0))) + 1) for t, c in counts.items()}
def cosine(a, b):
dot = sum(weight * b.get(term, 0.0) for term, weight in a.items())
norm = math.sqrt(sum(v * v for v in a.values())) * math.sqrt(sum(v * v for v in b.values()))
return dot / norm if norm else 0.0
INDEX = {doc_id: vectorize(tokens) for doc_id, tokens in doc_tokens.items()}
# Query time: retrieve the closest chunks, then build a grounded prompt.
def retrieve(question, k=2):
q = vectorize(tokenize(question))
scored = sorted(((cosine(q, vec), doc_id) for doc_id, vec in INDEX.items()), reverse=True)
return [(doc_id, round(score, 3)) for score, doc_id in scored[:k]]
def build_prompt(question, hits):
context = "\n".join(f"[{doc_id}] {DOCS[doc_id]}" for doc_id, _ in hits)
return (
"Answer using only the context below. Cite the [source] you used. "
"If the answer is not in the context, say you don't know.\n\n"
f"Context:\n{context}\n\nQuestion: {question}\nAnswer:"
)
question = "How long are database backups kept?"
hits = retrieve(question)
print(hits)
print()
print(build_prompt(question, hits))Output:
[('backups', 0.223), ('restore', 0.178)]
Answer using only the context below. Cite the [source] you used. If the answer is not in the context, say you don't know.
Context:
[backups] AcmeCloud takes a backup of every database every 6 hours and keeps each backup for 30 days.
[restore] To restore a database, open the project, choose a backup by date and click Restore.
Question: How long are database backups kept?
Answer:The retriever found the right passage, and the prompt does three jobs: it restricts the model to the context, asks for a citation, and gives the model a way out when the answer isn't there. That last instruction does more to stop made-up answers than any other line of the prompt.
The same index answers other questions correctly too. "Do I pay for a stopped server?" ranks billing first (0.296), and "Which regions can I run servers in?" ranks regions first (0.421).
Where keyword retrieval breaks, and why embeddings fix it
Now ask "How do I get my data back?". Every document scores 0.0. The answer is plainly in the restore passage, but that passage never uses the words "get", "data" or "back", so a method that matches words finds nothing.
This is exactly what embedding models solve. They place "get my data back" near "restore a database" because the meanings are close, even with no words in common. To upgrade the script, you would replace vectorize with a call to an embedding model and store the vectors in a vector database. The rest of the pipeline stays the same.
Many production systems use both: hybrid search runs keyword search (usually BM25) and vector search side by side and merges the results. Keywords catch exact terms such as error codes and product names; vectors catch paraphrases.
Scoring every document against every question also costs O(N) comparisons, which is fine for four documents and far too slow for ten million. Vector indexes such as FAISS and HNSW-based databases use approximate nearest-neighbour search to skip most of that work. It is the same instinct covered in how to reduce time complexity: stop doing work whose answer you can predict.
RAG vs fine-tuning: which one do you need?
| RAG | Fine-tuning | |
|---|---|---|
| What it changes | What the model knows at answer time | How the model behaves: style, format, task skill |
| Adding new information | Re-index the document; live in minutes | Retrain the model |
| Citing sources | Yes, you know which passages were used | No, knowledge is baked into the weights |
| Cost to start | Low: an embedding model and a vector store | Higher: training data, compute, evaluation |
| Main risk | Retrieving the wrong passage | Overfitting, forgetting, stale knowledge |
| Best for | Docs Q&A, support bots, internal search | A fixed output format, tone or a narrow task |
Use RAG when the problem is "the model doesn't know our information". Use fine-tuning when the problem is "the model doesn't behave the way we need". Teams often end up using both: a fine-tuned model that follows the house format, fed by a RAG pipeline that supplies the facts.
What decides the quality of a RAG system?
Almost every decision is about retrieval, because the model can only use what it is given:
- Chunk size. Too small and a chunk loses its context; too large and it drags in unrelated text and wastes the context window. A few hundred words with some overlap is a common starting point. Then measure.
- Embedding model. It decides what counts as "similar". Test it on your own questions, not just public benchmarks.
- How many chunks (k). More chunks raise the chance the answer is included, but also add noise and cost.
- Metadata filters. Restrict the search by product, date or user permissions before ranking, so one customer never retrieves another customer's documents.
- Reranking. A cross-encoder reranker reads the question and each candidate together, which is slower but noticeably more precise for the final few slots.
- Prompt instructions. Answer only from context, cite sources, and say "I don't know" when the context doesn't contain the answer.
How do you evaluate a RAG system?
Measure the two halves separately, or you won't know which one to fix:
- Retrieval. Build a small "golden set": 50 to 100 real questions, each with the passage that answers it. Then track recall@k (how often the right passage is in the top k) and MRR (how high it ranks).
- Generation. For each answer, check that it is faithful (every claim supported by the retrieved context) and relevant (it actually answers the question). Teams often use a second LLM as a judge for this, and spot-check its verdicts by hand.
Common ways RAG fails
| Failure | Cause | Fix |
|---|---|---|
| Confident wrong answer | The relevant chunk was not retrieved | Improve chunking, try hybrid search, add reranking |
| Half an answer | The answer spans two chunks | Use overlap, larger chunks, or retrieve neighbouring chunks |
| Outdated answer | The index wasn't updated when the document changed | Re-index on change, store dates and prefer recent sources |
| Instructions hijacked | A document contains text like "ignore previous instructions" | Treat retrieved text as data, never as instructions; filter sources |
| Truncated context | Too many chunks for the context window | Lower k, compress or summarise chunks |
RAG questions you might get in an interview
- Walk me through a RAG pipeline from document upload to answer.
- How would you choose chunk size, and how would you know it's wrong?
- Why can keyword search beat vector search on some queries, and what is hybrid search?
- How do you stop a RAG system from answering when the context doesn't contain the answer?
- How would you evaluate retrieval separately from generation?
- When would you fine-tune instead of using RAG?
If you can answer these with the pipeline above in mind, you understand RAG better than most demos explain it. Retrieval also leans on ideas from classic system design, such as caching embeddings and frequent results, covered in caching in system design.
LLMs, prompt design and RAG make up the AI & LLM Engineering module (weeks 16 and 17) of the DSA, System Design & AI Bootcamp, where you ship AI features into real applications with practising engineers.
Frequently asked questions
What does RAG stand for in AI?
RAG stands for retrieval-augmented generation. A system retrieves relevant passages from a knowledge base and adds them to the prompt, so a large language model answers from those sources instead of only from its training data.
Is RAG better than fine-tuning?
They solve different problems. RAG gives a model knowledge it lacks, such as your documents, and is easy to keep up to date. Fine-tuning changes how a model behaves, such as its output format or tone. Many systems use both.
Do you need a vector database for RAG?
Not for small projects. A few thousand chunks can be searched in memory or in a regular database with a vector extension such as pgvector. A dedicated vector database helps once you have millions of chunks or need fast filtered search at scale.
What is chunking in RAG?
Chunking splits documents into smaller passages before they are embedded, so retrieval can return just the relevant part of a long document. Chunk size is a trade-off: small chunks lose context, large ones add noise.
Does RAG stop hallucinations completely?
No. It reduces them by giving the model relevant sources, but the model can still misread the context, and if retrieval returns the wrong passage the answer can still be wrong. Telling the model to answer only from the context, and to say when it doesn't know, helps further.