ai//3 min read
Answering questions from your own documents: a practical guide to RAG
Retrieval-augmented generation lets a model answer from your documents instead of its memory. The model is the easy part. Chunking, search quality, citations and evaluation decide whether people trust the answers.
In short
- RAG quality is mostly retrieval quality: if the right passage is not found, the model cannot use it.
- Chunk by meaning (sections and paragraphs), keep titles with each chunk, and store where each one came from.
- Combine keyword and vector search; each finds things the other misses.
- Always show sources, and let the model say it does not know.
Language models know a lot about the world and nothing about your company's handbook, product docs or contracts. Retrieval-augmented generation (RAG) closes that gap: search your documents for passages relevant to the question, put them in the prompt, and ask the model to answer only from them.
The pipeline
- Ingest: pull documents from their sources and convert them to clean text.
- Chunk: split them into passages small enough to retrieve precisely.
- Embed and index: turn each chunk into a vector, and also index its text for keyword search.
- Retrieve: find the best chunks for a question.
- Generate: answer from those chunks, with citations.
Chunking decides more than you think
Split along the document's own structure: headings, sections, paragraphs. Fixed-size windows cut sentences and tables in half. A few hundred words per chunk with a small overlap is a reasonable start; tune it with your evaluation set.
Store each chunk with its document title, section heading and a link to the source. A chunk that says “this applies to all contractors” is useless if the model cannot tell which policy it came from.
Storing vectors in Postgres
If you already run Postgres, the pgvector extension is usually enough for millions of chunks, and it keeps documents, permissions and vectors in one place.
create extension if not exists vector;
create table doc_chunks (
id bigserial primary key,
document_id bigint not null references documents(id) on delete cascade,
heading text,
content text not null,
embedding vector(1536) not null,
search tsvector generated always as (to_tsvector('english', content)) stored
);
create index on doc_chunks using hnsw (embedding vector_cosine_ops);
create index on doc_chunks using gin (search);Set the vector size to match your embedding model's output. If you change embedding models, re-embed everything: vectors from different models are not comparable.
Hybrid search beats either half
Vector search finds passages with similar meaning even when the words differ. Keyword search finds exact terms that vectors blur: product codes, names, error messages. Run both and merge the results; reciprocal rank fusion is a simple, effective way to combine the two lists.
-- Nearest chunks by meaning (<=> is cosine distance in pgvector).
select id, heading, content
from doc_chunks
order by embedding <=> $1
limit 20;
-- Best chunks by keywords.
select id, heading, content
from doc_chunks
where search @@ websearch_to_tsquery('english', $2)
order by ts_rank(search, websearch_to_tsquery('english', $2)) desc
limit 20;Respect permissions
Filter chunks by what the asking user may read before they reach the model. A model cannot keep a secret it has been shown. Store access rules with documents and apply them in the retrieval query.
Prompting for honest answers
- Tell the model to answer only from the provided passages.
- Tell it to say plainly when the passages do not contain the answer.
- Ask for citations by passage number, and render them as links to the source.
- Keep the passages in a clearly delimited block, separate from the instructions.
Evaluate retrieval and answers separately
Question | Measure |
|---|---|
Did we find the right passage? | Share of test questions where it appears in the top results |
Is the answer grounded? | Share of claims supported by the cited passages |
Does it admit ignorance? | Correct refusals on questions the documents cannot answer |
When a RAG system gives a bad answer, look at what it retrieved before you touch the prompt.
Launch checklist
- To do: Documents chunked by structure, with titles and source links
- To do: Hybrid keyword and vector search
- To do: Permissions applied before retrieval results reach the model
- To do: Answers cite sources and can say “I don't know”
- To do: An evaluation set covering retrieval and answers
- To do: A way to re-index when documents change
