What actually happens inside a RAG pipeline
Chunk, embed, retrieve, re-rank, generate — and the three places it quietly goes wrong in production.
Retrieval-augmented generation is a search problem wearing an AI costume. You split your documents into chunks, turn each chunk into a vector, store the vectors, and at question time find the chunks nearest to the question and paste them into the prompt.
Because it is a search problem, it fails like one. If the right chunk is never retrieved, no model on earth will answer correctly — it will answer confidently instead, from whatever it did receive.
Instrument retrieval separately from generation. Log the retrieved chunk IDs for every answer. When a user reports a bad answer you need to know whether retrieval missed or the model ignored what it was given, and those have completely different fixes.
The shape of it
- 01
Chunk
Split on structure, not character count. Bad chunking is the most common root cause.
- 02
Embed
Same model for documents and queries, always. Re-embed everything when you change it.
- 03
Retrieve
Hybrid — vector plus keyword. Pure vector search misses exact IDs and product names.
- 04
Re-rank
A cross-encoder over the top 50 beats a bigger top-k every time.
- 05
Generate
Instruct the model to answer only from context and to say when it can't.
Want this applied to your situation?
Sessions are direct and specific — you leave with a decision, not a reading list.
Book a session