Teams often reach for fine-tuning when they need retrieval, or build a retrieval pipeline when the whole corpus fits in the prompt. This guide compares RAG, fine-tuning and long context on freshness, cost, latency, citations and effort, with a decision table and examples.
A customer saw a snippet from another company's document in an AI answer. A new 'related questions' feature queried the vector store through a helper that did not apply the tenant filter. We moved tenant scoping into the repository layer so no code path can skip it.
Our support assistant quoted an old refund policy that had been removed from the help center weeks earlier. The ingestion pipeline only ever upserted chunks and never deleted them. We added source-of-truth sync, document versions and a nightly reconciliation job.
Our internal assistant answered questions about supplier contracts and price lists, and its answers about prices were confidently wrong. Plain text extraction had flattened PDF tables into run-on text. How we found it, switched to layout-aware parsing with tables as chunks, and measured the fix.
Our docs assistant answered v4 questions with v3 settings, fluently and with citations. Fixed-size chunking split tables from their headings, chunks carried no version metadata and we had no retrieval eval. How we rebuilt chunking, added filters and hybrid search, and measured it.
Tenants upload terabytes of audio, video, PDFs and images, and expect an LLM to answer questions over all of it, with their data isolated and a bill that matches what they used. Here is the AWS design we shipped: ingestion pipeline, isolation layers and metering.
Our RAG prototype was one folder where the PDF loader, the vector client and the API route imported each other freely. Here is the layout we moved to: four packages with enforced boundaries, a shared contract module, versioned indexes and an eval gate in CI.
Small tenants in our RAG product got 2 or 3 chunks back instead of 10, and sometimes none. HNSW finds ef_search candidates first and applies the tenant filter afterwards. Iterative index scans in pgvector 0.8 plus partial indexes for big tenants fixed recall.
Semantic search in our docs assistant took 1.9 seconds and returned odd rankings. The query sorted by L2 distance while the HNSW index was built for cosine, so Postgres scanned every row. Matching the operator to the opclass brought it to 14 ms.
Our nightly embedding backfill died on the same few documents every night. A handful of chunks were far over the embedding model's token limit because the chunker split by characters and some files were minified JSON. Token-aware chunking and batching fixed it.