A production RAG repo layout: separating ingestion, retrieval, serving and evals

Every RAG prototype we have seen starts the same way: a rag/ folder with a loader, a chunker, a vector store wrapper, a prompt file and a FastAPI route, all importing each other. It works until the second engineer joins. Then a change to the chunker silently changes what the API returns, a re-index runs inside a web worker, and nobody can say which index version answered a customer's question last Tuesday.

The fix is not a framework. It is a repository layout that matches how the system actually runs: an offline pipeline that builds indexes, an online service that reads them, and an evaluation harness that judges both. This post shows the layout we use, the rules that keep it honest, and the CI wiring that turns it into a guard rather than a diagram.

Two pipelines, one contract

A RAG system has two very different runtimes:

• Offline ingestion loads documents, cleans them, chunks them, embeds them and writes an index. It is batch work, can take hours, and needs credentials for source systems.
• Online serving receives a question, retrieves chunks, builds a prompt, calls a model and returns an answer. It must be fast, stateless and safe to scale horizontally.

They meet in exactly one place: the index and the shape of what is stored in it. If that shape is defined in one module and both sides depend on it, the two pipelines can evolve independently. If each side defines its own idea of a chunk, they drift.

The layout

rag-platform/
  pyproject.toml
  Makefile
  .importlinter
  configs/
    ingestion.yaml
    serving.yaml
  src/
    ragcore/            # shared contracts, no I/O
      schemas.py
      index_manifest.py
    ingestion/          # offline: load, clean, chunk, embed, publish
      sources/
      chunking.py
      embedding.py
      publish.py
      cli.py
    retrieval/          # query-time search, filters, reranking
      search.py
      rerank.py
    serving/            # HTTP API, prompt assembly, model client
      api.py
      prompts/
      llm_client.py
    evals/              # datasets, metrics, runners
      datasets/
      metrics.py
      run.py
  tests/
    unit/
    integration/
  docker/
    ingestion.Dockerfile
    serving.Dockerfile

Each top-level package has a single reason to change:

retrieval sits between the two pipelines on purpose. Ingestion uses it in its smoke tests to confirm a freshly built index answers known queries; serving uses it for every request. Neither needs the other.

The shared contract

ragcore/schemas.py is small and boring, which is the point:

# src/ragcore/schemas.py
from datetime import datetime
from pydantic import BaseModel, Field

SCHEMA_VERSION = 3

class Chunk(BaseModel):
    chunk_id: str
    doc_id: str
    text: str
    source_uri: str
    section: str | None = None
    tenant_id: str
    updated_at: datetime
    token_count: int = Field(ge=1)

class RetrievedChunk(BaseModel):
    chunk: Chunk
    score: float
    retriever: str  # "dense", "bm25", "hybrid"

Any change to these models bumps SCHEMA_VERSION. Ingestion writes it into the index; serving refuses to start against an index with a different major version. That one check has saved us from two deploys where the API expected a field the live index did not have.

Versioned indexes and a manifest

Ingestion never overwrites the live index. It builds a new one under a versioned name and publishes a manifest describing exactly how it was made:

# src/ragcore/index_manifest.py
from pydantic import BaseModel

class IndexManifest(BaseModel):
    index_name: str            # e.g. "kb_v2025_03_14_1"
    schema_version: int
    embedding_model: str
    embedding_dim: int
    chunker: str               # e.g. "markdown_headings"
    chunk_size_tokens: int
    chunk_overlap_tokens: int
    source_snapshot: str       # commit or export id of the corpus
    doc_count: int
    chunk_count: int
    created_at: str

The serving config points at a manifest, not at an index name, and every answer logs the index_name it used. Promotion is a config change that can be rolled back, and when someone asks why an answer changed we can diff two manifests instead of guessing. The embedding model and dimension in the manifest also stop the classic mismatch where queries are embedded with a different model than the documents.

Enforcing the boundaries

Folder names are suggestions until a tool fails the build. We use import-linter, which checks import graphs against declared contracts:

# .importlinter
[importlinter]
root_packages =
    ragcore
    ingestion
    retrieval
    serving
    evals

[importlinter:contract:core-is-leaf]
name = ragcore imports nothing from the repo
type = forbidden
source_modules = ragcore
forbidden_modules =
    ingestion
    retrieval
    serving
    evals

[importlinter:contract:serving-no-ingestion]
name = serving never imports ingestion or evals
type = forbidden
source_modules = serving
forbidden_modules =
    ingestion
    evals

[importlinter:contract:retrieval-no-http]
name = retrieval knows nothing about serving
type = forbidden
source_modules = retrieval
forbidden_modules =
    serving
    ingestion

uv run lint-imports

The first run on our old code reported eleven violations, including the API importing the PDF loader "just for a helper function". Each one was a real coupling bug waiting to happen.

Separate images, separate dependencies

The two Dockerfiles install different extras from pyproject.toml:

[project.optional-dependencies]
ingestion = ["pypdf", "beautifulsoup4", "tiktoken"]
serving = ["fastapi", "uvicorn[standard]"]
evals = ["pandas"]

The serving image does not contain PDF parsers or source-system credentials, which shrinks it and removes a class of attack surface from the internet-facing service. The ingestion image runs as a scheduled job with read access to sources and write access to the index store, and nothing else.

Evals as a first-class package

evals/ holds versioned golden datasets (question, expected source documents, reference answer notes) and runners that score two things separately:

• Retrieval quality: did the expected documents appear in the top k? We track recall at 5 and mean reciprocal rank.
• Answer quality: is the answer grounded in the retrieved text and does it address the question? Scored by rubric, partly automated, partly reviewed.

Splitting them matters: when answer quality drops, the first question is whether retrieval changed. The runner takes a manifest and a serving config, so we can compare a candidate index against the live one before promoting it:

uv run python -m evals.run --manifest manifests/kb_v2025_03_14_1.json --baseline manifests/live.json --fail-under-recall 0.85

Wiring it into CI

.PHONY: lint test evals-smoke
lint:
	uv run ruff check src tests
	uv run lint-imports

test:
	uv run pytest tests/unit -q

evals-smoke:
	uv run python -m evals.run --dataset smoke --manifest manifests/live.json --fail-under-recall 0.80

Every pull request runs lint and test. Changes under ingestion/, retrieval/ or serving/prompts/ also run evals-smoke against a 50-question set, which takes about two minutes. The full eval set runs nightly and before any index promotion.

Common mistakes

Key takeaways

• Lay out the repo around runtimes: offline ingestion, online serving, and the retrieval code both share.
• Keep the contract between them in a tiny, dependency-free module with an explicit schema version.
• Publish indexes as immutable versions with a manifest, and log which index answered each request.
• Enforce package boundaries with import contracts so the layout survives deadlines.
• Treat evals as code that runs in CI, scoring retrieval and answers separately.

References

• import-linter documentation: https://import-linter.readthedocs.io/
• Pydantic documentation: https://docs.pydantic.dev/
• FastAPI, bigger applications with multiple files: https://fastapi.tiangolo.com/tutorial/bigger-applications/
• Python Packaging User Guide, pyproject.toml: https://packaging.python.org/en/latest/guides/writing-pyproject-toml/