Our RAG assistant quoted the wrong numbers from PDF tables: extraction flattened rows and columns

Our procurement team used an internal assistant to ask questions about supplier contracts: "What is the delivery fee for orders under 500 units from Supplier B?", "What discount applies above 10,000 units?". The assistant used retrieval-augmented generation (RAG) over about 900 PDF contracts and price lists. For questions about clauses and terms, it was good. For questions about numbers in tables, it was confidently wrong often enough that the team stopped trusting it.

A typical failure:

Q: What is the delivery fee for 100-499 units from Supplier B (2025 price list)?
A: The delivery fee for 100-499 units is EUR 45.00 per order. [source: supplier-b-pricelist-2025.pdf, p.3]

The real fee in that table row was EUR 120.00. EUR 45.00 was the fee for a different tier.

Looking at what the model saw

We logged the retrieved chunks for the question. The chunk from page 3:

Delivery fees Quantity (units) Standard Express 1-99 EUR 45.00 EUR 90.00
100-499 EUR 120.00 EUR 180.00
500-1999 EUR 0.00 EUR 60.00 Volume discount Quantity Discount 2000-9999 3% 10000+ 5% Prices exclude VAT

The table had been extracted as one long line of text. Columns and rows were gone. To the model, 100-499 appeared next to EUR 45.00 from the previous row as much as next to EUR 120.00. Sometimes it guessed right, sometimes not.

Checking the extraction pipeline

The ingestion used a basic PDF text extractor that returns text in reading order, followed by a fixed-size splitter of 1,000 characters with 100 characters of overlap. For prose, that works. For tables:

• cell boundaries are lost, values from adjacent cells are joined with spaces;
• multi-line cells are split across lines in the wrong order in some PDFs;
• the splitter can cut a table in half, separating headers from values.

What we ruled out

• Retrieval missing the right page. The correct page was retrieved in 92% of failing cases; the content was there but scrambled.
• The model's arithmetic or reasoning. Given the same table as markdown, the same model answered all test questions correctly.
• Embedding model quality. Retrieval quality was not the bottleneck for these questions.

Root cause

Plain text extraction destroyed the table structure before anything else happened. Each table became an unstructured sequence of numbers and labels, and the model had to infer which value belonged to which row and column. Chunking sometimes separated headers from data as well. The model's errors were a reasonable response to ambiguous input.

Why PDFs are hard

PDF is a page description format: it records where to draw characters, not that they form a table. A table in a PDF is often just text positioned in a grid, sometimes with lines drawn around it. Recovering rows and columns requires layout analysis: grouping text by position, detecting ruling lines, inferring column boundaries. Simple extractors skip that.

What the model needs

LLMs read tables well when the structure is explicit, for example as markdown or HTML with one row per line and headers repeated or clearly attached. They read badly when they must reconstruct structure from token order.

Who is affected

Any RAG system over PDFs with tables: financial statements, price lists, specifications, invoices, lab results, insurance schedules. Prose-heavy documents hide the problem, so systems often pass early tests and fail on the documents users care most about.

Environment

Step 1: layout-aware parsing

We switched ingestion to a layout-aware document parser that detects tables and outputs them as markdown (several open-source and hosted options exist; we evaluated three on 50 of our own documents and chose by table accuracy on those). The same page now produced:

### Delivery fees

| Quantity (units) | Standard | Express |
|---|---|---|
| 1-99 | EUR 45.00 | EUR 90.00 |
| 100-499 | EUR 120.00 | EUR 180.00 |
| 500-1999 | EUR 0.00 | EUR 60.00 |

Step 2: tables as their own chunks

The splitter now treats tables as atomic units: a table is never split, and it is stored as one chunk with context attached:

def table_chunk(doc, page, table_md, heading):
    return {
        "text": f"Document: {doc.title}\nSection: {heading}\nPage: {page}\n\n{table_md}",
        "metadata": {"doc_id": doc.id, "page": page, "type": "table", "supplier": doc.supplier,
                     "valid_from": doc.valid_from},
    }

Including the document title, section heading and validity date in the chunk text means retrieval finds the right table for questions like "Supplier B 2025 delivery fee", and the model knows which document a table came from. Very large tables are split by rows with the header row repeated in each part.

Step 3: metadata filters

Price lists change every year. We extract supplier and valid_from at ingestion and filter retrieval by them when the question names a supplier or year, so the 2023 table does not compete with the 2025 one.

Step 4: answer with the row

The generation prompt now asks the model to quote the table row it used, which makes answers checkable:

When answering from a table, include the exact row (as markdown) that contains the answer.
If no row matches the question exactly, say so.

Re-ingesting the corpus

Changing the parser meant re-ingesting all 900 documents. We built the new index next to the old one, ran the evaluation set against both, and switched the assistant's retrieval to the new index only when it scored better. The old index stayed available for a week as a fallback, then was deleted.

Alternatives and trade-offs

• Vision models reading page images handle complex layouts well, at higher cost per page; we use them as a fallback for pages where the parser reports low confidence.
• Extract tables into a database and answer numeric questions with SQL. More precise for highly structured data like price lists; we are piloting it for that document type.

How to test the fix

We built an evaluation set of 120 questions with known answers taken from tables, across 40 documents, and score exact match of the numeric value and correctness of the cited page:

def score(answer, expected):
    nums = re.findall(r"\d[\d.,]*", answer)
    return any(normalize(n) == normalize(expected["value"]) for n in nums) and expected["page"] in answer

Confirming in production

On the evaluation set, numeric accuracy went from 61% to 94%. Most remaining errors came from scanned documents with poor image quality. The procurement team's feedback button showed "wrong number" reports dropping from about 20 per week to two or three.

Regression guard

• The table question set runs on every change to parsing, chunking, retrieval or prompts; a drop of more than two points blocks the release.
• New document types get ten evaluation questions before being added to the corpus.

Security implications

Contracts and price lists are confidential. The parser runs inside our environment rather than as an external service for these documents, and the vector store enforces the same access groups as the document system, so retrieval never returns chunks a user could not open directly.

Prevention checklist

• Inspect the text your chunks actually contain for table-heavy documents.
• Use layout-aware parsing that preserves tables as markdown or HTML.
• Keep tables whole, with document and section context, in their own chunks.
• Filter by metadata such as supplier and validity date.
• Evaluate numeric answers explicitly, not only prose quality.

References

• Docling documentation: https://docling-project.github.io/docling/
• Unstructured, partitioning PDFs and tables: https://docs.unstructured.io/open-source/core-functionality/partitioning
• Anthropic, Contextual retrieval: https://www.anthropic.com/news/contextual-retrieval
• OpenAI cookbook, Parsing PDFs for RAG: https://cookbook.openai.com/examples/parse_pdf_docs_for_rag