Invoice Extraction Is Easy. Trusting the Numbers Is the Hard Part.

Pulling fields off an invoice is a solved problem. Getting a finance team to act on those fields without re-checking them by hand is not, and the gap between the two is where most automation projects quietly die.

Why 95% field accuracy still fails a finance team

Vendors love to quote a single number: 95% extraction accuracy, sometimes 98%. It sounds like a passing grade. It is not, and the reason is arithmetic, not skepticism. That figure is almost always measured per field. A typical invoice you want to touch has a dozen fields that matter: vendor name, invoice number, date, PO number, subtotal, tax, total, and a handful of line items with quantities and unit prices. If each field is correct 95% of the time and the errors are roughly independent, the chance that an entire invoice is captured with zero mistakes is 0.95 raised to the twelfth power. That is about 54%. Put plainly, a 95% system hands you a document with at least one wrong field close to half the time.

Finance does not care about your average. Accounts payable is not a recommendation engine where a slightly-off suggestion is fine. A transposed digit in a total becomes a payment, a payment becomes a reconciliation headache, and enough of those become an auditor asking why the controls did not catch it. The cost of a wrong field is asymmetric and it lands downstream, weeks later, on someone who never saw your accuracy chart. That asymmetry is the whole game, and it is why invoice data extraction that stops at "the model read the numbers" never earns real trust.

Three layers, not one model

The useful mental model for automated invoice processing is three distinct layers, each with its own failure mode, rather than one clever model that does everything.

The first layer is OCR: turning pixels into characters. This is genuinely close to solved for clean digital PDFs and merely good for scans. It fails on faxed thermal-paper receipts, stamps that overlap totals, and 200-DPI phone photos taken at an angle. The second layer is field extraction: deciding which characters are the invoice number and which are the total. This is where large language models earn their keep, because they read layout and context the way a person does instead of relying on brittle positional templates. The third layer, the one almost everyone skips, is reconciliation: checking whether the extracted numbers are internally consistent before anyone trusts them.

The trap is treating extraction as the finish line. A model returns a total of $4,812.00 with high confidence and a clean layout, and it looks done. It is not done until you have asked the one question the model cannot answer about itself: do the parts add up to the whole?

Math validation as a hard gate

This is the differentiator, and it is embarrassingly simple to state: extracted numbers must reconcile, and anything that does not reconcile does not pass. Line items times their quantities must sum to the subtotal. Subtotal times the tax rate must equal the tax. Subtotal plus tax must equal the invoice total. These are not heuristics or model outputs. They are ground truth that the document carries within itself, and they let you audit the model with the model's own answers.

def reconciles(inv, cents=0.01):
    line_sum = sum(li.qty * li.unit_price for li in inv.line_items)
    subtotal_ok = abs(line_sum - inv.subtotal) <= cents
    tax_ok      = abs(inv.subtotal * inv.tax_rate - inv.tax) <= cents
    total_ok    = abs(inv.subtotal + inv.tax - inv.total) <= cents
    return subtotal_ok and tax_ok and total_ok

The penny tolerance matters more than it looks. Real invoices round each line independently, so a strict equality check will reject perfectly valid documents and flood your reviewers with noise. One cent of slack per aggregate absorbs honest rounding without letting a real transposition through, because a real error is off by dollars, not pennies. When I built Ledger, this gate was the feature. The extraction was the commodity; the promise that a passed invoice is arithmetically sound is what a controller actually pays for.

The model can tell you what it read. Only the math can tell you whether what it read is a coherent invoice, and that difference is the entire product.

There is a deeper point here. Reconciliation catches errors the model is confident about. A language model will happily report a wrong subtotal with 0.98 confidence because it has no idea it misread a line item three rows up. Confidence measures the model's certainty about its own reading, not the reading's correctness. Math is the independent second opinion, and independence is exactly what makes it trustworthy.

Routing on confidence and reconciliation together

Once you have both a confidence signal and a reconciliation result, routing becomes a clean three-way decision. If the invoice reconciles to the penny and every field cleared its confidence threshold, auto-approve it and let it flow to the ledger untouched. If it reconciles but a field or two came back shaky, flag it for a light-touch glance where the reviewer confirms rather than re-keys. If it does not reconcile at all, escalate it as an exception with the specific broken equation attached, so the human opens the document already knowing the tax line is $12 short instead of hunting for the problem.

The two signals do different jobs and you need both. Confidence guards individual fields; reconciliation guards the document as a whole. An invoice can have high confidence on every field and still fail to reconcile because two of those confident fields contradict each other. Routing on confidence alone would wave that document straight through. That is the failure mode most AP automation demos never show you, because it only appears at volume.

The world is messier than the demo

Every clean pipeline meets reality and reality wins. Scanned documents arrive skewed, stapled, and stamped. Multi-currency invoices carry a total in EUR and a converted figure in USD, and if your reconciliation does not know which currency each field is in, it will "correctly" flag a perfectly valid invoice as broken. Some vendors put tax inside the line items; some add it once at the bottom; some are in jurisdictions with two tax lines that must be summed separately. A template you have never seen shows up every week.

Document automation that survives contact with this mess does so by being explicit about assumptions, not by pretending they do not exist. Currency is a field you extract and carry through every calculation, never a default you assume. Tax handling is configured per vendor once you have seen a few of their invoices, because vendors are remarkably consistent with themselves even when they differ wildly from each other. And when a document is genuinely unreadable, the honest move is to route it to a person immediately rather than emit a confident guess. Knowing when not to automate a document is a feature. A system that refuses cleanly on 8% of inputs is worth far more than one that silently fabricates numbers on that same 8%.

The human loop is supposed to shrink

Human-in-the-loop is often sold as a permanent safety net, which quietly concedes that the automation never really works. Done right, the loop is a teacher and it is meant to get smaller. Every exception a reviewer resolves is a labeled example: this vendor formats dates day-first, this template puts freight below the tax line, this scanner drops the leading digit on faded totals. Those corrections feed back as vendor-specific rules and better prompts, and the share of documents needing a human hand falls over time.

What makes this loop trustworthy is that trust is earned per vendor, not granted globally. You do not flip a switch and auto-approve everything. You auto-approve the twenty vendors whose invoices have reconciled cleanly for three months, watch the exception rate on the rest, and expand the trusted set as the evidence accumulates. The reviewer's job shifts from re-keying every document to handling only the genuinely novel ones, which is both cheaper and far less soul-crushing.

Measure exceptions, not accuracy

Accuracy in the abstract is a vanity metric. The number that runs the operation is exceptions per hundred invoices: how many documents need a human touch. It is honest because it maps directly to labor cost, and it is legible because a finance lead knows exactly what "six per hundred" means for their team's Tuesday.

On one Ledger deployment the exception rate opened at 22 per hundred, mostly from three vendors with genuinely awful scan quality and one currency-handling bug that flagged valid euro invoices. After a month of feeding corrections back and fixing that bug, it settled near 6 per hundred, and most of those six were real problems worth a human's attention: duplicate invoice numbers, totals that did not match the purchase order, a vendor who had changed their bank details mid-quarter. That is the shape of success. Not a bigger accuracy number, but a small and steadily shrinking pile of exceptions, each one a document a person genuinely should see.

Extracting the numbers was never the hard part. Earning the right to skip the human check is, and you earn it with math that reconciles, routing that respects its own uncertainty, and a loop that gets quieter every month.