Reading supplier invoices with an LLM, from photo to structured lines

A restaurant receives supplier invoices as paper at the delivery door, as PDFs by email, and as photographs taken in a hurry with the box still on the floor. The job is to turn each one into lines a system can use: product, quantity, unit, unit price, line total, and the tax and discount rules that connect them. We built this pipeline for bilz.ai, and the model turned out to be the smallest part of it. This is the shape that held up.

Capture decides most of the accuracy

The worst invoices never recover from the photograph. A phone held at an angle in a dark corridor produces an image where the price column is readable and the quantity column is a smear. No model fixes that afterwards, so the capture step does the work: an outline that shows where the page should sit, a sharpness check before the shot is accepted, and a prompt to retake when the page is cut off. Taking the photo again costs ten seconds. A wrong quantity costs a wrong stock count for a week.

PDFs have their own problems. Some are born digital and carry a text layer. Some are scans wrapped in a PDF container with no text at all. Some have a text layer that is wrong, produced by the supplier’s own OCR, with digits swapped and columns merged. We treat the text layer as a hint and check it against the image, and we look for it before deciding how to read the page.

Pages

A multi-page invoice is one document with a header on the first page, line items across several and totals on the last, and the model has to see it that way. Split the pages apart and the totals never reconcile with the lines. Stitch everything into one image and the resolution drops below what the digits need.

We render each page at a resolution where the smallest font is legible, send the pages together with their page numbers, and ask for lines tagged with the page they came from. Pages that are a receipt stapled to an invoice, or a delivery note that repeats the lines without prices, get classified first so the extraction does not count the same delivery twice.

Vision model or OCR plus text model

Two architectures work. The first sends the page image to a vision-capable model and asks for structured output directly. The second runs OCR, hands the text with layout coordinates to a text model, and lets it rebuild the table.

Vision models read tables better than OCR did two years ago, and they cope with stamps, handwriting in the margin and the crease across the page. They are slower and dearer per page, and when they get a digit wrong there is nothing to point at. OCR with a text model is faster and cheaper, and it gives every token a bounding box, which matters for the citation step below. It is worse on skewed photographs and on tables without ruling lines.

We run OCR for the coordinates and a vision model for the reading, and we give the vision model the OCR text alongside the image. The two disagree on a few per cent of fields, and the disagreements are where we look first.

The schema is the specification

The extraction schema is the one document everyone on the project reads. For an invoice it is the supplier, the invoice number and date, the currency, each line with the product text as printed, quantity, unit, unit price, line total and VAT rate, and the document totals. Every field has a type, and the type is strict: a quantity is a number, a unit is one of a short list, a date is ISO. If the model returns a unit the list does not contain, the field fails validation and we see it, rather than discovering a “crate” six weeks later in a cost report.

We keep the product text exactly as printed. Matching it to an inventory item is a separate step with its own article, matching invoice lines to inventory items. Mixing extraction and matching in one prompt makes both worse and makes neither measurable.

Every value cites its region

Each value comes with a citation: the page number and a bounding box in the page’s coordinates. The person checking the invoice taps a number and the app highlights where it came from. That is what makes a review take seconds.

The citation also catches the model’s favourite failure, which is to fill a field with something plausible when the page does not contain it. A value whose citation points at empty space, or at a region whose OCR text does not contain the digits, is dropped and marked missing. Missing is a fine answer. A confident wrong unit price is the one that costs money, because it flows straight into recipe cost.

We ask for a confidence per field as well, with the instruction that uncertainty should be reported rather than resolved. Models are poorly calibrated, so we read the number as a ranking within one document and never as a probability. It is good enough to put the shaky fields at the top of the review queue.

Credit notes, discounts and deposits

A credit note looks like an invoice with the sign flipped, and the sign is often a word in the header rather than a minus in the column. We classify the document type before extracting lines. A credit note’s quantities enter stock as negative and its totals reduce what is owed. Getting this wrong doubles a delivery instead of cancelling it.

Discounts arrive in every form suppliers can invent: a percentage on the line, a separate negative line, a net price printed next to a gross one, a total discount at the bottom with no allocation. The schema has a place for each, and a reconciliation step checks that the lines, the discounts and the VAT add up to the printed total within rounding. When they do not, the document goes to review instead of into the books. The total is the one number a supplier checks carefully, so it is the best anchor we have.

Deposits on crates and kegs, delivery charges and returned goods are lines in the table that are not ingredients. They are extracted and typed so they can be kept out of food cost without being lost from the ledger.

Measuring accuracy per field

An extraction pipeline is only as good as its number, and the number has to be per field. A document-level score hides that supplier names are right every time and unit prices are wrong one time in twenty. We keep a set of real invoices with lines a bookkeeper has verified, and every change to the prompt, the model or the page rendering runs against it before it ships. How we build and keep that set is its own article, building an evaluation set from past cases.

The per-field numbers decide what to work on. Quantities fail on handwritten amendments. Unit prices fail on invoices that print a price per kilogram next to a quantity in pieces. Dates fail on suppliers who use two formats on one page. Each of those is a fix in capture, schema or prompt, and the set tells us which fix paid off.

The size of the review queue is measured too. If it grows, either the suppliers changed or the model did, and both show up first as a drop in one field.

If you have invoices, delivery notes or any other documents that someone is typing into a system, the AI agents and automation page describes how we take on that work, and you can write to us with a sample of the documents.