Building an evaluation set from past cases, and keeping it alive

Every LLM feature we ship has a number next to it, and the number comes from an evaluation set: real cases from the past with answers the people who do the job have agreed. The agent article says to build it before the agent. This is how we build it, how big it has to be, and how it stays useful after launch.

Choosing the cases

The set is drawn from what the feature will see in production, in the proportions it will see it, plus the cases that would hurt most if they went wrong.

For an invoice reader that means invoices from every supplier the restaurant has, in the mix they arrive in, and a deliberate overweighting of the ugly ones: the photograph with a thumb over the totals, the credit note that looks like an invoice, the supplier who prints the discount on a separate line, the two-page invoice where page two is a delivery note. The ugly cases are a minority in production and a majority of the failures, so a set sampled at random under-represents the thing we most need to measure.

We also include cases where the right answer is to do nothing: a document that is not an invoice, a message that needs no reply, a record that should be left alone. A feature that is never tested on these learns that every input deserves an output.

We leave out anything synthetic. A case we wrote ourselves tests whether the model can read our handwriting. The failures that matter come from inputs nobody on the team would have thought to write.

Labelling with the people who do the job

The answers come from the person who did the job before the model. For invoices, the bookkeeper. For support replies, the agent who wrote the real reply. For a classification, the operator who made the call.

They do not label in a spreadsheet of model outputs, because then they are grading, and graders are generous. They label the raw case, blind, in a tool that looks like their own work, and the label is their answer. Where two people do the job, two people label a sample of the same cases, and the rate at which they disagree is the ceiling for what the model can be measured against. On messy document work that ceiling is lower than anyone expects, and finding it early stops a quarter of arguing about whether the model is wrong when the two bookkeepers do not agree either.

Labelling sessions also produce the specification. Every time a labeller asks what to put when the invoice says “crate” and the kitchen counts in kilograms, the question and its answer go into the definition of correct.

Defining correct per field

Correct is defined per field, in writing, before the first run, and that definition is the real spec for the feature.

For an invoice line, the product text must match the printed text after whitespace and case normalisation. The quantity must match as a number. The unit must map to the same canonical unit, so “kg”, “KG” and “kilo” are equal. The unit price must match to the cent. A missing value is correct when the page has no value and wrong otherwise. A value with a citation that points at the wrong region is wrong even when the number is right, because a reviewer will be misled by it.

For free text, such as a reply or a summary, exact match is useless, and we fall back to a rubric of yes-or-no questions a second model or a person can answer: does it address the question, does it contain any claim the record does not support, does it route to a person when the rules say it should. Each question gets its own number.

The per-field numbers drive the work. An overall score hides that supplier names are at 99 and unit prices at 94, and it is the unit prices that cost money.

Size

Twenty cases is enough to argue about. It shows whether a change broke something, and it is what we run while iterating on a prompt in the first days.

Two hundred is enough to trust. At that size a difference of a few points between two versions is more likely real than noise, and the set covers enough suppliers, formats and edge cases that a version which wins on it will usually win in production. Beyond a few hundred the return on more cases drops, and the time is better spent on the ugly ones and on the production failures below.

The set is stratified, so the numbers can be read per supplier, per document type and per field. A model update that improves the average and ruins one supplier is a regression for that supplier’s restaurants.

Running it on every change

The set runs as part of the build, against the prompt, the model, the parsing code and the pre-processing. Any of the four can move the number, and the last two are the ones people forget. A change to how pages are rendered before they reach the model has broken more extraction than any prompt edit we have made.

The run produces a report with the per-field numbers, the change from the last accepted run, and the list of cases that went from right to wrong. The list is what the engineer reads. A number going from 96 to 95 is a shrug; the three invoices that broke are a bug report.

Each run is stored with the versions of everything it used, so the number has a history. The history answers whether the vendor’s model update last month made us worse, and it is what a client asks to see once they have been burned by a demo.

Growing it from production

After launch the set grows from production in two ways. Every correction a reviewer makes is a failure with a known answer, and it goes into the set once it has been checked. A sample of confirmed cases, where the model was right and the reviewer agreed, goes in too, so the set keeps tracking what production looks like as suppliers, formats and customers change.

Growing it this way shifts the mix towards failures, which is what we want for finding regressions and what we do not want for estimating the production error rate. We keep the two readings separate: the full set for regressions, the random sample for the rate.

The set also has to be pruned. Cases from a supplier that no longer exists, or a format the product no longer accepts, add noise. Once a quarter someone reads the failures and asks whether each still matters.

What it does not catch

An evaluation set measures the past. It does not catch the supplier who appears next month with a format nobody has seen; shadow mode and the review queue are for that. It does not catch latency or cost at production volume, because it runs as a batch. It does not catch the reviewer who approves everything in two seconds; that is a log entry. And it does not catch the case where the model is right and the workflow around it is wrong, where the extraction is perfect and the inventory has three entries for the same tomato.

It also does not say whether the feature is worth building. A feature at 98 per cent that nobody opens and a feature at 90 per cent that saves an hour a day look the same in the report.

If you have an LLM feature, shipped or planned, and no number next to it, the AI consulting page describes how we run an evaluation on real data before anything is built, and you can write to us with the cases you already have.