How to put an AI agent inside a workflow you already run

Most of the agent work we do is adding one to a process that already runs on people and spreadsheets. bilz.ai started that way. Restaurants were already receiving supplier invoices, and someone was already typing them into something. The agent replaced the typing. The process stayed. This is the order we do it in now, and where it has gone wrong.

Pick the workflow by its exceptions

The obvious candidate is the job that eats the most hours. That is often the wrong one, because the job that eats the most hours is usually the one with the most unwritten rules.

What we look for instead is a job where a person makes the same kind of decision many times a day, where the inputs are documents or messages rather than conversations, and where someone can say afterwards whether the decision was right. Invoice reading qualifies. A PDF comes in, lines go into a system, and the stock count at the end of the week tells you whether it was right.

Three questions decide it. What does the person look at before deciding? What happens when they get it wrong? Can you find the last two hundred cases with their answers? If the last answer is no, you do not have a workflow to automate yet. You have a workflow to start recording.

Build the evaluation set before the agent

We have never regretted the time this takes and have always regretted skipping it.

Take real past cases. For bilz.ai that meant real invoices from real suppliers, with the lines a bookkeeper had already entered. Twenty cases is enough to start arguing about; two hundred is enough to trust. Include the ugly ones on purpose: the photographed invoice with a thumb in the corner, the supplier who puts the discount on a separate line, the credit note that looks like an invoice.

Then write down what correct means per field. For an invoice line it is the product, the quantity, the unit and the price, and we accept a match on the product name only if it maps to the same inventory item. That definition is the real specification. The prompt is downstream of it.

With the set in hand, every change to the prompt, the model or the parsing code becomes a number. Without it, you are choosing between two demos by feel, and the louder demo wins.

Shadow mode

The agent goes into production on day one, but nobody sees its output except you. It reads every invoice the person reads and produces the same entry they would, into a draft table the person never looks at. A week or two later you compare.

Shadow mode finds the things the evaluation set missed, because the evaluation set was built from the past and the business keeps changing. A new supplier appears. Someone starts sending invoices as screenshots from a messaging app. A line item turns up in a language nobody warned you about.

It also tells you the agent’s real latency and cost at real volume, which the eval set, run as a batch at night, does not. An extraction that takes forty seconds is fine in a batch and unacceptable when a chef is standing at the delivery door waiting for the app.

We keep shadow mode running until the agent’s agreement with the person is at least as high as two people’s agreement with each other. On messy document work that bar is lower than people expect.

Widen permissions one step at a time

An agent’s first permission is to read and draft. Its second is to write records a person reviews before anything downstream uses them. Its third is to write records that flow through, with the person reviewing a sample. Only after that does it get to trigger anything irreversible, and some actions never get there.

In bilz.ai, the agent extracts invoice lines and matches them to inventory items on its own. It flags price increases and over-portioning on its own. It does not change a recipe’s cost basis without a person confirming the match, because a wrong match there pollutes every report after it. Every number it produces links back to the document it came from, so a person can check in seconds rather than minutes.

The error rate on the latest cases decides the widening. If a model update or a new supplier pushes the error rate up, permissions go back a step. Making that reversible from the start is cheap. Making it reversible after the fact is a weekend.

What goes wrong

The most common failure is that the agent is right and the workflow is wrong. The invoice is extracted perfectly and the inventory system it feeds has three entries for the same tomato. Automating a process exposes every ambiguity that people were quietly resolving in their heads.

The second is confident extraction of the wrong thing. Models do not leave fields blank when unsure; they fill them with something plausible. The fix is to require the agent to cite where in the document each value came from, and to treat a value without a citation as missing.

The third is drift. The supplier changes their template, the model vendor ships a new version, the restaurant adds a second location with a different POS. Each one shifts the numbers. The evaluation set has to grow with the business, and someone has to look at the dashboard.

The fourth is the reviewer. If a person gets a draft that is right ninety-five times out of a hundred, by the third week they approve without reading. Put the agent’s uncertain cases first and make them look different, and make sure a reviewer who approves everything in two seconds shows up in the log.

What the first version should cost you

Two to four weeks for one workflow, including connecting the systems and the shadow period. Most of that time goes to the connectors and the evaluation set. The prompt is a small part. If a proposal spends most of its time on the prompt, it has misjudged where the work is.

The deliverable is the agent, the evaluation set, the action log and a page describing how to roll permissions back. The agents and automation page describes how we run this, and you can write to us with the workflow you have in mind.