Keeping a RAG index current
Changed documents, deleted files and revoked access need different handling. How to keep a RAG index current without serving fragments of old revisions.
Designing a human review queue for AI output
An invoice reader proposes twelve kilograms of tomatoes. The invoice says twelve cases. A reviewer sees a green status, a product name and an Approve button. To catch the error, they have to open the PDF, find the line, check the pack size and return to the draft.
That review screen has given the person responsibility without giving them the evidence. A useful queue brings the source and the proposed change together, explains why the case needs attention and lets the reviewer correct the smallest relevant part.
Decide what needs a person
Start with the mistakes that would change the business outcome. For invoice processing, a missing unit, an unfamiliar supplier or a mismatch between line totals and the document total can justify review. A harmless difference in capitalisation usually does not. Write these rules with the person who currently enters the invoices.
Give each queued item a reason a reviewer can act on. “Pack size missing” points to a question. “Low confidence” leaves the reviewer to discover it. A confidence value supplied by the model should not decide which records bypass review unless you have measured how it relates to errors on representative cases.
Separate permission rules from uncertainty. An action can be easy to understand and still require approval because of its value or consequence. NIST’s AI Risk Management Framework calls for defined human oversight roles and assessment of the oversight process. A queue needs an owner, an escalation route and a clear account of what approval permits.
| Reason for review | Evidence to show | Decision needed |
|---|---|---|
| Unit is ambiguous | Printed line and supplier pack size, if known | Confirm a conversion or request the missing detail |
| Two inventory matches | Both items, their units and previous confirmed matches | Choose an item or leave the line unmatched |
| Total does not reconcile | Printed total, extracted lines and calculated difference | Correct the extraction or hold the invoice |
| Possible duplicate | Both documents and the existing record | Keep the new document separate or link it to the existing record |
Put the evidence beside the field
For a document task, show the original page region next to the proposed value. Keep a way to open the full page; a crop can omit a column heading or a note that changes the meaning. The reviewer should be able to see where the value came from without trusting a model-written explanation of it.
Allow field-level corrections. If the quantity is right and the unit is wrong, the reviewer should not have to reject and retype the whole invoice. Keep the original proposal alongside the correction, the reason and the reviewer identity. This gives the next person enough context to investigate a disputed entry.
Approval should apply to the version the person saw. If another worker changes the extraction while the page is open, reject the stale approval and show the new version. Recheck permissions and relevant business conditions when executing the approved action. A record can change while it waits in the queue.
Plan for the queue to fill up
Suppose a workflow receives 1,000 documents a day and routes 15 per cent to a person. At two minutes each, that is five hours of review before breaks, follow-up questions or corrections. These are planning assumptions, but they make the staffing question concrete. Measure the actual arrival rate and handling time during shadow mode.
Sort by consequence and deadline, then track age so difficult cases cannot sit at the bottom forever. Assign cases to people who can resolve them. A reviewer who cannot look up a supplier’s pack size needs a way to request it and leave the case visibly waiting for information.
Set a response when arrivals exceed capacity. You might narrow automatic processing to familiar suppliers, postpone non-urgent imports or add reviewer hours. A backlog must not silently turn into automatic approval. Track the oldest unresolved case as well as the total; a small queue can still contain yesterday’s urgent invoice.
Measure what happens after approval
An approval rate tells you how often people pressed the button. To learn whether review works, have someone independently check a sample of approved records against their sources. Compare the errors before and after review, including errors introduced by corrections. Give this check its own time allowance.
Sample records that bypassed the queue too. Reviewing only flagged cases cannot show how many errors the routing rules missed. Keep the samples separate so an increase in difficult cases does not look like a drop in overall extraction quality.
Add confirmed corrections to the evaluation set, after checking the label. A correction records a person’s decision; it can still be mistaken. Preserve the document and enough context to reproduce the disagreement later.
Give the reviewer a way to stop
Include a hold action with a reason, an owner and a next step. A missing delivery note may need a supplier response. An uncertain external write needs reconciliation before a retry. Neither case benefits from forcing the reviewer to choose between Approve and Reject.
Try the screen with ten past cases, including one incomplete document and one ambiguous match. Watch a person find the source, make a correction and hand off a case they cannot resolve. Record where they leave the screen to look something up. Those trips tell you what evidence the next version needs to bring into view.
