Keeping a fleet of wearables synced through phones that want to sleep
BLE reconnection, iOS background modes and Android Doze, device buffering, idempotent uploads, clock drift, and fleet monitoring that catches a quiet device.
AI customer support automation: what to automate first, and what to keep
A business with three support agents and a shared inbox opens on Monday to a few hundred tickets. Most of them are the same dozen questions: where is my order, how do I change my address, why was I charged twice, how do I reset my password. A few are the ones that matter: the customer whose delivery missed a funeral, the chargeback, the solicitor’s letter. The pitch from support software is a bot that answers all of it. What works is narrower and comes in an order, and the order decides whether customers notice an improvement or start writing in twice.
Read a month of tickets first
Before choosing a tool, export the last thirty to sixty days of tickets with their replies and outcomes. Have a language model propose an intent for each, then have the person who answers most of them correct the list until it settles at fifteen to thirty intents. For each intent, write down four things: how many tickets it had, whether the answer is in the help centre, whether the answer needs the customer’s own data, and whether it needs a decision someone could get wrong with money attached.
That table is the plan:
| Intent | Example | When to automate |
|---|---|---|
| High volume, documented answer, no decision | how do I reset my password | first |
| Needs the customer’s own data | where is my order | second, once lookups exist |
| Needs judgement with money attached | a refund outside written policy | stays with people |
The same export becomes the evaluation set: past tickets with the reply a person gave and whether the customer came back. How we build and maintain one is in building an evaluation set from past cases.
AI ticket triage comes before any answer
The first thing to automate is reading, not replying. A model reads each incoming ticket, assigns an intent from the list, pulls out the order number, the product and the language, and flags anger, urgency and anything that looks like a legal or safety issue. The helpdesk routes on those fields.
Triage goes first because a mistake there is cheap and invisible to the customer. A wrong label costs one reassignment. A wrong answer costs a second contact and some trust. Triage also forces the intent list to become real, and every later step depends on that list.
Support triage is not the inline classification problem we describe in classifying messages at line rate. A ticket can wait a few seconds, so a large model is fine, and the cost per ticket is small next to an agent’s minute. Measure it by the reassignment rate per intent and by how many tickets land in “other”. A growing “other” bucket means customers have started asking something new.
Drafted replies an agent approves
The second step is a draft. For each ticket in a supported intent, the model writes a reply in the helpdesk and the agent edits it or sends it as it is. Nothing reaches a customer without a person pressing send.
Drafts give you the best evaluation signal you will get for free: the difference between what the model wrote and what the agent sent. Log both. An intent where agents send the draft unchanged nine times out of ten is ready for the next step. An intent where they rewrite the second paragraph every time has a gap, usually a missing or outdated help article, sometimes a policy that lives only in the agents’ heads.
Watch the reviewers as well as the drafts. A draft that is right most of the time trains people to approve without reading. The article on putting an agent inside an existing workflow covers shadow mode, widening permissions one step at a time, and making a reviewer who approves everything in two seconds show up in the log. The same applies here.
Answers from the help centre, with citations
Once an intent’s drafts go out unedited, it can answer customers directly in chat or by email. The model retrieves from the help centre, answers only from what it retrieved, and links the article it used. If retrieval finds nothing relevant, or the model cannot point to the passage that supports the answer, the reply is not sent and the ticket goes to a person. A missing answer is a handover. A plausible invented answer is a liability.
That liability is not hypothetical. In Moffatt v. Air Canada, British Columbia’s Civil Resolution Tribunal held the airline responsible after its website chatbot told a customer he could claim a bereavement fare retroactively, which its policy did not allow. The tribunal rejected the argument that the correct policy was on another page and treated the chatbot as part of the website. Whatever the bot says, the business has said.
The citation rule is the one we used on bilz.ai, which we worked on as technology partner: every extracted number links back to the document it came from, so a person can check it in seconds. In support, the citation lets the customer check the answer, and it lets you find the article to fix when the answer was wrong. Whether to retrieve or to train a model on your documents is a separate question, covered in RAG versus fine-tuning for company data. For a help centre that changes weekly, retrieval is the answer.
If you serve customers in the EU, Article 50 of the EU AI Act applies from 2 August 2026 and requires that a system talking directly to people is built so they know it is an AI, unless that is obvious. Say so in the first message.
Order and account lookups through tools
Most of the volume in a small business is about one customer’s own order or account, and no help article answers “where is mine”. This is where the model gets tools: functions it can call, such as order status, delivery tracking, invoice resend and subscription details.
Start with read-only tools, and scope every one to the customer who has been identified. The model never chooses whose data to look up. The conversation carries a verified customer ID, set by your login or by a check the bot cannot skip, and the tool uses that ID and ignores any other ID the model passes. A customer who types someone else’s order number gets nothing back.
Write actions come later and one at a time: change a delivery address before dispatch, cancel an order that has not shipped, resend a confirmation. Each has a limit written into the tool rather than into the prompt. A customer message is untrusted input that the model reads, which is the problem described in prompt injection defences for tool-using agents. The tool’s own checks are what hold when the prompt does not.
What stays with people
Some tickets should go straight to a person without the bot trying first. Write the list down and enforce it in triage.
| Ticket | Goes to | Why |
|---|---|---|
| Refund above a set amount | a person | the threshold is a number a manager picks, and the tool refuses anything over it; below it, a refund within written policy can be automated once lookups are trusted |
| Angry or vulnerable customer: bereavement, illness, financial hardship, a mention of harm, a second or third contact | a person, on the first message | triage can detect these well enough to route them; a bot reply to someone in that state, however accurate, makes things worse |
| Legal: threats of legal action, chargebacks and disputes, regulator correspondence, data protection requests | a person, often outside support | under Article 12(3) of the GDPR, a data subject request has to be answered without undue delay and, in most cases, within one month of receipt, so it needs an owner and a clock, not a chat reply |
| Customer asks for a person | a person, at the first request | making them ask three times turns an automated channel into a complaint |
Handover to a human with context
A handover that makes the customer repeat themselves wastes most of what the automation saved. When the bot hands over, the agent should see a short summary at the top of the ticket: the intent, what the customer asked in one or two sentences, what the bot looked up and found, what it already told the customer, and why it handed over. The transcript sits below.
Tell the customer what happens next and roughly when, in business hours that are true. A handed-over conversation should keep its place rather than join the back of the general queue, because the customer has already waited once.
The handover reason is also data. Count it per intent each week. “No article found” is a writing task. “Customer asked for a person” on an intent the bot handles well may mean the tone is wrong. “Lookup failed” is an integration bug.
Deflection versus resolution
Deflection is the share of conversations that never reached a person. It is the number most tools report, and it counts the customer who gave up as a success. A deflected conversation that comes back by email two days later, or by phone, or as a chargeback, was delayed rather than resolved.
Measure resolution instead. A conversation counts as resolved if the same customer does not contact you again about the same intent within a set window, on any channel, and the order shows no refund, return or dispute that the answer should have prevented. Seven days is a reasonable start for retail and subscription businesses. Matching across channels needs a customer ID on every ticket, which is another reason triage comes first.
In that made-up month the dashboard says 60 per cent deflection. Of the 600, 120 customers write again within seven days about the same thing and 30 call. Real resolution is 45 per cent, and the 150 who came back reached an agent anyway, later and more annoyed. The 45 is the number to improve, and the 150 are the cases to read.
Read some by hand each week. Twenty bot conversations picked at random, plus every conversation where the customer came back, will show problems no metric does. Satisfaction surveys help less than they seem to, because the customers who answer them are not the ones who gave up.
Keeping the knowledge base current
An answering bot is as current as the help centre, and help centres tend to lag the product. The bot repeats every outdated article to every customer who asks.
| Case | What to do |
|---|---|
| Every article | give it an owner and a review date |
| A change in pricing, shipping or policy | add help centre updates to the checklist, so the article changes on the day the policy does |
| Anything seasonal, such as holiday delivery cut-offs | put an expiry date on it |
| An article is published | rebuild the retrieval index then, not on a weekly schedule |
| Two articles contradict each other | find the pair, because the model will quote whichever one it retrieves |
Every handover marked “no article found” and every draft that agents rewrite the same way is a candidate article, and that list deserves a weekly review.
Tone and languages
Write the tone guide from your best agent’s replies rather than from adjectives. Pick twenty replies the team agrees are good and give them to the model as examples. Short answers, the customer’s name if they used it, one apology when something actually went wrong, and the next step stated plainly. The usual failures are over-apologising, repeating the question back and padding a two-line answer to six.
For languages, the model can answer in the customer’s language from help articles written in yours, and still cite the original article. Keep product names, plan names and legal terms untranslated with a short glossary. Before switching a language to customer-facing, have a native speaker read a sample of fifty answers. A language with little volume can stay at the draft stage for a long time without much cost.
A sequence over six weeks
| Week | Step |
|---|---|
| 1 | export the tickets, settle the intent list and build the evaluation set from past tickets and their outcomes |
| 2 | run triage on live tickets in shadow, compare its labels with the agents’ and switch routing on when they agree |
| 3 and 4 | turn on drafts for the five largest intents and log every edit |
| 5 | move the two or three intents with the fewest edits to customer-facing answers with citations, and add read-only order lookups |
| 6 | measure resolution against the deflection number, read the customers who came back, and decide which intent goes next |
Some intents will never leave the draft stage. The AI agents and automation page describes how we take on this kind of work. Most of the plan comes out of that first export, so if you send us a month of anonymised tickets we can start from those.
