Classifying messages at line rate

A classifier that sits in the delivery path of an SMS platform has a few milliseconds to decide, on every message, whether to pass it, hold it or drop it. We built virtual-number and messaging products for years, and the shape of this problem was the same in each one. The model is the easy part. The budget, the queue and the failure mode are where the design lives. The overview of LLMs in telecom covers where classification sits among the other uses. This article is about the inline case.

The budget

A messaging platform’s own processing adds latency to every message before it reaches the carrier, and customers notice when one-time codes arrive late. The classifier’s share of that is small: single-digit milliseconds at the median, with a hard ceiling in the low tens, because a one-time code that takes an extra half second is a support ticket. Within that budget there is no network round trip to a hosted model, no large model, and no tokenisation of a long prompt. There is a small model, or a fine-tuned classifier, running in the same process or on the same rack as the router.

That constraint rules out most of what people imagine when they say AI classification. It rules in a lot that works. A linear model over character n-grams decides in under a millisecond. A small transformer with a few million parameters, distilled or fine-tuned for the task, decides in a few milliseconds on a CPU. Either is enough for most of the traffic, which is repetitive.

Train on your own routes

Public spam datasets are email from a decade ago. Your traffic is one-time codes, delivery notifications, marketing blasts with and without opt-out text, and the grey-route attempts that imitate all of them, in the languages and the sender-ID conventions of the countries you route to. A classifier that scores well on a public set and poorly on your traffic is the common outcome, and it is caught only by measuring on your own messages.

The labelled set is built from outcomes you already have. Messages that were blocked by rules and upheld on review. Messages that were delivered and complained about. Messages from accounts later suspended. A few thousand of these, labelled by a person or by a large model and checked by a person, is the first training set and the permanent evaluation set. Every candidate model, and every later version, is measured against it before it goes near the delivery path.

The uncertain tail

The small model is confident about most messages and unsure about a few percent. Those are the messages that look like a one-time code from a sender ID that has never sent one, or marketing text in a language the model has seen little of. The inline decision for an uncertain message is made by policy, usually to pass it and flag it, and the message is also sent out of band to a large model that can take a second and a cent to read it properly.

The large model’s verdict does not change the delivery of that message, which has already gone. It changes two other things. It feeds the account’s score, so the next message from the same sender meets a different threshold. And it becomes a labelled example for the next training run, which is how the uncertain tail shrinks over time. The large model is a labeller and an auditor, and it is never in the path.

Fail open or fail closed

The classifier will be unavailable sometimes. The process restarts, the model file is being swapped, a dependency times out. What happens to messages during those seconds is a business decision, and it should be made in a meeting rather than in an incident.

Fail open means messages pass unclassified. The risk is a window during which spam reaches handsets and the platform’s sender reputation takes the hit. Fail closed means messages queue or are rejected. The risk is that legitimate traffic, including one-time codes, is delayed by a component that was added to improve quality. Many platforms fail open for established accounts and fail closed for new ones, which is a reasonable compromise, and whatever the choice it has to be written down, implemented as a timeout with an explicit default, and tested by killing the classifier on purpose.

Queues and backpressure at peak

Messaging traffic is spiky by nature. A bank sends a security notice to every customer at once, a retailer runs a sale, a sports result goes out. The classifier’s throughput at peak is the number that matters, and the queue in front of it is where a slow model shows up as late messages. The queue needs a bound and a policy for what happens when the bound is reached, which is the fail-open question again in another form.

The design we use is a bounded queue with a per-message deadline. A message that cannot be classified inside its deadline takes the default decision for its account class and is logged as unclassified. The count of unclassified messages is a first-class metric with an alert on it, because a rising count means the model has become too slow for the traffic, and that is the signal to scale out or to swap in a cheaper model before the queue becomes an outage.

Horizontal scaling of a small model is cheap. Each instance holds the model in memory and the router spreads messages across them. The shared state is the account score, which has to be read and updated consistently from every instance, and that store becomes the bottleneck long before the model does.

Feedback from delivery

The platform receives delivery reports from carriers and complaints from recipients and from the carriers’ own filters. Both are labels that arrive late. A message the classifier passed and the carrier filtered is a false negative. A message the classifier held, that the customer escalated and that review found clean, is a false positive. These flow back into the evaluation set and into the account score, and the review log of operator decisions does the same.

Watch the drift. The people sending grey-route traffic adapt to whatever is blocking them, so a classifier that was accurate in March is less accurate in June without anything having changed on your side. A weekly measurement against fresh labelled samples catches this, and that measurement is what justifies a retraining run.

Keep the model swappable

The model behind the classifier will change several times a year, through retraining runs, architecture changes and the next model that is cheaper and better. The code that calls it should be a thin interface: message in, label and confidence out, within a deadline. The evaluation set sits behind that interface, so a candidate can be run through it and compared on precision, recall and latency before anyone decides.

Rollout is a shadow run first. The candidate classifies the same live traffic as the current model without acting, and the disagreements are reviewed. Then a percentage of accounts, then all. The old model stays loadable for a rollback that takes a minute.

If you run a messaging platform and are deciding whether a classifier belongs inline or after delivery, the telecom and messaging AI page covers what we build, and you can write to us about your routes.