LLM observability in production: what to log, trace and alert on

An LLM feature fails without raising an error. The provider returns a 200, the JSON parses, the latency is normal, and the unit price on the invoice line is wrong. Application monitoring reports a healthy service, because by every measure it was built for, it is one. LLM observability is the record that answers the questions that come afterwards: which prompt produced this, which model answered, what it saw and called, what it cost, and whether it has been happening more often since Tuesday. This is what we record, where personal data goes, and what we alert on.

What to log for every LLM call

Every call to a model writes one record, and the fields that earn their place are the ones somebody needs when a result is disputed.

Field	Why it is there
Prompt version	A name and version for the template, plus a hash of the rendered system prompt and tool definitions, so two calls that claim the same version can be checked
Model requested	What the code asked for
Model that answered	Taken from the response, because the two differ more often than people expect
Parameters	Temperature, output limit, reasoning effort and the tool list change behaviour
Inputs and outputs	Or a reference to them, as the privacy section below explains
Tokens by type	As the provider reports them, because cached input, uncached input, output and reasoning tokens are billed at different rates
Latency	Time to first token apart from total time
Finish reason	Where a truncated answer shows up: a response that stopped at the output limit is a failure that parses
Cost	Computed when the record is written, from the token counts and a price table with its own version, because prices change and a cost recomputed months later from today’s prices is fiction
Outcome	Whether the output passed schema validation, and later whether a person accepted it, corrected it or threw it away; it arrives minutes or days later, so the record needs an identifier the review screen can write back to

The prompt version comes first. A prompt is code and needs an identifier the way a release does. Without it, “which prompt did this” turns from a ten-minute question into an afternoon.

The model is recorded twice because a gateway falls back to a second model on an error, an alias moves to a new snapshot, a router picks a cheaper model for an easy request. On EroPlay.ai, where we routed requests to several models through OpenRouter as technology partner, the model was a property of each request rather than of the deployment, and only the response could say which one had answered.

Tracing multi-step agent runs

One log line per call is enough for a feature that makes one call. An agent makes a dozen, between tool calls, retrievals and database writes, and the question becomes why the run ended where it did. That needs a trace: one parent span for the run, a child span for each model call and each tool call, all sharing a trace identifier and ordered in time.

OpenTelemetry has semantic conventions for generative AI that name these spans and their attributes. They are worth following even without an OpenTelemetry backend, because a trace in a standard shape can move between tools. A model call is a span named after the operation and model, such as chat followed by the model name, with attributes including gen_ai.request.model, gen_ai.response.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens and gen_ai.response.finish_reasons. The agent conventions add an invoke_agent span for the run and an execute_tool span for each tool call, named after the tool, with gen_ai.tool.name and gen_ai.tool.call.id.

Two cautions. The conventions are marked Development, which means names still change: gen_ai.system was renamed to gen_ai.provider.name in version 1.37.0, and a dashboard built on the old name goes quiet after an upgrade. Pin the version you emit and treat an upgrade as a migration. And the conventions cover the model and the tools, not your own steps. The schema check, the retrieval, the database write and the hand-off to a person are spans you add yourself, and they are often where the run went wrong. On the invoice pipeline we built for bilz.ai, every extracted number links back to the region of the document it came from. Store the run’s trace identifier next to the value as well, and a disputed number leads to the page, the model call and the prompt version in one step.

Keeping personal data out of logs

Prompts contain whatever the user typed and the agent retrieved: names, addresses, account numbers, health details, whole email threads. A log store is usually the least protected copy of a company’s data, kept longest and read by the most people. Copying prompts into it unchanged is how a privacy incident happens without anyone deciding anything.

The OpenTelemetry conventions agree. Inputs, outputs, system instructions and tool arguments are opt-in attributes, instrumentations should not capture them by default, and for production, where volume or sensitive data is a concern, they recommend storing content externally and recording only a reference on the span, so that content can have its own access controls.

We split it the same way. The telemetry store holds metadata only: versions, models, token counts, latencies, costs, outcomes and identifiers, kept for a long time and readable by the whole team. The content store holds the prompts and outputs, encrypted, readable by a few named people, with every read logged, kept for a short period, and reachable by the same deletion process as the rest of a user’s data. We describe that access log and deletion in consent and access logs in a health platform. User identifiers in telemetry are replaced with a keyed hash, so a dashboard can still count distinct users and group a user’s runs without holding who they are.

Redaction happens before anything is written. Pattern rules catch the structured things reliably: email addresses, phone numbers, card numbers, IBANs. Named-entity detectors such as Presidio catch some names and addresses in free text, and miss others. Redaction reduces what reaches the content store. It does not make the store safe to open up, so the retention and access rules stay. The GDPR asks for data minimisation and storage limitation in Article 5, and an LLM log with no expiry date fails both.

Content sent to a hosted tracing service is personal data sent to another processor, with its own contract, region and retention. Self-hosting the tracing tool is often the simplest way to keep content under the same rules as the database.

Sampling without losing the failures

Metadata is small, so we keep it for every call. Content is large and sensitive, so we keep less of it, and sampling decides which part.

Deciding at the start of a run keeps a random slice and drops most failures with everything else. Deciding at the end, once the outcome is known, keeps what is worth reading. OpenTelemetry’s collector has a tail sampling processor built for this, with policies on status, latency and attribute values. Our rules keep the content of every run that errored, failed validation, was corrected by a person, cost more than a multiple of its usual amount, took more steps than usual or was flagged by a user, and a small uniform sample of the rest.

The uniform sample is not optional. A store of failures alone cannot show what normal looks like or estimate the error rate, because the denominator is gone. Record the sampling rate on every kept trace so dashboards can weight it back up to real volume.

As an example with hypothetical numbers, for a feature that runs 20,000 times a day:

Runs per day	Count	Content kept	Metadata kept
Failures by the rules above, 3 per cent	600	all 600	all
The rest, uniform sample of 2 per cent	19,400	about 390	all
Total	20,000	about 990	all 20,000

From production traces to evaluation cases

A trace with its content is most of an evaluation case. It has the exact input, prompt version, model and tool results, so a failure can be replayed against a new prompt or model. What it lacks is the right answer: a reviewer’s correction supplies it, and an uncorrected failure needs labelling by the person who does the job before it counts. Building, labelling and balancing that set is covered in building an evaluation set from past cases.

Each case keeps the identifier of the trace it came from, so a case that fails a year later leads back to the conditions that produced it. And promoting a trace to a case copies personal data into another store, so the evaluation set needs the content store’s redaction, access and deletion rules, and a user who asks to be deleted leaves the set too.

Online checks and alerts

Some checks run on every call: the output parses against its schema, the required fields are present, the finish reason is not the output limit, the tool arguments are valid, the run stayed under its step ceiling. They are cheap and exact, and their failure rates are the first numbers to chart. Making the schema check pass more often is its own article, reliable structured output from LLMs.

Other checks run on a sample, after the fact: a rubric scored by a second model or a person, asking whether a reply answers the question, makes a claim the record does not support, or should have gone to a person. They are slower and noisier, so we chart them as a daily rate rather than alerting on single results.

The alerts we set on every feature are few.

Alert	How it is counted
Cost per completed task	Against its own recent baseline, not tokens per call; what an AI agent costs to run explains why the task is the unit
Schema failure rate	Share of calls whose output fails the schema check
Refusal rate	From the finish reason where the API reports one, and from a cheap classifier on the output where it does not, because a refusal often arrives as polite text
Tool error rate	Per tool
Answered by another model	Share of requests answered by a model other than the one configured, which is how a silent fallback shows up
Correction rate	From the review queue, the closest thing to the business’s own measure of quality

Every one is relative, because an absolute threshold set in the first week is wrong by the second month. We compare the last hour or day with the same period over the previous weeks and alert on the ratio, per feature, since a regression in one feature disappears inside the average of five.

The last alert is drift after a change. Models change under you: providers retire them on published schedules, aliases move to new snapshots, and a gateway can change which provider serves an open model. Every change to the prompt version, the response model or the price table becomes a marker on the time axis, and for a few days we compare the distributions either side of it: output length, refusal rate, schema failures, tool calls per run, cost per task and corrections. Longer answers show within hours. The quality change behind them may take a week to reach the corrections.

Tools fall into three groups, named here as examples rather than recommendations: open-source tracing and evaluation platforms you can host yourself, such as Langfuse and Arize Phoenix; OpenTelemetry instrumentation libraries that emit the gen_ai spans into whatever backend you already run, such as OpenLLMetry and OpenLIT; and LLM features inside general monitoring products, such as Datadog’s. What decides between them is where the content is stored, whether it can stay in your infrastructure, whether the data leaves in OpenTelemetry form if you do, and how the price grows with trace volume.

A dashboard the business owner reads

Engineers need the trace view. The person who pays for the feature needs one page per feature, in the units of their business. For an invoice reader that is invoices processed, the share that went through without a correction, the cost per invoice, the size and age of the review queue, and corrections to the field that costs money, the unit price. For a support agent it is conversations handled, the share resolved without a person and the cost per resolved conversation.

Each chart carries the version markers, so a drop has an obvious first suspect, and nothing on the page is in tokens. A business owner who sees, say, “cost per invoice rose from 3 to 5 cents the day the model changed” can make a decision. One who sees a token chart forwards it to the engineers.

If you add one thing this week, add the prompt version and the model that actually answered to every call you log. Every question that comes later starts with those two values, and they cannot be filled in for the calls already made. The AI agents and automation page describes how we build agents with this record from the first release, and you can write to us about a feature that is already live and hard to see into.