Reliable structured output from LLMs: JSON mode, schemas and validation

A model that answers in prose is easy to demo and hard to build on. Once its output feeds code, a ticket queue or a ledger, it has to arrive as a record with known fields and types, every time. The failures come in two kinds. The first is syntactic: a sentence before the opening brace, a markdown fence, an object cut off at the token limit. The second is worse because nothing breaks: the JSON parses, every field has the right type, and the order number in it does not exist. Provider features now handle most of the first kind. The second stays your problem.

JSON mode vs structured outputs vs tool calling

The three features are often described as one. They guarantee different things.

Feature	What it guarantees	What it does not
JSON mode	Syntactically valid JSON, and that is all	Any schema: OpenAI says it “will not guarantee the output matches any specific schema, only that it is valid and parses without errors”. Fields can be missing, renamed or of the wrong type
Structured outputs (schema-constrained decoding)	Output that follows your JSON Schema, not merely output that looks like JSON	Correct values; a match when the model refuses or hits the token limit
Tool calling	With strict on, arguments that follow the function’s schema; without it, best effort	Correct values

Structured outputs take a JSON Schema and restrict which tokens the model may produce at each step. OpenAI says the feature “ensures the model will always generate responses that adhere to your supplied JSON Schema”. Anthropic’s version, set with output_config.format, works by “constrained sampling with compiled grammar artifacts”. Google’s Gemini API has a JSON Schema response format too, and its documentation carries the sentence worth remembering: “While output is syntactically correct JSON, always validate values in your application.”

Tool calling applies the same mechanism to the arguments of a function the model decides to call. OpenAI says setting strict to true makes function calls “reliably adhere to the function schema, instead of being best effort”, and Anthropic’s strict tool use does the same for tool names and inputs. Our rule follows OpenAI’s guidance. If the model always produces the same kind of record, such as an extraction or a classification, it is a response format. If the model chooses between actions, each action is a tool with its own strict schema, and the choice is the output.

What the guarantee leaves out

A refusal takes precedence over the schema, and both OpenAI and Anthropic document that a response cut off by the token limit may not match the schema either. Code has to check the stop reason before it parses.

The supported schema is a subset of JSON Schema, and the subsets differ. At the time of writing (October 2026):

	OpenAI	Anthropic
Refusal	In a separate refusal field	stop_reason: "refusal" with a normal 200 status, and output that may not match
Schema rules	Every field listed as required; additionalProperties: false on every object	No numeric bounds such as minimum and maximum, no string length limits
Size limits	Up to 5,000 properties and 10 levels of nesting	At most 24 optional parameters and 16 parameters with union types across all strict schemas in a request
First request with a new schema	Slower, because the provider compiles it	Slower for the same reason; the compiled grammar is cached for 24 hours from last use

Anthropic’s SDK helpers remove unsupported constraints from the schema the model sees and copy them into the field description, so a minimum: 0 becomes a request rather than a rule unless something validates the response afterwards. Anthropic also notes that string enum values are not guaranteed to keep their capitalisation.

So read “guaranteed” as “the shape is right in the common case, for the keywords your provider supports”.

Does constrained decoding make the answers worse

It can. A 2024 study, Let Me Speak Freely?, compared JSON mode, format instructions in the prompt, and free answers converted afterwards. Reasoning tasks got worse as the constraints got stricter, while classification often got better, because the constraint removes answers that are not on the list. One detail explains a lot: under JSON mode, GPT-3.5 Turbo put the answer key before the reason key in every response, committing to an answer before writing any reasoning.

Two rules follow. Put the fields the model needs to think with before the field it decides, because generation runs left to right and a decision written first cannot use evidence written after it. And make that order real: Anthropic places required properties before optional ones whatever the order in your schema, so a field you want early must be required. Keep it short, too. Anthropic warns that a property asking for step-by-step reasoning can cause a reasoning_extraction refusal and advises asking for a short explanation instead. When a task needs long reasoning, let the model reason freely first and extract the record in a second, cheaper call.

Designing a schema the model can fill

A schema is a form, and the model fills it the way a hurried clerk would: something goes in every box. The design work is making the honest answer easy to write. Here is an example schema for triaging support messages.

import { z } from 'zod'

// Evidence comes first, so the model writes it before it decides.
export const Triage = z.object({
  evidence: z
    .array(z.string())
    .describe('Short quotes, copied verbatim from the message, that support the fields below'),
  category: z.enum(['billing', 'delivery', 'refund', 'account', 'other']),
  category_note: z
    .string()
    .nullable()
    .describe('Only when category is other: what the message is about, in a few words'),
  order_id: z
    .string()
    .nullable()
    .describe('Exactly as written in the message; null if the message gives none'),
  refund_amount: z.number().nullable().describe('Null unless the customer states an amount'),
  currency: z.enum(['EUR', 'GBP', 'USD']).nullable(),
})

Every closed set is an enum, including a way out. Without other, a message about a cancelled subscription gets filed as billing or refund, whichever is closer, and nothing tells you. With other and a short note beside it, the misfit is visible, and the share of other tells you when the enum needs a new value. Avoid values that differ only in case.

Every field that might be absent from the source is nullable rather than optional, and its description says what null means. A required, nullable field makes the model state something: the order number is this, or there is none. An optional field lets it stay silent, and silence is ambiguous downstream. Null is a good answer. An invented order number is the expensive one. On Anthropic each nullable field counts towards the 16 union-type parameters, so a large schema spends its nullables where absence is genuinely possible.

Evidence goes in the record. A verbatim quote per value costs a few tokens and buys two things: a reviewer sees why the model chose what it chose, and code can check that the quote appears in the source. In document work the evidence can be a region of the page. The supplier invoice pipeline we built for bilz.ai as technology partner cites a bounding box for every value, and drops a value whose citation points at empty space.

Keep it shallow. Each level of nesting is another place for the model to lose track of which object it is filling, and deep schemas reach compile limits first. A schema that needs arrays of objects containing arrays of objects is usually two jobs: extract the flat facts in one call, and group them in code or a second call. Descriptions are prompts as well. “Exactly as written in the message” gets different behaviour from “the order ID”.

Validate at the boundary, then repair or retry

Every response goes through a schema library on the way in, whether or not the provider constrained it: Zod in TypeScript, Pydantic in Python. One schema object generates the JSON Schema sent to the model and parses the response, so the two cannot drift apart. This covers what constrained decoding does not: refusals, truncation, the constraint a helper moved into a description, and the day a request is routed to a model that supports less of the schema. Every route in a model routing setup has to produce records the same validator accepts.

When validation fails, repair in a fixed order. First, deterministic fixes that cannot change meaning: strip a markdown fence, normalise the case of an enum value. Second, one retry with the validation errors in the message, naming the field and the rule it broke, so the model corrects that field instead of regenerating everything. For truncation the retry is a higher token limit or a smaller job. Third, the record goes to a person, or the request fails closed, depending on what a missing record costs. We cap retries at one or two. A loop that retries until something parses turns a bad input into a bill.

Log every repair with the field and the reason. The repair rate is one of the first numbers to move when a provider ships a new model version, and it belongs beside latency and cost; LLM observability in production covers the rest of that dashboard.

Semantic validation: valid JSON can still be wrong

The schema says a field is a string. Only your code knows whether the string means anything. We write four kinds of check after the schema check.

Check	What it compares	Why
Arithmetic reconciles	Line totals sum to the subtotal; subtotal plus tax equals the total within rounding	On invoices this is the best single check we have, because the printed total is the number the supplier checked most carefully
References exist	An order_id is looked up in the orders table, a product code in the catalogue, and the record must belong to the account that sent the message	A well-formed ID that does not exist is a typical confident error, because models are good at plausible identifiers
Quotes occur in the source	Each evidence string is matched against the input after normalising whitespace	A quote that is not there was paraphrased or invented, and the fields it supports are suspect
Cross-field rules hold	category_note is set only when the category is other; a refund amount comes with a currency; a delivery date falls after the order date	Provider schema subsets mostly cannot express this, so it lives in code

A failed semantic check gets one retry like a schema failure, with the rule stated in plain words: “the line totals sum to 412.50 and the stated total is 421.50”. Told where to look, the model often finds its own misread digit.

Streaming partial objects

When a person is waiting, a record should stream like prose does. The provider streams the JSON as text, and a partial JSON parser turns each prefix into a partial object the interface can render; Gemini’s documentation describes its chunks as “valid partial JSON strings that can be concatenated to form the final JSON object”.

Render partial fields, but never act on them: a half-streamed order_id is a prefix of an ID, and a lookup on it either fails or matches a different order. Validate only the final object, after the stop reason says it is complete. Order the schema so the fields the interface shows first are generated first, which on Anthropic means making them required. No tool runs until its arguments are complete and validated.

Versioning the schema

The schema is a contract with three parties: the model, the code that reads the record, and every record already stored. Changing it without a version breaks the third quietly.

Each schema gets a version in code, not in the model’s output, stored with every record next to the model identifier and the prompt version.

Change	Version	Why
Add a nullable field	Minor	Old records read it as null
Rename a field, change a unit, split an enum value	Major	refund before the split and refund after it mean different things, and a report that counts them together is wrong without anyone noticing

Old parsers stay for as long as old records do.

A schema change runs through the evaluation set like a prompt change, and the first production request after it pays the compile latency again. On Anthropic, editing only a field’s description keeps the cached grammar, while a change to the structure invalidates it.

Measuring accuracy per field

A valid-JSON rate near 100 per cent says the plumbing works, not that the records are right. Accuracy is measured per field against cases a person has labelled: exact match for enums and identifiers, a tolerance for amounts, a normalised comparison for free text. How we build such a set is in building an evaluation set from past cases.

Nulls need two numbers. A null where the source had a value is a miss, and a person fills it in. A value where the source had none is an invention, and it enters the system as fact. The second is the one to drive down, and averaging the two hides it. For enums, a confusion matrix shows which pairs the model mixes up, and the share of other shows where the enum no longer fits the traffic.

If you are starting from nothing, start with the log: for every call, the schema version, the stop reason, whether validation passed, which fields were repaired and which semantic checks failed. A week of that shows where the work is. The custom AI apps page describes how we build LLM features into products, and you can write to us with a sample of the output you need to rely on.