Prompt injection defences for AI agents that call tools

Say an agent answers a support inbox. It reads each email, looks up the customer’s orders, and can send a reply or issue a refund. One morning an email arrives whose body is not written to a person at all. It is written to the agent, in the same plain language as the agent’s own instructions, and it asks for something the business never intended. The model reads your system prompt and that email as one stream of text, and nothing in the model reliably tells the two apart. That is prompt injection, and for an agent with tools it is a security problem rather than a quality problem.

Direct and indirect prompt injection

Direct injection is a user typing instructions into the chat to make the model misbehave. The attacker is the user, so the damage is mostly bounded by what that user could already do. If they can only see their own account, the worst case is usually an embarrassing answer.

Indirect injection is the one that matters for agents. The instructions arrive inside content the agent reads on someone else’s behalf: an email, a web page, a PDF, a support ticket, a calendar invite, a code comment. The attacker never talks to the agent. They only need to get text in front of it. Greshake and colleagues demonstrated it against real applications in 2023. Prompt injection is LLM01 in the OWASP Top 10 for LLM Applications 2026, and excessive agency, the risk that a model has more tools and permissions than its job needs, rose from sixth place in the 2025 edition to third.

A supplier invoice that arrives by email is untrusted content too. So is a product review, a CV, or a forwarded thread. If your agent reads anything written by someone outside the company, it reads untrusted content.

Why filtering prompts does not work

The tempting fix is a filter: scan incoming text for anything that looks like an instruction, or add a line to the system prompt telling the model to ignore instructions found in emails. Both help a little and neither is a boundary.

SQL injection is prevented by parameterised queries, which keep code and data in separate channels. A language model has no separate channel. The instruction and the email arrive as tokens in the same context, and the model’s behaviour depends on all of them. Delimiters and “treat this as data” instructions shift the odds. They do not change the mechanism.

Classifiers that detect injections are probabilistic, and attackers adapt. A 2025 paper, The Attacker Moves Second, used attacks that adapt to the defence and bypassed 12 published defences with attack success above 90% for most of them, where the original papers had mostly reported near-zero. Simon Willison’s point about vendor guardrails is the right frame: catching 95% of attacks is a failing grade in web security, because the attacker only needs the other 5%.

Filters also have a cost on legitimate traffic. A customer who pastes an IT department’s instructions into a ticket looks exactly like an attack. Tune the filter tight and real requests go to the review queue; tune it loose and it catches only the clumsy attempts. We keep a classifier as a tripwire that flags cases for review and writes to the log, and we design as if it were absent.

The lethal trifecta: count what the agent can reach

Willison’s lethal trifecta (June 2025) names the three capabilities that turn an injection into a data leak: access to private data, exposure to untrusted content, and the ability to communicate externally. With all three, an attacker plants instructions in the content and the agent carries the data out. His conclusion is that the only safe option is to avoid the combination. Meta’s Agents Rule of Two (October 2025) makes the same argument with state changes included: within one session an agent should have at most two of untrusted input, access to sensitive systems or data, and the ability to change state or communicate externally. If it needs all three, it should not run unsupervised.

Take the support agent. It reads inbound email, which is untrusted. It sees order history, which is private. It sends replies, which is external communication. All three legs are present, and the agent needs every one of them to do its job.

The way out is usually to narrow a leg rather than remove it. The agent replies to the sender, and that is the job. The leak would be sending the sender someone else’s data. So the private data the agent can see is scoped by code to the account whose verified email address sent the message. An injection can then make the agent reveal the attacker’s own orders to the attacker, which is not a breach.

Exfiltration channels are wider than a send-email tool: a reply rendered as Markdown, a web fetch tool, a comment on a public ticket, a link the customer is likely to click. When we draw the trifecta for an agent, we list every way text leaves the system, and a rendered image counts.

Least privilege, enforced in the tool

OWASP’s excessive agency entry is mostly about tools that are broader than the task. A check in the prompt is a suggestion and a check in the tool is a rule.

Broader than the task	Narrowed in code
send_email(to, subject, body), where the model picks the recipient	reply_to_ticket(body), where the code fills in the recipient from the ticket
refund(order_id, amount) that trusts its arguments	refund(order_id, amount) that checks the order belongs to the current customer and the amount is under a cap
the service account the rest of the backend uses	a token scoped to the task and the customer
customer ID or tenant as arguments the model can fill	bound by the controller from the session, never exposed
a generic HTTP tool, a shell, database access with free-form SQL	none: each is a full exfiltration channel and does not belong in an agent that reads untrusted text

How an agent earns wider permissions over time is covered in putting an AI agent inside a workflow you already run. The boundary here holds at every step of that ladder.

Separate the model that reads from the model that acts

The strongest pattern so far is architectural. Willison’s Dual LLM pattern (April 2023) splits the agent in two. A privileged model takes instructions only from the trusted user and can call tools. A quarantined model reads the untrusted content and has no tools. Ordinary code sits between them, stores the quarantined model’s output in variables, and the privileged model refers to those variables by name without ever seeing their contents.

CaMeL, from researchers at Google, Google DeepMind and ETH Zurich (2025), makes this rigorous. The privileged model turns the user’s request into a program, so the control flow is fixed before any untrusted data is read. A quarantined model parses the untrusted data, and a custom interpreter tracks where every value came from and enforces policies when a tool is called, so data from an email cannot be sent to a destination the policy does not allow. On the AgentDojo benchmark it completed 77% of tasks with provable security, against 84% for the same agent undefended. That gap is the honest price: some tasks cannot be done safely in this shape. Beurer-Kellner and colleagues catalogue six related patterns, including plan-then-execute and map-reduce over isolated sub-agents, under one principle: once an agent has read untrusted input, that input must not be able to trigger consequential actions.

For most business agents we build a simpler version. The reader model gets the email and returns a strict schema: an intent from a fixed list, an order number that matches the format, a requested amount as a number. Nothing free-text crosses to the model that holds the tools. A field that fails validation sends the case to a person. How we keep that output reliable is in reliable structured output from LLMs. Free text that a person needs to see, such as the customer’s own words, goes to the screen, not into the acting model’s context.

The trade-off shows up quickly. A real customer who writes “please send the replacement to my office instead” is giving an instruction in untrusted content. In this design the agent cannot act on it directly. It becomes an intent, “change delivery address”, which a person confirms. That is the point, and it is also why these agents handle fewer cases on their own than a demo suggests.

Confirmation gates for irreversible actions

Some actions should wait for a person: sending money, sending anything to a new destination, deleting data, changing permissions, publishing. The gate has to show the action itself, with its real parameters: the recipient address, the amount, the diff. It should not show the model’s summary of the action, because the summary is model output and the injection may have written it.

Gates fail when they are frequent. Meta’s post names this failure itself: a user blindly confirming a warning. If a reviewer approves fifty routine drafts a day, the fifty-first gets the same two seconds. We keep gates for consequential actions only, make each one look different from routine review, and put caps and rate limits behind them so that a missed gate costs a bounded amount.

Allow-listed destinations

Every place data can go should be an allow-list enforced outside the model.

Way out	How data leaves	What the allow-list permits
outbound email	whatever the body says, to whoever is named	addresses already on file for the customer
reply rendered as Markdown	an image URL with data in the query string, or a link	no images; links outside the allow-list are rewritten
fetch tool	whatever is in its URL	listed domains only, and the network egress rules say the same, so a bug in the tool does not open the internet
webhook	whatever it posts	configured endpoints only

Allow-lists are dull and they work, because they do not depend on the model understanding anything. An injection that convinces the model to send data somewhere new produces a refused call and a log line, which is the outcome you want.

Log what the agent read and what it did

When something goes wrong, the question is which piece of content led to which action. The log has to answer it.

Event	Logged as
each source in context	a reference and a hash
each tool call	its full arguments and the result
each gate	whether one was involved and who approved
a refused call to a destination not on the allow-list	an alert, not a debug line
the classifier tripwire firing	an alert
a spike in a tool the agent rarely uses	an alert

This is the same trace you want for cost and quality, and LLM observability in production covers how to keep it. The security use only needs one extra rule: keep the record of what the agent read for as long as you keep the record of what it did.

Red-team test sets

An injection test set is built like any evaluation set from past cases, with one difference. Each case is a normal task with hostile content planted in what the agent reads, and the check is on the state of the system afterwards, not on what the model says. Did a tool get called with a destination outside the list? Did data from another account appear in a reply? Did a refund go through without a gate? AgentDojo works this way, with 97 tasks and 629 security test cases scored by deterministic checks on the environment, partly because a model asked to judge could be hijacked by the same injection.

Write each case from the attacker’s goal and vary where the text hides: the subject line, a quoted earlier message, an attachment, the alt text of an image, white text in an HTML email, another language. Run the set on every model, prompt or tool change. It should also track the false-positive side, the share of legitimate cases sent to review, since a defence that blocks half the real work will be switched off.

If the architecture is right, the set mostly tests plumbing. The model will still be fooled by some cases, and that should show up as refused tool calls and review items, never as a completed action. When a case does complete an action, the fix belongs in a tool, a scope or an allow-list, not in the prompt.

Before an agent that reads outside content goes live, write down every tool it has and mark which legs of the trifecta each one gives it. If all three are present in one session, narrow one in code before launch. The AI agents and automation page describes how we build these, and you can write to us with the agent you are planning.