AI Agent Threat Modeling: Trust Boundaries, Tools and Recovery

Threat modelling does not have to begin with a forty-page document. Start with one ordinary story. An agent receives an email, searches internal records, calls a CRM tool, and sends a reply. Now ask what each step can see, who supplied the information, and what could go wrong if one part of the story is dishonest or simply out of date.

Draw where trust changes

Mark the boundaries between user input, retrieved content, the model, the tool gateway, a third-party service, and a human approval screen. List the data, accounts, queues, and business actions within reach. For each boundary, decide what is trusted, what is only evidence, and which service is responsible for the permission decision.

Walk through believable abuse

A document may tell the agent to reveal a secret. A low-privilege user may try to make it act as a more powerful service. A tool may accept an unrestricted URL or record ID. A timeout may make a retry repeat an action that already succeeded. A retrieval query may return one tenant’s information to another. These scenarios are specific enough to test.

Choose controls that match the story

Narrow tools, tenant-scoped queries, server-side authorization, outbound allowlists, approval for high-impact actions, idempotency keys, timeouts, and rate limits are practical answers. Keep secrets out of prompts and model-visible logs. Treat webpages, tickets, emails, and uploaded files as untrusted content even when they look official.

Finish with the recovery question

Decide how to pause the queue, revoke a credential, disable one workflow, notify affected people, and compensate for an action that cannot be undone. Preserve enough evidence to understand the incident without keeping unnecessary personal data. Revisit the model when the agent gains a new tool, data source, or business capability. The best threat model is the one that changes the design before an incident has to.