AI agent metrics are useful when they help a team decide what to improve, pause, or scale. A single success rate hides too much: an agent can complete many easy requests while failing badly on the cases that matter most.

Measure the outcome first

Start with the workflow result rather than model trivia. Track task completion, useful resolution, time to resolution, rework, escalation quality, and the cost of reaching a satisfactory outcome. If the agent supports revenue or service delivery, connect those measures to the business process without claiming that every outcome was caused by the agent.

Add quality and safety measures

Review correctness, relevance, completeness, citation or source use, refusal behaviour, and the severity of errors. For tool-using systems, measure correct tool selection, argument validity, approval compliance, retries, and recovery. Keep separate thresholds for sensitive data, external communications, financial actions, and irreversible changes.

Make the numbers actionable

Define an owner, a review period, a target or warning threshold, and the response when a measure moves. Segment results by intent, audience, language, source set, model route, and escalation path. The pilot criteria guide helps turn metrics into a decision rule, while the drift guide covers changes after launch.

• Keep a representative evaluation set beside live metrics.
• Report severe failures separately from averages.
• Measure user corrections and repeat contacts.
• Review cost and latency alongside quality.

Good evaluation metrics create a shared operating language. They do not make an agent safe by themselves; they make weak assumptions visible early enough to change the design.