AI Agent Vendor Scorecard: Compare Platforms on What Matters

A polished demo can make an agent platform feel finished. It may not show what happens when a tool times out, a tenant asks for another tenant’s record, an approver is away, or usage doubles overnight. A vendor scorecard is a way to slow that decision down just enough to see the parts that matter.

Ask what happens around the model

Look at data boundaries, retention, deletion, training use, residency, and export. Ask how permissions, approvals, secrets, environments, audit trails, evaluations, retries, queues, rate limits, and incident controls work. These details are not separate from the product. They are the product once a real customer depends on it.

Give portability its own line. Can you export prompts, tools, data, traces, and workflow state? Can another team understand the configuration six months from now? A low starting price is less attractive if leaving later means rebuilding everything from screenshots.

Ask for your workflow, not their favourite demo

Give the vendor a small scenario that resembles your work: retrieve a tenant-scoped record, draft a response, request approval, perform one idempotent update, and recover from a timeout. Ask to see the trace, the permission decision, the evidence used, and the failed state. A demo that skips those parts is still useful for seeing the interface, but it is not evidence of production readiness.

Make the cost honest

Include model tokens, retrieval, tool calls, storage, human review, observability, support, minimum commitments, and the cost of a provider outage. Ask what happens when demand doubles. The cheapest model line is not necessarily the cheapest workflow.

Run a small pilot with an exit

Choose one workflow, use representative data, define accuracy and safety thresholds, and include negative tests. Decide what must be exportable before the pilot begins. Choose the platform that gives your team convincing evidence and a workable escape route, not simply the one with the longest feature list.