A vendor demo can make every AI agent look production-ready. The difficult part begins after the demo, when the agent meets your real documents, permissions, exceptions, and support team.

Use procurement to test the whole operating model, not just the quality of a few impressive answers. The questions below are designed to reveal what you would actually own after launch.

Ask what the agent can see and do

• Which sources can it retrieve from, and how are permissions enforced at retrieval time?
• Can it call tools, send messages, create records, or trigger workflows? Which actions require approval?
• Can access be scoped by tenant, team, role, or record?
• What happens when a source is deleted or a user’s permission changes?

“The model cannot see that data” is not a sufficient answer. Ask where prompts, retrieved passages, tool results, transcripts, logs, and backups are stored, and who can access each one.

Ask how quality is measured

Ask the vendor to evaluate your questions, not only a benchmark. Can you supply a private test set? Can you inspect the retrieved sources? How are unsupported answers, missed escalations, and unsafe tool calls recorded? What is the process for adding a failed example to the next evaluation run?

A strong answer includes a way to compare versions before release. A model change that sounds better in a demo may be worse for your most important workflow.

Ask about operations and cost

• What are the separate costs for model calls, retrieval, tools, storage, and support?
• What limits exist for context size, message volume, concurrency, and file ingestion?
• How are latency spikes and provider outages handled?
• Can you export prompts, knowledge sources, conversation records, and configuration?
• What is the incident process, and how quickly can an agent or tool be disabled?

Get a realistic estimate using your expected conversation length and tool usage. “Per user” pricing can hide the costs that matter most in a busy workflow.

Ask what happens when you leave

Request a written exit plan. You should know how to retrieve your content, delete it, rotate credentials, remove integrations, and keep the links that users already rely on. A platform is easier to trust when it makes portability ordinary rather than awkward.

Finally, ask for a small pilot with a clear owner and a fixed success measure. You are not buying a chatbot; you are deciding whether a new operational system deserves access to your business.