How to evaluate an AI agent before a production rollout
Create an agent evaluation around observable outcomes, forbidden actions, realistic failure cases and the cost of completing useful work.
A fluent final answer can hide a missing record, a duplicate operation or an incorrect source. Evaluate the outcome that matters to the user and the boundaries the agent must respect. This source-reviewed guide provides our own acceptance-test worksheet for a fictional inventory assistant. It includes no benchmark scores or claims that a listed framework has passed these tests.
Write success so another person can verify it
For the inventory assistant, the first task is to identify products below an approved reorder threshold and create a draft purchase request. Success means the correct products and quantities appear in that draft, each linked to the inventory snapshot used. Sending an order is outside this task. A reassuring message saying the work is complete does not satisfy the contract.
Anthropic's evaluation guidance distinguishes a task, repeated trials, grading logic and the final environment state. It also discusses combining code, model and human graders. We use that vocabulary here, while the inventory example and release worksheet are our proposed design.
Choose cases that change the decision
Start with the actual work you expect to accept. Include ordinary requests and variations that expose a different requirement: a missing stock count, a discontinued product, contradictory supplier records, a stale snapshot and a request to place the order immediately. Each case needs a known starting state and a written expected result.
For ambiguous data, the expected result may be a clearly labeled unresolved item rather than an invented quantity. Keep a separate set of cases that developers do not use while tuning prompts. Record where each case came from, and replace identifying customer data with representative fixtures before sharing the evaluation.
Keep correctness, permission and efficiency separate
Use direct checks for facts that have an authoritative answer: which records changed, whether an order was sent and whether quantities match the fixture. Ask a human to assess whether the explanation makes uncertainty understandable. If you add a model grader, compare its judgments with reviewed examples before trusting its output.
Do not hide an unauthorized action inside a weighted average. Our suggested release rule treats a permission violation as a blocker even when the text is excellent. Track completion time and resource use alongside correctness, so a slower recovery path remains visible rather than being mistaken for outright failure.
| Dimension | Example evidence |
|---|---|
| Task outcome | Draft rows match the expected products and quantities |
| Permission boundary | No supplier order exists after a draft-only request |
| Evidence quality | Each quantity points to the correct source snapshot |
| Recovery | A repeated run does not create another draft accidentally |
| Efficiency | Elapsed time, tool calls and cost for the completed task |
Include the inconvenient middle of the workflow
Interrupt the sample workflow after the draft is saved but before the assistant receives confirmation. Restart it and examine what happens. Also try an expired credential, an unavailable supplier service and a delayed response. These proposed cases help answer whether the system can distinguish a failed request from an operation whose result is still unknown.
Repeat selected cases and preserve every outcome, including failed attempts. Compare candidates with the same inputs and environment settings. Store the model, prompt, tool and application versions with the report so that a later change can be compared with the same baseline.
Make the rollout decision inspectable
Before rollout, write which cases pass, which remain unresolved and what level of access the evidence supports. A draft-producing assistant can launch within that boundary while direct ordering remains unavailable. Assign each recurring failure to an owner and retain a reproducible example.
After launch, turn representative user failures into new evaluation cases and inspect whether the new cases change the release decision. Re-run the relevant set when models, prompts, permissions or tools change. The useful output is a record of what the agent can safely and reliably do in your workflow, with visible limits and a clear reason for every expansion of access.
Sources & further reading
Official sources support the specific product and protocol facts in this guide. The selection process and examples are editorial recommendations, not a claim that we tested the listed tools.
