Write one row per test case before the agent goes live. Each row needs an input, the expected action, the prohibited action, whether a person must approve, the evidence the agent has to read back, who owns the failure, and a pass or fail decision made by a named person. Writing those fields exposes disagreements about what the agent is allowed to do before you give it access.
Copy the worksheet below into your spreadsheet and agree the cases with your provider before running them. The result fields stay empty until you have evidence to record.
What to record for each case
Input
Use real input. Take the exact request, record or message from your inbox or queue, with names changed if you need to.
Expected action
What the agent should do, written so two people would agree whether it happened. For a practice refund case, specify the test order, the requested amount and that the request must remain pending. The reviewer can then check the order record and its state.
Prohibited action
What the agent must never do on this input. In the practice refund case, completing the payment before approval fails the test even if every other field is correct.
Approval requirement
Whether the action can complete on its own, or has to wait for a named person. Money, pricing, contract terms, deletions and account changes normally wait.
Evidence and readback
What the agent must show after acting: the record ID it wrote, the message it sent, the state it read back from the system. Without this column you are grading the agent's own summary of its work.
Failure owner
Name the person who picks the case up when it fails. Use one name.
Status and observed result
Each template starts with status not_run and a blank observed result. After running it, record what happened and who reviewed it in the separate columns. You can then filter unfinished cases.
The acceptance worksheet download below ships with ten blank case templates covering the common shapes: a routine in-scope task, an out-of-scope request, a missing-data case, an approval-required action, a destructive request, an ambiguous input, a duplicate, a system error, a handoff to a person, and a repeat run of a case that already passed.
Microsoft's agent evaluation checklist describes a test case the same way, as a prompt with an expected response and written acceptance criteria, and it adds a second row type for questions the agent should refuse and route to a human. LangChain's readiness checklist makes the same point about ambiguity: if two experts cannot agree on pass or fail, the case needs rewriting, and someone has to own it. Anthropic's guidance on building effective agents recommends extensive testing in sandboxed environments with guardrails around autonomy.
Set the passing rule for each case before running it. For a case that requires approval, completing the action without that approval is a failure, even if the agent passes every answer-quality test. Keep that result visible on its own row instead of hiding it inside an average score.
A worked example, and exactly what it proves
Our read-only Search Console build record, published on 17 October 2026, inspected three URLs and saved the result as a report file. The next day we ran a small checker against that saved report.
The checker accepted the real saved report. We then fed it eight deliberately broken copies of the same report: one with a URL missing, one with a URL duplicated, one where an index verdict was not PASS, one where a canonical pointed somewhere else, one with a performance row belonging to another site, one where the declared total did not match the saved rows, one where an incomplete data window was labelled complete, and one that reported an external write inside a read-only scope. All eight were rejected, with the reason printed for each.
You can run it yourself. Download the checker and the saved report, put them in one folder, then run:
python3 ai-agent-acceptance-check.py search-console-review-2026-10-17.json --self-test
This tests a deterministic validator against a saved file. It does not test a language model, live permissions, a real blocked write, prompt-injection defenses, agent reliability or production security. The eight broken inputs are synthetic mutations we wrote on purpose; none of them is an incident we observed in production. A report that declares zero external writes cannot enforce that boundary. Test the connected account's permissions and the application's approval checks separately.
You can inspect the passing rules in the script, run the same checks and compare the output with the saved result. Ask your provider to show the same connection between a written rule, a test input and the observed result for your workflow.
What you still have to test on the real system
Three things the worksheet covers that a file checker cannot:
- Send the real inputs to the real agent in a sandbox or a copy of your data, and record what it actually did.
- Try an action that requires approval and confirm it stopped and waited for the named person. Then confirm the person was actually notified. A permanently prohibited action must stay blocked even when someone asks the agent to proceed.
- Simulate a failed connection or expired token in a test environment. Check that the agreed retry limit holds and an unresolved failure reaches the named person. Leave production credentials alone.
Repeat the cases with the same starting conditions and with realistic input variations. Agree the repeat count for the risk involved. If a repeat fails, keep that failure in the record and investigate it before expanding the agent's access.
Where to start
Start with the cases that would hurt most if the agent got them wrong, then cover the ordinary work and the failures it must hand back. Fill in the failure owner for every row and agree which cases must pass before launch. Leave anything untested outside the agent's approved scope.
If you want to walk through this on your own workflow, you can book a 20-minute AI employee call.
Sources and files
- Acceptance worksheet, CSV
- The checker script
- The saved Search Console report it validates
- The checker run output, 18 October 2026
- Build record: a real agent workflow with owner review
- Microsoft, review the agent evaluation checklist
- LangChain, agent evaluation readiness checklist
- Anthropic, building effective agents
Related reading: how we run a build, human approval is infrastructure, how to choose an AI implementation partner.
This article describes operating principles and public sources. It is not a client result. Build Records carry the basis for every Outproduce system claim.