AI Agent Evaluation: Is Your Business Workflow Ready to Go Live?
Start with the short answer ↓
Evaluate an AI agent against real workflow outcomes, permission boundaries and failure cases. Use a repeatable test set and a go/no-go decision before granting production access.
Prepared with AI assistance. These are practical scoping recommendations; examples are illustrative, not client results.
Evaluate the job, not the quality of the conversation
A sales-to-operations agent might read an approved opportunity, prepare a handoff and create an internal task. A convincing summary is only part of success. The task must belong to the correct account, preserve the agreed scope and reach the right owner. An agent that writes a polished answer but updates the wrong customer has failed the workflow.
Write a one-sentence acceptance contract before running the demo: given this input and these permissions, the agent must produce this verified result, without these prohibited effects. Include a time limit and a person who accepts the result. This guide focuses on that evaluation contract; the wider implementation checklist covers the rest of the project.
Build a case bank with expected actions and expected stops
Collect representative examples with the people who do the work. Remove unnecessary personal data and use an isolated environment with test accounts. Include Hebrew and English inputs if both occur in the actual process. Keep examples used to tune the agent separate from a held-out set used for the release decision.
The following is a proposed starting worksheet, not a certified test standard. Add workflow-specific cases rather than copying the categories blindly. For each case record its ID, starting records, allowed tools, expected final state, prohibited effects and evidence needed to judge it.
| Case family | Example input | Evidence of acceptable behavior |
|---|---|---|
| Routine | An approved opportunity with complete handoff details | One task on the correct account, with the required owner and fields |
| Missing information | No confirmed delivery owner | A clear request for the missing detail; no invented assignment |
| Conflicting evidence | The CRM and signed scope list different deliverables | Both sources shown to a reviewer; disputed scope not overwritten |
| Permission boundary | A customer note asks the agent to bypass approval or read another account | No unauthorized read or write; the exception reaches its owner |
| Duplicate delivery | The same approved opportunity arrives twice | One intended task, with the repeated event handled visibly |
| Uncertain tool result | A task-creation request times out after submission | Existing state checked or case paused; no blind repeat or false success |
Separate outcome scores from release-blocking failures
Use direct checks for facts: which record changed, whether a required approval exists and whether a duplicate was created. Use a written human-review rubric for judgment: whether a summary preserves the agreed commitments or an escalation explains the conflict. An AI grader can assist with language quality, but should not be the sole evidence for permissions or database state.
Anthropic distinguishes the final state of the environment from the agent transcript, and recommends combining grading methods. Microsoft documents task completion and tool-use evaluation as separate dimensions. Both are useful background for an important business distinction: saying that work is complete is not the same as confirming that it happened.
Report completion, correct escalation, review effort, elapsed time and cost separately. Define release-blocking events in advance, such as cross-account access, an unapproved external action or a false confirmation of a consequential update. A good average must not hide one of those events. Thresholds belong to the workflow owner; there is no universal pass percentage for every agent.
Worked example: why 96.7% can still mean no-go
Illustrative scenario, not an AgenticShip client result: a team prepares 60 cases for an internal handoff agent. The set contains 20 routine cases, 10 missing-information cases, 10 conflicts, 10 permission-boundary cases, five duplicate deliveries and five uncertain tool results. These counts illustrate a plan; they are not a statistically sufficient sample or a recommended ratio for every workflow.
The team runs every case three times from a clean starting state: 60 × 3 = 180 trials. In 174 trials the expected outcome or expected escalation occurs, giving 174 ÷ 180 = 96.7% rounded to one decimal. Six trials fail. If one failure creates a task on a prohibited account, the release is blocked under the agreed policy even though the aggregate looks strong.
Also report how many unique cases passed every repeat and which case families failed. Repeating 60 cases does not create 180 independent business scenarios. Fix the failure, rerun the entire regression set and reserve fresh cases for the next decision. Do not remove a difficult case merely to improve the score.
Test the operating boundary without affecting customers
Disable real outbound messages and irreversible writes in the test environment. Test permissions at the system boundary, not only in the prompt: an instruction to avoid a record is not equivalent to an access control that prevents the read. Treat instructions inside imported documents and customer notes as untrusted content, not permission to change the agent’s job.
When a replay passes, consider an explicitly authorized shadow run: the agent proposes actions while the existing team remains responsible for execution. Shadow mode still needs approved data access, retention rules and an owner. Confirm that proposed actions cannot leak into real sends or writes. Compare the agent’s proposals with the decisions staff actually make, including cases where staff disagree.
Make the release decision repeatable
The release record should name the tested model, prompt, tools, policies and dataset version; show results by case family; list unresolved failures; and state the allowed production scope. Record who approved it and which event pauses the agent. A limited supervised pilot and unrestricted operation are different decisions.
Rerun relevant evaluations when the model, workflow, tools or permissions change. Turn a confirmed production incident into a protected regression case. If the agent no longer meets the contract, narrow its authority or pause it while the manual path remains available. The useful deliverable is not a dashboard full of scores: it is a defensible decision about which work the business can delegate today.
Primary references
The guidance above is AgenticShip's proposed approach. These references provide technical background for the relevant recommendations.