Measure the work people finish

You ask an expense assistant to submit a reimbursement report. It reads your receipts and says the report is complete. In the expenses system, though, the report is still a draft. Two attachments are missing, and a personal dinner has been included.

A test that ends at the assistant's answer could miss all three problems. Receipt extraction and arithmetic need checking, but the task continues into another system. Completion means a submitted report with the right expenses, the supporting receipts and a reference you can find later.

Benchmarks help teams compare approaches and find weaknesses. A product needs tests drawn from its own work, too: the documents people upload, the corrections they make and the conditions under which a submission can fail.

Check the saved result

Anthropic's January 2026 guidance on agent evaluations¹ distinguishes the record of an interaction from the resulting state of the world. An agent saying it booked a flight and a reservation existing in the database are two separate things to check. The same distinction applies to the expense report.

I would write the completion criteria before polishing the conversation. Which expenses belong in this report? What confirms that it was submitted? If a receipt is unreadable, the assistant should identify it and keep the entries already checked. Asking for a replacement is a reasonable outcome. Calling the report complete would mislead the person waiting for reimbursement.

Include the ordinary mess: a hotel receipt with a personal charge, two photos of the same taxi receipt, an upload that fails halfway through. For each case, name the expected result and where to look for proof. A saved attachment or submission ID gives the team something firmer to inspect than a reassuring reply.

Record the repairs

Someone may finish the report after checking every amount, fixing categories and restarting an upload. Record those repairs alongside the final result: what went wrong, how it was discovered and how long it took to fix. Otherwise, successful submissions can conceal how much work the assistant left behind.

Some checking belongs in the task. Reviewing a large expense before submitting it is sensible. Showing the receipt beside the amount makes that review easier. Making someone search a conversation for the same receipt adds effort without helping them judge the expense. Counting clicks alone would treat both kinds of activity alike.

The researchers behind τ²-bench² tested coordination in a telecom setting. An agent could change service settings while guiding a simulated user through actions on a phone. Tested models performed worse when they had to coordinate with that user than when they controlled all the tools. The simulated user was another model, so this does not tell us how real customers would fare. It does identify a capability that a tool-use test can overlook: helping someone else complete their part.

Corrections deserve their own tests. In a 2025 Microsoft and Salesforce study³, researchers split instructions across several turns. Models often made early assumptions and struggled to recover as more information arrived. These were simulated conversations with a fixed structure. In a product study, people should be able to clarify a request in their own words while the task is underway.

Try “the second dinner was personal” after the report has been created. Check the saved total and the treatment of that receipt. Show which entry changed so the person can review the correction without checking the whole report again.

Check what people believe happened

Google's 2010 HEART paper⁴ connects product goals to observable signals and then to metrics. Its task-success category includes effectiveness, efficiency and errors. For this assistant, the goal might be to help employees submit correct expenses with less effort. Conversation volume would say little about whether that happened.

Watch people attempt the task, then ask what they believe has happened. Compare their answer with the saved result. A report can be correct while its status is unclear. Someone who thinks a draft has been submitted may close the application and miss a reimbursement deadline.

Keep consequential failures visible in the results. A category corrected before submission and a duplicate reimbursement request should remain distinguishable, even if a dashboard combines them into an error rate. An improving average could otherwise hide a rise in duplicate submissions.

The status shown in the interface needs to agree with the state being tested: saved, submitted or returned for correction. When a research session exposes a missing case, add it to the test set. The team can then reproduce that failure when the interface or model changes.

If the assistant says “submitted,” the report should be there. The person should know where to find it and whether anything still needs their attention.