Testing Computer-Use Agents on Real Accounting Work
I spent the last few months building cua-bench, an evaluation environment that puts computer-use agents in front of a real ERP system and asks them to do accounting work: create a purchase order, record an expense accrual, reconcile a bank statement against the general ledger, or investigate a trial-balance misclassification and post the correction. The agent sees screenshots of a Chromium window running ERPNext v15 and acts by clicking, typing, and scrolling. When it stops, a grader reads the resulting accounting state and scores it.
Most of this post is about the grader, because that is where the interesting engineering is. It scores the ledger the agent leaves behind, not the agent’s transcript, and most of the design work went into making that score hard to game.
Benchmarks like OSWorld and TheAgentCompany already put agents in front of real software and grade them on execution or checkpoint state, and tau-bench scores the final database instead of the dialogue transcript. cua-bench’s addition is narrower: state-based grading hardened against reward-hacking, run against a real accounting system that the agent drives entirely through screenshots.
The environment
cua-bench ships as Harbor tasks. Each task packages a pre-seeded ERPNext stack, an agent-api that runs inside the same container, and a grader. The agent-api is a small FastAPI service driving a headless Chromium through Playwright, and it exposes three endpoints on port 5000: /reset restores the database to a golden snapshot and starts an episode, /step applies one action, and /evaluate runs the grader. The agent sees PNG screenshots at 1280 by 800 and issues actions in a canonical schema, with per-vendor translators mapping each lab’s native computer-use format onto that schema so one grader scores every model the same way.
There are 25 task packages in the repo. The sweep in this post covers 19 of them, spanning purchase orders, journal entries, sales invoices, three-way matching, bank and subledger reconciliation, and a handful of multi-step investigations. The six left out are two canary packages that test the harness itself, and four tasks I swept once and cut for saturating at or near 1.00 with both vendors, kept in the repo as design notes.
Grading on state, not on the transcript
A computer-use agent leaves behind two things a grader could read: a transcript of what it did, and the state of the system after it stopped. Transcript grading is tempting because it is legible, but it rewards the appearance of work. An agent that narrates a correct-sounding plan, or fills a form that looks right but never posts, can score well without ever changing the books. So cua-bench grades the end state. After the agent stops, the grader reads the ERPNext documents and the general ledger through the Frappe REST API and checks a set of weighted checkpoints. Partial credit reflects how much of the correct final state the agent actually produced.
Most of the anti-gaming work comes down to a few grader decisions.
Checkpoints cross-validate against general-ledger entries instead of trusting a single document field. A submitted invoice with the right numbers on screen still fails if the matching GL entries are not there. The near-miss this catches is concrete and common: a form filled in correctly but never submitted, which looks finished on screen while nothing has posted to the ledger, so the checkpoint stays red.
Document queries are then scoped to a creation timestamp captured at episode start. The environment is seeded with realistic golden data, and without this scoping an agent could collect credit for records that were already present, or for a document it happened to open but did not create this episode. Scoping to the episode makes the grader count only the work done in this run. The scoping is applied selectively, which matters for reconciliation tasks whose correct move is to modify or cancel a pre-seeded record: those records are graded by reading them back by identity and checking their final state, so a cancelled invoice registers as docstatus 2 and a corrected payment leaves the pre-seeded invoice’s outstanding balance where it belongs, while the creation-timestamp filter is held in reserve for the anti-hack checks that have to tell an agent-created document from a golden one.
A third guard runs when the grader loads: existence-only checkpoints are rejected outright. A shared helper refuses to register any checkpoint that would pass merely because a document exists, or because a filter-redundant field matches, such as a supplier name the query already filtered on or a docstatus that is always 1 for a submitted record. Every checkpoint has to test at least one field the agent had to get right, which removes the free credit a padded rubric of trivially-true checks would otherwise hand out.
On top of those, anti-hack penalties fire on the shortcuts that reach the right number the wrong way: a duplicated document, a relaxed over-billing tolerance setting, a cancelled invoice standing in for a real correction, a posting made against a pure timing difference, or a reclassification booked into the wrong account. In this sweep they stayed rare, firing on 13 of the full log’s 534 scored rollouts and only on four tasks; the cluster was pi-misapp, where every scored rollout from both vendors tripped the fabricated-extra-invoice penalty by opening a new purchase invoice instead of cancelling the misapplied one. Penalties should be rare: one only fires when an agent reaches a plausible end state through a shortcut, and most failures are just incomplete work that scores low on its own.
The trial-balance task, tb-invest, shows why the wrong-target-account penalty exists. The agent has to find two misclassified amounts and post a correcting entry that moves each one out of the account it is wrongly sitting in and into the account it belongs in. The grader checks both halves of each correction on their own: that the source account is reduced, and that the intended target account is increased, plus a total-target-delta check across the episode. So an agent cannot earn a finding just by making the trial balance net to the right total. If it reduces the source but parks the money in the wrong account, the target checkpoint stays red.
State grading tells you whether the books came out right. A separate transcript-level predicate library in the repo looks at how an agent tried to cheat when it did, but the reward depends only on the state side, so that is what this post covers.
What the models did
The sweep recorded 854 rollout records across four models: two Anthropic generations (claude-opus-4-6 and claude-opus-4-7), OpenAI’s gpt-5.4, and Google’s gemini-3-flash-preview. Those records span 28 tasks; the 19 with mature graders are what this post reports, and they hold 794 of the records. The other 9 were the four saturating tasks mentioned above plus five early tasks that were cut entirely and no longer ship in the repo; most of their scored rollouts passed, which is why they were cut. Of the 794, 483 produced a graded reward and the rest errored or hit a step or quota limit before scoring. Across the scored rollouts on the reported tasks the overall pass rate, where a pass is a reward of 1.0, is 0.45, or 218 of 483.
Broken out by model, those scored rollouts land like this:
| Model | Scored n | Pass rate | 95% CI |
|---|---|---|---|
| Anthropic (Opus 4.6 + 4.7) | 267 | 0.35 | 0.29–0.41 |
| OpenAI (GPT-5.4) | 204 | 0.61 | 0.54–0.68 |
| Google (Gemini 3 Flash), excluded | 12 | 0.00 | 0.00–0.24 |
The intervals are Wilson 95%, and the three scored counts add up to the 483 above, so the table, the per-task chart, and the repo’s failure-rate tables all reconcile. Counting the smoke and cut tasks too, the full log scored 534 rollouts with 248 passes (0.46).
Discarding 37 percent of runs could reshape the comparison, so I checked the 320 rollouts that never reached a graded reward. They split as 189 of 482 for Anthropic (39 percent), 110 of 339 for OpenAI (32 percent), and 21 of 33 for Google (64 percent). In the run summary these all carry a null reward and an unknown stop reason, meaning they failed before the grader ever ran. The operational notes attribute the bulk to sandbox instability at concurrency (Playwright launch failures, Frappe containers stuck starting, a /reset returning 503), which sits at roughly a 20 percent baseline, with a handful of hard-killed processes (exit 137) on top.
Rollouts that exhausted a step, wallclock, or repeated-action budget are a separate category and stayed in the numbers above, because the grader still scored them, usually low. The attrition is a property of the sandbox infrastructure, and it lands at close rates on the two models being compared, so dropping it is unlikely to move the Anthropic-versus-OpenAI gap. One asymmetry: Anthropic emits a single action per response and runs longer per rollout, so it absorbs slightly more of the timeout and hard-kill exposure, which if anything would depress its scored set.
I dropped Google from the comparison. Every gemini-3-flash rollout that scored returned 0.00, driven by API quota throttling at concurrency, so leaving it in would have read as a dead column. Those 0.00s are real scored runs: under throttling the model’s calls kept failing, so the agent spent its step budget without ever changing the ledger, and the grader then ran against the untouched golden state and returned zero. The Anthropic figures aggregate both opus generations, which matters on one task below.
Sample sizes vary a lot from cell to cell, from a single rollout on the least-swept tasks up to n=50, because the summary pools rollouts across several dated sweep rounds under a deliberate policy: n=3 to validate a new task, n=1 or 2 for regression checks, more only when a task looked bimodal or sat on a capability wall. The tasks I kept re-running while hardening graders or probing plateaus carry far more rollouts than the procedural ones that passed on the first try.
What I did not expect was how far apart the models land on specific tasks while both collapse together on others. On je-reversal, an accrual plus its next-period reversal, OpenAI passes every rollout (1.00, n=10, 95% CI 0.72 to 1.00) while Anthropic passes one in five (0.20, n=25, 95% CI 0.09 to 0.39), and the intervals do not overlap. Every passing Anthropic rollout there is claude-opus-4-6.
tb-invest shows why the aggregated Anthropic number can mislead. Anthropic scores 0.38 (n=37, 95% CI 0.24 to 0.54) against OpenAI’s 0.76 (n=21, 95% CI 0.55 to 0.89), but that 0.38 hides a generation gap. claude-opus-4-6 scores 0.17 (n=23) while claude-opus-4-7 scores 0.71 (n=14), close enough to OpenAI’s 0.76 to be indistinguishable at this sample size.
Two tasks are walls for both labs. On pi-price-var, a purchase invoice whose rate has to be overridden away from the value the form fills in, Anthropic passes 0 of 50 and OpenAI 0 of 19. On pi-3way, a three-way match with a quantity discrepancy across three linked documents, Anthropic passes 0 of 30 and OpenAI 0 of 13. Both tasks come down to overriding a defaulted value or reconciling a quantity across a document chain, and every model I tested plateaus just short of it.
Cost
Cost per rollout is asymmetric by more than an order of magnitude. Anthropic averages $8.02 per rollout and runs up to $75.80 on the hardest reconciliation tasks, against OpenAI’s $0.55. The gap says nothing about capability and stays out of the reward, but it decides whether a benchmark like this is affordable at scale, so I report it next to the pass rates.
Code
cua-bench is on GitHub at github.com/vaiyr/cua-bench, with the tasks, the graders, the multi-vendor runner, and the sweep summary that every table here regenerates from. The task I most want someone to beat is pi-3way, the three-way match with a quantity discrepancy across three linked documents, where every model I tested still passes zero of its rollouts. The seed, the grader, and the failing transcripts are all in the repo if you want to try.