Claude Code vs OpenHands: 48 real coding tasks, one bar
We ran Claude Code and the open-source OpenHands through the same 48 real coding tasks, on the same model, graded by tests neither engine saw. Both passed 47. Claude Code was about twice as fast and half the cost.
The open-source engine matched Claude Code on quality. On 48 real coding tasks, graded by tests neither engine saw, each passed 47. Both passed all 18 hard ones, with zero guardrail violations. Claude Code finished in about half the time, at about half the cost, on the same model.
That's the result we wanted to know before putting OpenHands on real work. The factory doesn't build its own coding agent: it runs the best ones and holds them to one bar. This is that bar, applied.
The results
| Claude Code | OpenHands (open source) | |
|---|---|---|
| Tasks accepted | 47 of 48 (98%) | 47 of 48 (98%) |
| Accepted on the first try | 47 | 47 |
| Hard tasks (larger app) | 18 of 18 | 18 of 18 |
| Tasks it was never tuned on | 15 of 16 | 16 of 16 |
| Median time to a pull request | 1.6 min | 3.4 min |
| Median cost per task | $0.38 | $0.79 |
| All 48 tasks | $20.35 | $37.43 |
| Guardrail violations | 0 | 0 |
Each engine missed one task, and a different one. Claude Code missed a schema change (tags on documents). OpenHands missed a small display change (upload dates in the library). Both engines ran the same model, Claude Opus 5.5, so the cost difference is the engine: OpenHands takes more steps per task.
How we tested it
-
48 real tasks. These are requests the factory has actually handled, replayed on two practice apps:
- 30 on a small document-sharing app;
- 18 on a larger app of about 60 files, where finding the right code is part of the job.
They range from a one-line copy change to a feature that touches the database, the interface and background jobs.
-
Hidden tests. Each task has one test per requirement, and the engine never sees them. A task counts only when every test passes on the pull request the engine opened.
-
The same conditions. Same day, same runner, same model, same guardrails, same budget. Each engine gets up to three tries with the failing tests' feedback, the way the factory's QA step sends work back.
-
A harness that checks itself. Applying the known-good change passes 48 of 48, and doing nothing passes 0. A pass means the work was done.
What it means
- Open source is a real option. A client who needs an open engine, or needs their code to go only to a model provider they choose, gives up nothing on quality here.
- Claude Code stays the default. It's faster and cheaper, so workspaces that don't need an open engine keep it.
- Every run keeps being measured. OpenHands now takes real work on one of our own workspaces, and its results land on the Engines page next to Claude Code's.
What this doesn't show
- One run per task. With 48 tasks, a 98% result has a 90% confidence range of 91% to 100%. The two engines are even, and neither is "better".
- Our tasks, not yours. Results on your own code will vary, and the Engines page tracks that run by run.
- One model. OpenHands can run other models. That comparison is next.
- One fix to our setup. Both engines started at the same moment against one practice repository. Three OpenHands pull requests were rejected because Claude Code had just pushed a branch of the same name, though OpenHands' work had already passed its checks. We re-ran those three alone, and all three passed first time.
Run the same eval on any engine: the factory eval. Docs: Coding engines.