Open benchmark · loading…
How often does an AI agent get a browser task right, and what does it cost?
We give coding agents real tasks on live websites and swap only the browser tool. The model, harness and prompt stay the same for every tool. An independent judge decides success. Every run, prompt and judgment is published.
Leaderboard
The newest version of each tool that ran every task. Click a column to sort. Cost, tokens, time and tool calls are medians per task. Tokens include cache reads and writes.
The shaded band is the leader's 95% confidence interval. A tool whose interval overlaps it is not reliably worse: with this many tasks, the gap could be chance.
Cost vs. success
Each point is a tool. The line through it is its 95% confidence interval. Up and to the left is better.
passed
passed in some runs
failed
not run
Methodology
The full description, including every way we keep it fair, is in the README.
Tasks we didn't write
The tasks are a fixed random sample of 100 tasks from Online-Mind2Web (COLM 2025): 300 tasks on 136 live websites, written by researchers at Ohio State. The sample keeps the dataset's mix of easy, medium and hard tasks and uses a fixed seed, so anyone can recompute it. No tool vendor, including us, picked these tasks.
An independent judge
The benchmark's own judge, WebJudge, decides success with o4-mini. That model comes from another family than the agents. It sees the task, the commands the agent issued and screenshots the harness took after every step. It never sees a tool's own output. Every judgment and its reasoning is published.
Only the tool changes
Every tool gets the same model, the same harness (Claude Agent SDK) and the same system prompt, plus one sentence on how to reach the browser. Tools that ship an agent skill get it. Built-in web access is switched off. Every tool runs in the same job, from the same IP address, in a seeded random order.
What we measure
Success is the share of tasks the judge accepts, with a 95% Wilson interval. Cost, tokens (input, output and cache), wall-clock time and tool calls are medians per task. Cost comes from the provider's prices. Tool versions are pinned and recorded with every run.
Limitations
- Live websites change, block bots and show CAPTCHAs, so runs are not exactly repeatable. A blocked run counts as a failure for every tool it hits.
- 100 tasks and one run per task give intervals about ±10 points wide. Close results are ties.
- WebJudge disagrees with human reviewers on about one run in seven.
- A tool that does several steps in one call leaves fewer screenshots for the judge to check.
- This benchmark is maintained by the WebdriverIO project, which builds two of the tools. That is why the code, prompts, raw data and judgments are all public. If a setup looks wrong, tell us.
Reproduce it
Every number on this page comes from a workflow run you can inspect. To run it yourself you need Node.js 24 and Chrome:
git clone https://github.com/webdriverio/benchmark
cd benchmark && npm install
npm run bench -- --suite online-mind2web --runs 1
node src/judge.ts results/<id>
Online-Mind2Web is gated on Hugging Face. Accept its terms, then set HF_TOKEN. The judge needs OPENAI_API_KEY.
Tested versions
Every run of a tool version with this model counts toward its row, across workflow runs. Open a version for its results per task and the runs behind them.
Run log
Every published workflow run for these tasks and this model, newest first.
Cost vs. success, results per task, methodology and every run on benchmark.webdriver.io →
No published benchmark runs yet.