Web Bench: A Fresh Approach to Comparing AI Browser Agents (Updated June 2026)

Every agent vendor right now claims state-of-the-art performance on browser tasks. Skyvern, Browser-use, OpenAI's Operator are all shipping numbers that look strong on paper. But the benchmark most teams use to compare these agents, WebVoyager, covers just 643 tasks across 15 websites. That's a narrow slice of the internet, and it mostly tests agents on read-heavy tasks where the real friction hasn't kicked in yet. The harder problems are logging in, solving 2FA, filling out forms, and downloading files, to name a few, which are underrepresented or missing entirely. And WebVoyager says nothing about the browser infrastructure running underneath each agent: whether it can get past Cloudflare, handle a CAPTCHA, or even load the target page in the first place.

So, we built something better: Web Bench. This benchmark covers 5,750 tasks across 452 websites, separates read tasks from write tasks, and measures infrastructure performance alongside agent accuracy. Here's what the data shows.

TL;DR

• Web Bench is a new benchmark for AI browser agents: 5,750 tasks across 452 websites, with 2,454 tasks open sourced
• Anthropic Claude 4 leads on read-heavy tasks (extracting information from websites)
• Skyvern 2.0 leads on write-heavy tasks (logging in, form filling, file downloads) where all agents struggled most
• Browser infrastructure matters as much as agent quality: proxy failures, CAPTCHAs, and auth blocks account for a meaningful share of failures
• Full results are open source and viewable task by task

The Growing Demand for Agent Automation

Browser automation agents such as Skyvern, Browser-use and OpenAI's Operator have taken the world by storm. These agents have been used in production for a variety of tasks, from helping people apply to jobs, downloading invoices, and even doing SS4 filings for newly formed companies. The Skyvern examples below show how browsing through a website and taking action is tackled by automated agent workflows.

Skyvern attempting to purchase a product

Skyvern attempting to fill out the IRS form

The Challenge of Assessing Web Browser Agent Performance

The examples below show the challenges that agent automation has with web browsing, failing on issues like logging or closing a pop-up dialogue box.

Can’t access chase.com

Can’t close a popup dialog

Reimagining Agent Automation Benchmarking

As a result, we partnered with Halluminate and created a new benchmark to better quantify these failures. Our goal was to create a new consistent measurement system for browser automation agents by expanding the foundations created by WebVoyager by:

1. Expanding the number of websites from 15 → 452, and tasks from 642 -> 5,750 to test agent performance on a wider variety of websites
2. Introduce the concept of READ vs WRITE tasks
   1. READ tasks involve browsing websites and fetching data
   2. WRITE tasks involve entering data, downloading files, logging in, solving 2FA, etc and were not well represented in the WebVoyager dataset
3. Measure the impact of browser infrastructure (eg access the websites, solve captchas, not crash, etc)

The result was Web Bench, a new dataset to assess web browsing agents that consists of 5,750 tasks on 452 different websites, with 2,454 tasks being open sourced.

Assessing Write- and Read-Heavy Tasks

As the Web Bench graph below shows, all agents performed surprisingly poorly on write-heavy tasks (e.g., logging in, filling out forms, downloading files), which implies that this is the area for the highest opportunity for growth

On the other hand, agent performance for read heavy tasks (e.g. extracting information out of websites) was better than we expected (as you can see, Skyvern had the best performance for write-heavy tasks).

For read-heavy, Anthropic’s CUA had the highest performance.

You can even check out the entire performance assessment to see the specific agent performance of each task.

The Web Bench Dataset

The 452 websites in the Web Bench dataset span 17 primary categories. We sampled them from the top 1,000 websites in the world by web traffic, covering a broad range of layouts, authentication patterns, and interactive elements that agents encounter in production.

Cleaning Criteria

We then cleaned the dataset by removing:

• repeat domains
• sites without English translations
• sites blocked by paywall

The graph below details the results of the data normalization.

Assessing Agents With Web Bench

With the Web Bench dataset formalized and a benchmark of agent performance against write- and read-heavy tasks, we took to seeing how agents performed across the range of website categories. Our methodology constisted of:

• We ran OpenAI Operator with a human in the loop to set a baseline for performance
• We used consistent browser infrastructure (Skyvern’s infrastructure) when comparing the API-only models without a runtime to eliminate variables
  • We also ran Skyvern 2.0 on Browserbase to compare the impact of infrastructure, but found (surprisingly) that Skyvern’s infrastructure was able to reliably access more websites and encountered less anti-bot issues during navigation
• Each agent was allowed a max of 50 steps per execution
• Each result was validated by a human in the loop to assert evaluation data quality

We gathered the results across several different metrics:

• Accuracy (overall)
• Accuracy (read-only tasks)
• Accuracy (write-heavy tasks)

Accuracy (Overall)

These results were a little bit surprising, so we decided to cut the data along 2 dimensions to understand where agents may falter:

1. Read only tasks (i.e. extracting visible data from a particular website)
2. Write heavy tasks (i.e. logging into websites, filling out forms, downloading files)

Accuracy (Read-only tasks)

Read only tasks constitute tasks that involve agents going to different websites and moving through the sitemap until a particular answer or state has been found. Unsurprisingly, these results matched the WebVoyager dataset more closely, as the WebVoyager dataset was largely curated to help agents move through websites and answer questions.

The biggest 2 sources of failures for read-heavy tasks are:

• Navigation Issues (cannot figure out how to work through a page, can't solve popups)
• Information extraction issues (doesn't pull the correct information)

Accuracy (Write-heavy tasks)

Write-heavy tasks involve agents entering information as a user would including:

• filling out forms
• logging in
• solving 2FA
• downloading files

In general, the agents had a much lower pass rate across the board. Digging a bit deeper into the failures, there were two culprits for failures that popped up:

• Incomplete execution (hallucinating that it’s achieved the goal when it has not)
• Unable to identify the correct element to interact with (eg can’t close a popup dialog)

In the example below, the agent is unable to close a subscription pop-up.

In the example below, the agent is unable to find and click the coupon buttons.

These issues manifest as agents making adverse changes when filling out forms or optimistically assumes that clicking a "Submit" button completed the task when in reality a captcha appeared that now needs to be solved. This issue is very similar to the phenomenon observed in coding agents where smarter models try to "overhelp" with code changes, by either making changes to unrelated parts of the codebase, or repeatedly suggesting things that are incorrect because they're missing some important context. For a detailed breakdown of these failure modes, see our guide on common mistakes in browser automation.

Breaking Down Agent Failures

So why do agents fail at cetain tasks? Digging into the data, we identified several primary culprits which fell into two primary buckets:

1. Agent Failures (eg Agents hallucinated / made poor decisions / didn’t interact with important elements)
2. Infrastructure failures (eg Agent can’t access the website, solve a captcha to log in)

The 4 biggest categories of agent errors are:

• Navigation Issue (cannot figure out how to work through a page, can't solve popups). Dynamic overlays, cookie banners, and multi-step modal flows that appear mid-session are common culprits. The agent has no explicit instruction for handling these interruptions, so the task stalls at the obstruction instead of working past it.
• Incomplete execution (hallucinating that it's achieved the goal when it has not). Agents see a UI state that looks like success (a loading screen, a redirect, or a thank-you message) and stop short of verifying the actual outcome. In write-heavy tasks, this often surfaces when a form submission triggers a CAPTCHA rather than a confirmation.
• Time outs (exceeds step limits). Each run is capped at 50 steps. Agents that take redundant actions (re-reading the same page, retrying failed clicks repeatedly) burn through the step budget before reaching the goal.
• Information extraction issue (doesn't pull the correct information). The agent reaches the right page but returns the wrong data: a cached value, a nearby field mistaken for the target, or a partial result from a table that required scrolling to complete.

Infrastructure Issues

The problem is that agents can only perform as well as the infrastructure on which they run. Here are the three biggest categories of infrastructure issues which lead to the failure of agents to perform those tasks:

• Proxy (Failed to access website / website blocked). Many high-traffic websites actively block datacenter IP ranges and proxy servers. When the infrastructure can't get past this layer, the agent never sees the page. The failure happens before any reasoning takes place.
• Captcha (Verification required to proceed and infrastructure unable to solve it). Some CAPTCHAs are built to resist automation at the browser level. If the infrastructure can't solve them, the agent is blocked before it takes a single action on the page.
• Login/Authentication (Google Auth detecting you're a bot). Services like Google OAuth detect bot-like session patterns (new device fingerprint, no browsing history, suspicious timing). When this triggers, the login flow either fails outright or drops into a human verification loop the agent can't complete.

These findings imply that the browser infrastructure powering the agents is equally as important as the quality of the agent itself.

Other Interesting Characteristics

While accuracy is the most important characteristic of a browser automation agent, there is also a desire to get "faster" and "cheaper" agents. Fast and cheap agents can be characterized by tracking the following metrics:

1. Runtime duration
2. Number of steps

While pricing models for browser agents continue to evolve, this data gives an important insight into whether pricing per hour (common amongst hosted browsers + older robotic process automation) and pricing per step (common amongst computer use APIs) is the right methodology.

Agent Runtime Duration

This metric is important for a few latency sensitive market segments / situations:

1. Copilot-like products where a human is supervising 1 or many agents executing in parallel
2. Phone agents referencing information / doing real-time lookups while a user is talking
3. Websites aggregating information in real-time to show to the user (e.g. looking up flight or domain name availability)

Number of Steps

Most web browsing agents’ costs scale with the number of steps (i.e. page scans) required to complete a specific task. And agents may use a varied number of steps for a few reasons:

1. Most agents use different architectures to batch actions together to minimize the number of steps
2. Agents eager to solve the problem may use a lot of steps in error situations to try to independently resolve their issues (e.g. chatting with support for solutions instead of terminating early)
3. Agents using pessimistic approaches to reduce hallucinations may invalidate batches of actions whenever the website changes after a particular action (i.e. filling in a zip code field might invalidate the action plan for the rest of the page)

Code Example: Automating a Write-Heavy Task with Skyvern

The write-heavy tasks where agents struggled most in Web Bench (logging in, solving 2FA, and downloading files) are exactly the workflows Skyvern is built for. The example below runs a login-plus-file-download task using the Skyvern Python SDK, with 2FA handled automatically via a stored TOTP identifier.

import asyncio
from skyvern import Skyvern

skyvern = Skyvern(api_key="YOUR_API_KEY")

async def download_invoice():
    task = await skyvern.run_task(
        url="https://portal.example.com",
        # Goal: log in, work through 2FA, and download the latest invoice
        prompt=(
            "Log into the portal using the stored credentials. "
            "If a 2FA code is requested, wait for it to arrive and enter it. "
            "Once logged in, go to the Invoices section, find the most recent invoice, "
            "and download it as a PDF. "
            "COMPLETE when the file has been downloaded successfully."
        ),
        # Identifier Skyvern uses to match incoming TOTP codes to this task
        totp_identifier="billing@yourcompany.com",
        # Extract the invoice metadata for downstream use
        data_extraction_schema={
            "type": "object",
            "properties": {
                "invoice_number": {"type": "string"},
                "invoice_date":   {"type": "string"},
                "amount_due":     {"type": "number"}
            }
        },
        # Block until the task finishes before continuing
        wait_for_completion=True,
        max_steps=30,
    )

    print("Status:", task.status)
    print("Extracted data:", task.output)
    print("Downloaded files:", task.downloaded_files)
    print("Recording:", task.recording_url)

asyncio.run(download_invoice())

The same pattern extends to form submissions, multi-step checkout flows, and any portal workflow that requires authentication — all task types that showed the widest gap between agents in the Web Bench results.

Final Thoughts on Comparing AI Browser Agents

Web Bench tells a cleaner story than most benchmark comparisons do. Read tasks are largely a solved problem: agents can move through websites and pull information with reasonable reliability. Write tasks are where the real gap opens up. Logging in, filling out forms, solving 2FA, and downloading files are the operations that matter most for production agentic process automation. These are the portal-heavy, credential-guarded workflows where APIs don't reach, and every agent tested here has room to grow. The other finding that holds up on closer inspection: browser infrastructure is not a footnote. Proxy failures, CAPTCHA blocks, and login walls can account for a meaningful share of failures before the agent even gets a chance to act. At the end of the day, picking a browser automation agent means weighing two things at once: the quality of the agent's reasoning and the reliability of the infrastructure it runs on. Web Bench gives you a way to look at both, and we plan to keep expanding the dataset and the set of agents tested to make those comparisons sharper over time.

FAQ

What is the difference between read and write tasks in Web Bench?

Read tasks involve going to a website and pulling information out of it: finding a product price, checking a shipping status, reading a policy. Write tasks involve changing the state of a website as a user would: logging in, filling out a form, solving 2FA, or downloading a file. WebVoyager was built almost entirely around read tasks, which is why agents looked better on that benchmark than they do in production. Web Bench separates the two so you can see exactly where agents hold up and where they fall apart.

Why did agents struggle so much more on write-heavy tasks?

Two failure patterns showed up repeatedly. First, incomplete execution. Agents hallucinate that they've finished a task when they haven't, often because they see a confirmation-style UI state and stop short of verifying it. Second, element identification failures. Agents can't find or interact with a specific element like a popup close button or a coupon toggle, which blocks the rest of the workflow. Write tasks also expose infrastructure gaps more directly: a failed CAPTCHA or a login wall stops the agent entirely, whereas a read task might still return a partial result.

How much does browser infrastructure affect agent performance?

More than most benchmarks measure. Proxy failures, unsolved CAPTCHAs, and Google Auth detecting the agent as a bot all account for a meaningful share of total failures, and none of that is the agent's fault. An agent running on weak infrastructure will post worse numbers than the same agent on better infrastructure, even if the underlying model and reasoning are identical. That's why Web Bench tracks infrastructure failures separately from agent errors.

Why not include more websites?

452 websites is the starting point, not the ceiling. The next iterations of Web Bench will expand the site count, add non-English websites, and broaden the category coverage. The goal for version one was to build a consistent methodology across a large enough sample to produce meaningful signal, and to do it with human validation on every result, which takes time and money to do at scale.

Why were only some agents benchmarked?

Human evaluation costs can run approximately $3,000 per agent run through the full benchmark, which makes it impractical to test every agent on the market in one pass. The Halluminate team plans to release an open-source automated evaluation framework so teams can run their own agents against Web Bench without needing human review at that scale. If you want to submit results for your own agent, reach out to the Halluminate team.

Where can I read the full technical writeup?

The Halluminate team published a complete breakdown of the dataset creation process and evaluation methodology. You can read it at the Halluminate team writeup.