AI Agent Web Automation: ReAct Loops vs CSS Selectors
Hand-written CSS selectors are cheap to run and expensive to maintain. ReAct agent loops are flexible to build and expensive to run. The useful pattern is not picking one; it is assigning each job to the layer that does it best. This guide walks through what a ReAct loop actually does, why selectors break, what it costs to feed a page to a model, and how to gate a self-healing extractor so a confident wrong value never reaches your database.
What a ReAct agent loop does
A ReAct loop alternates reasoning and action. The model writes a short thought, picks an action from a fixed set, runs it, reads the result, and reasons again until the goal is met. For web automation the action set is small: navigate, find something on the page, extract a value, click an element, or return the result. The observation is whatever the page returns after each action. The name comes from the 2022 paper by Yao and co-authors, which introduced the interleaving of reasoning traces and task-specific actions in language models. That loop is the core of browser agents and computer-use models, and it underpins both AI browser automation and AI agent web automation more broadly. The example below runs against a real documentation site. The actions are real, but the policy that picks the next action is a placeholder; in production you swap it for a per-step model call that returns the next action as JSON. The same loop also has to face less cooperative targets, where a plain request draws a challenge instead of a page. A ReAct loop is not a single model call. It is a control structure: a fixed action space, a policy that chooses the next action from the goal and the latest observation, and a stopping rule. The policy can be a script, a model, or a mix. What makes it ReAct is the interleaving of reasoning and acting, not the model behind it.
A scripted trace, then a live model
Running the scripted loop prints one Thought, Action, and Observation per step. The trace below shows the loop navigating to a topic page, finding a tutorial link, opening it, and pulling its metadata from the page's own JSON-LD.
A scripted policy is deterministic; a live model is not. When the same loop was driven by a live model, its first guess at link text missed, it read the miss, and it corrected itself on the next step. Two more steps returned the same fields as the scripted run. That self-correction is the ReAct behavior a script cannot fake. It also came with friction: the model wrote actions as extract() and navigate(url), so the harness normalizes them, and the endpoint returned nothing on some calls, so the policy retries.
Why hand-written CSS selectors keep breaking
A CSS selector binds to structure, not data. div.main-content h1 assumes the headline sits under a div.main-content. A redesign ends that: the selector returns nothing, and the pipeline writes a null where the data used to be. That is the core failure: the selector encodes a route through the markup, not the identity of the value. When the route changes, the selector does not degrade gracefully; it returns nothing, and the pipeline records a null where the data used to be. That is not hypothetical. A 2018-era selector written against a real article page returned the headline on the archived version and nothing on the live one, because the wrapper changed from a div to a main element. The page's own JSON-LD survived the redesign, because it reads the data the site publishes about itself, not a route through the markup. The selector was tied to a wrapper element; the JSON-LD was tied to the page's self-description, which the redesign left intact. JSON-LD is not a universal answer. Across 40 real pages, of the ones that came back, about 60% had some JSON-LD, but under 20% carried the page's own article or product data. The rest was boilerplate, Organization and WebSite schema, not the content you want. Article and product pages carry main-content entities more often than homepages, so check for one first and keep a fallback. When the JSON is not there, the fallback is to describe the field in natural language and let a model read the page. That works across layouts, but it trades structural brittleness for semantic ambiguity. The model is not following a route through the markup; it is interpreting the page's meaning, which is more flexible and less exact. On a page with a list price, a sale price, and a shipping fee, a model can return the wrong one. A confident wrong value is worse than an empty one, because an empty value is visible and a wrong value is not. The bigger cost is not one broken selector: hand-written selectors break as their targets change, so the maintenance load scales with how many sites you track, not how much data you pull.
What it costs to send a full page to an LLM
The direct version of AI extraction is one line of intent: fetch the HTML, hand it to a model, ask for the fields. It gets expensive in production, because HTML is far larger than the data inside it. The model pays for every tag, attribute, and script, not just the values you want. Measured with the o200k_base tokenizer, a normal article page came to 77,081 tokens of raw HTML. The visible text was 11,530 tokens, about a seventh of that. A slimmed version, with scripts, styles, SVG, comments, and most attributes stripped, still ran to 45,800 tokens, more than half the raw size. Across the same 40 pages, of the ones that came back, the median was about 35 times heavier in tokens than its useful text, near 120,000 tokens of raw HTML. The heaviest overran the context window of many models before a question was even asked. The cost follows the tokens. At an input price of $5 per million tokens, sending raw HTML for that one page costs about $385 per 1,000 pages. Sending the text only costs about $58. Same data, and sending the full page is roughly 7 times more expensive per page, on every run. Preprocessing changes more than price. A 2025 web-extraction benchmark ran the same model over the same pages in different input formats. The format alone moved the F1 score from 0.10 with slimmed HTML to 0.96 with a flattened JSON that keys each value by its path. The hallucination rate fell from 91% of pages to 3%. HTML-based input, even slimmed, hallucinated the most, and full HTML was too large to feed at all. The most accurate format was also the largest in tokens, so you choose the representation on purpose, per target. A screenshot is another input entirely. Computer-use agents skip the HTML and read the rendered page as an image, which can cost fewer tokens than a bloated DOM. It trades that saving for lower precision on structured fields and slower runs. It is a different tool with its own costs, not a drop-in replacement for HTML or text input. Price alone is not the deciding factor. Per-page extraction is still a recurring cost on every run, and it is still non-deterministic. For a stable, high-volume target, code you compile once wins. That is what pushes the winning pattern away from live extraction at scale.
Where AI agents win in 2026
The win from agent loops is not a model reading every page on every run. That version is expensive at scale. The real win is using the reasoning layer where reasoning is scarce, and deterministic code where volume is high. The work splits across two layers, each with its own job: - Build time: let the agent write and heal the scraper. Point a ReAct loop at a new site. It explores the page, generates extraction code plus a test, and the code runs cheaply after that. When the site changes and the test fails, the same loop regenerates the code and opens a change for a human. This is smooth on cooperative sites and rough on hard, defended ones, so human review is not optional. - Run time: let deterministic code do the extraction. Compiled selectors or a small parser run in milliseconds, cost almost nothing per call, and return the same output every time. For a stable, high-volume target, parsing a page you already fetched is cheaper and faster than a model call, and it doesn't hallucinate. The healing half is easier to claim than to run. Detecting drift is not the hard part: a test fails when a selector returns nothing. Trusting the regenerated selector is the hard part, because for live data you have no ground truth to check it against. You cannot pin an expected price the way you can pin a headline, since the price changes. The dangerous case is not a failed test. It is a redesign where the new selector returns a wrong but plausible value, the null check passes, and the pipeline ships confidently wrong data. Catching that needs invariants, a value that looks like a price and agrees with the page's JSON-LD, not an exact-match fixture. So the loop is detect, propose, and gate on a human, and the human is there because the regeneration is not trustworthy alone. That division is not new in machine learning. To build the FineWeb-Edu dataset, its authors labeled a sample of web pages with a slow model, then distilled those labels into a cheap classifier that scored the whole corpus. Web extraction is following the same pattern: the agent is the expensive judgment, and the generated extractor is the cheap code. Live LLM extraction is still worth it, in a narrow band. It fits when layouts change faster than you can maintain selectors, or when you pull many different sites into a single schema at low volume. A one-time research pull across 200 unrelated sites is a good fit. Scraping a single retailer 50,000 times a day is not.
The reliability gap the demos do not show
Benchmarks explain the gap between how agents look and how they run. On single-site benchmarks like WebVoyager, the best 2026 agents are near saturation, in the mid-90s. Real work is longer and spans several sites. Odysseys, a 2026 benchmark of long-horizon web tasks built from real browsing sessions, puts the strongest model at a 44.5% success rate. The next frontier model sits at 33.5%. Both reach even those scores with wasted steps.
The arithmetic is the story. Give each step a 95% chance of success, and ten steps compound to about 60%. Twenty steps fall to about a third. A short task hides the per-step error that a long, multi-site chain exposes.
Two failure modes hide inside those averages. The first is that status codes lie. A response can return HTTP 200 with a body that is actually a challenge page or a soft block. An agent that trusts the 200 will reason over an error page and emit confident nonsense. The second is hallucination on structured values. A model can produce a clean, plausible number that is not on the page, or invent a total that no line item supports.
Loading the page is the harder problem
An agent that can't load the page can't reason about it, and this limit sits below extraction entirely. Modern anti-bot systems fingerprint the TLS handshake and the browser itself. A standard HTTP client has a fingerprint that matches no real browser, so it gets a challenge instead of HTML.
The fingerprint problem has a DIY answer. Libraries like curl_cffi impersonate a browser's TLS handshake, and stealth browsers like nodriver and camoufox drive a real engine. A plain request advertises a python-requests JA3 fingerprint; curl_cffi in Chrome mode presents Chrome's instead. That closes the gap until the vendor changes the test. Self-hosted stealth is endless maintenance against a moving target. For a few undefended targets, that maintenance is light. It grows with every defended site you add and every time they change the test. That, not the absence of a DIY path, is the real argument for a fetch service.
The wall is also getting taller on purpose, aimed at agents specifically. From September 2026, Cloudflare blocks AI crawlers by default on many ad-supported pages, and its pay-per-crawl scheme charges bots for access instead. So the question is shifting from whether you can fetch the page to whether you are allowed to. A growing share of pages are also client-side rendered, so the first fetch returns an empty shell.
A plain request to 40 real pages was defeated by about 1 in 5: some blocked or challenged the request, one timed out, and a few returned an empty JavaScript shell. The full script and its 40-URL list are on GitHub, spanning news, docs, reference, and commerce, so you can reproduce these numbers yourself. The results move run to run as different pages block or shell.
Take one hard target end to end. A plain request to a big retailer's product page returns HTTP 200, but the title and price are stripped out. A residential proxy gets the real page, and one call extracts the fields and validates them. The plain 200 is the earlier failure mode: a real status code with no data behind it. The proxy call costs more, 30 credits and about ten seconds here, and it took a couple of tries to find settings that worked. That is normal for defended targets.
The extraction looks trivial but is not. This page carries dozens of prices, other editions, used copies, and related titles, so a naive selector grabs the first one, a different book. A model has a better chance at the right product's price, and the guard is there because it can still miss.
Whatever fetches the page, verify the body before you trust it, because a 200 is not proof. A real content page is mostly text; a challenge or a JavaScript shell is almost all markup. The text-to-HTML ratio does most of the work. Two of four sample pages returned a 2xx status with no page behind it.
None of that is a model problem. Model capability keeps rising, but it is not the whole story. METR's time-horizon work shows the best models finishing roughly 12-hour tasks at 50% reliability by mid-2026, doubling every few months. Those are software, machine-learning, and security tasks, though. Web work is a different, harder distribution, and no capability curve erases anti-bot walls, page size, or the per-page cost of a model.
A practical architecture that reasons, then runs
Put the pieces together, and the architecture has four layers, not a single clever agent. Each layer does the job it's cheapest at. - Fetch. Get past anti-bot and render JavaScript so a real page reaches you. This is the unavoidable layer, and it is the same whether extraction is deterministic or model-driven. - Reason. Point an agent loop at a new or changed target. It explores, plans, and proposes an extractor. This is where declarative, describe-the-data automation belongs. - Compile. Turn the agent's plan into durable output: selectors or a small parser, plus a schema and a test that pins expected values. - Run and heal. Execute the compiled extractor at scale, cheaply and deterministically. Monitor it. When the test fails, send the target back to the reason layer, regenerate, and route the change through a human. Assume scraping is necessary and allowed. The decision of which method to run then reduces to two questions. How much does the layout vary, and how much volume do you push? Match the method to the pair. The matrix hides a sharper question: where is the line? Live extraction adds about 5 credits per page over a plain fetch, on every run. Regenerating a broken selector costs about 10 credits per model call, and a break can take a call or two. So compiled code saves 5 credits per page and pays 10 to 20 per heal. Those are machine costs; each heal also routes through a human review, the larger and unpriced one. It wins once a template serves more than a few page-fetches between redesigns. A one-time build moves the line to roughly ten fetches. Most targets sit far above that, so compiling is cheaper, and it does not hallucinate. Whatever you choose above the fetch layer, the fetch layer still has to work. That is what a scraping API is for. When you want the declarative path, AI extraction takes a natural-language instruction through the ai_query parameter, or a JSON schema through ai_extract_rules. When an agent is in control, it calls tools over MCP, an open standard for connecting agents to tools; the MCP server exposes fetch and extract as two of them. A single rule holds across every layer that touches a model: validate the output before you trust it. If a value informs a decision, check it against a schema and against the page, then retry on failure rather than accept a confident guess. A guard with a typed model catches a bad value before it reaches your database. A real extraction from a product page passes; a request for a price on a page that has none returns a sentinel string, which is honest but still not a price, and the guard rejects it so that string never lands in your database as a value. A real price validator also handles currency codes, trailing symbols, and ranges, so treat this as a pattern, not a drop-in. Set the temperature to 0 to cut variance. It does not make the model deterministic or correct, which is the reason you validate at all. Keep the schema strict, and treat a validation failure as a signal to re-fetch, not a value to store.
Putting it into practice
The move from CSS selectors to ReAct agent loops is not a one-for-one tool swap. It is a split of labor. Let the reasoning layer do what reasoning is for: explore unknown pages, plan extraction, and repair scrapers when sites change. Let deterministic code do what it is for: run at scale, cheaply, with the same output every time. And keep a real fetch layer under both, because a page you can't load is a page you can't extract.
If you're building this, first check whether an official API or dataset already offers the data. The cheapest scraper is the one you never write. Then determine whether the target even permits scraping, because robots rules, terms, and pay-per-crawl are now part of the build.
Classify each target on two axes, variance and volume, and pick the method that fits. Put a validation guard on anything a model touches. The split of labor holds whether you build the fetch layer or buy it.
| Layout variance | Volume | Recommended method |
|---|---|---|
| Low | High | Compiled selectors or small parser |
| Low | Low | Compiled selectors or small parser |
| High | Low | Live AI extraction |
| High | High | Agent-generated extractor, then compiled code with a human gate |
Method guidance reflects the captured 2026 evidence; re-check current conditions for your own targets.
| Failure mode | Symptom | Guard |
|---|---|---|
| Selector drift | Selector returns nothing | Test fails, route to reason layer |
| Plausible wrong value | Null check passes, value is wrong | Invariant against a second source (e.g. JSON-LD) |
| Soft block | HTTP 200 with challenge or shell body | Text-to-HTML ratio and marker check |
| Hallucinated structured value | Clean number not on the page | Typed schema validation, re-fetch on failure |
Guards are illustrative patterns from the captured evidence, not drop-in validators.
Frequently asked questions
Can AI agents replace web scrapers?
Not for high-volume, stable targets in 2026. A fully autonomous scraping agent that handles anti-bot, huge pages, and constant site changes on its own doesn't exist yet. Agents are strong at building and repairing scrapers. Deterministic code still runs the extraction at scale.
Is LLM extraction cheaper than CSS selectors?
It depends on the model and the scale. A small model on cleaned text can cost cents per 1,000 pages, but you pay it on every page and every run. A compiled selector wins on stable, high-volume targets. LLM extraction is worth it when layouts vary or you cover many different sites.
Can AI scrape data without CSS selectors?
Yes. You describe the fields in natural language or a schema, and a model returns them, so you skip hand-written selectors. Often better: when a page embeds its data as JSON-LD, you parse that directly. Both still need a fetch layer to load the page past anti-bot.
Do AI agents get past anti-bot systems?
Not by themselves. Anti-bot systems fingerprint the connection below the page, and an agent that can't load the page can't act on it. Getting a real page back is a separate fetch problem. A browser and proxy layer handles it before any extraction runs.
When should I use AI extraction over selectors?
Use AI extraction when layouts change often, when you pull many sites into a single schema, or when per-page volume is low. Use selectors for stable, high-volume targets. For high variance at high volume, let an agent generate the extractor and run that code.