Skip to content

Self-healing Selenium scrapers that check values, not just errors

A selector can keep matching after a redesign while quietly returning rounded or wrong data. This guide builds a repair loop around that failure: fetch the real page, parse strictly, compare each record with the last good one, and save a replacement rule only after a second page from the same template confirms it.

Why regeneration alone is not self-healing

Regenerating a test script is not the same as repairing extraction. In the captured comparison, a plain fetcher received real content on only 3 of 14 production pages, while a managed fetch returned the requested page on all 14. A challenge page can also return a successful status, so a model may happily repair the wrong page. Treat page identity and field values as separate checks.

A passing test can still mean nothing

One generated set from a challenge page contained 18 always-true length checks, 4 checks with no condition, 7 checks true on almost any loaded page, and 1 check for a heading that the challenge also had. Counting the word Passed would mark all 30 as passing. That single set proves the failure mode is possible, not how often it occurs. A useful check must be able to fail when the extracted value is wrong.

The silent break: matching, but wrong

Selectors written in 2019 for Selenium's repository page still matched in 2020, but the counts had become abbreviated forms such as 16.5k. A strict integer parser rejects that form; a loose cleanup that strips non-digits would turn it into 165 and store a plausible-looking wrong number. Later, the same selectors returned nothing. Exact counts survived in attributes and embedded JSON even when visible text was rounded. Decision rule: reject abbreviated numbers, search attributes and embedded JSON, and send rounded-only values to review rather than guessing.

Step 1 and 2: rules as data, strict parsing, value validation

Store each rule as data with a selector, optional attribute or JSON path, and a type. Use int for counts and decimal for prices so cents are preserved. Keep the last valid record beside the rules. Validation flags missing required fields and, when the baseline is fresh, numeric jumps over 50 percent. Mark genuinely optional fields so an empty sale price or missing description does not trigger a repair on every run. Set the baseline age limit longer than your run interval.

Step 3: confirm the real page before repairing

Fetch with headless Chrome first, then check for anchor text that only the requested page would contain. If the anchor is absent, escalate through the configured fetching service and check the anchor again; if it is still absent, stop instead of repairing. Choose anchor text a login, error or challenge page from the same site would not show. In containers, the documented Chrome flags for sandbox and shared memory may be needed, but only in that environment.

Step 4: model returns values, code builds rules

Ask the model for values only, never for executable code. Deterministic code then searches embedded JSON first, then element text and value attributes, and builds the closest rule that reads the value exactly. It skips generated class names and classes that are not plain CSS names. For rounded model values, allow a tolerance derived from trailing zeros; for decimals, require an exact cent match. This separation keeps generated code out of the pipeline and makes each rule diffable and auditable.

Step 5: promote only after a holdout confirms

Request the same fields on a holdout page from the same template whose values differ. Save a rule only if it reads the holdout value correctly; otherwise write the field to a review file. In 15 runs from the 2019 rules, no run saved a wrong rule. A second run with valid rules made no model calls. Checklist before promotion: correct page confirmed, strict parse passed, holdout value matched, old rule archived in history, and the new record revalidated.

Adapting an existing Selenium scraper

Move existing selectors into the rule file, keeping login and navigation code and feeding the browser page source to the extractor. XPath rules can coexist with CSS rules produced by repair. Add field descriptions that name the exact location, such as a sidebar rather than a README, and put fallback settings on both the fetch and the model call. When values can change between requests, request page HTML together with extracted values so rules and values come from the same copy.

Model-written rules versus value-first rules, and coding agents

Model-written rules can work when the format allows JSON paths, but in the captured tests CSS-only model rules read zero for all counts, and one model run read null everywhere. The value-first workflow checked every rule on a second page. For breaks needing judgment, a coding agent can edit rules from a review file and must pass saved fixtures without network calls; in captured runs it took longer than routine repair. Use the agent for new templates or moved fields, and the holdout workflow for routine breaks.

When to skip selectors entirely

Compare cost per usable page, not per request, because a blocked response still needs another attempt. If extraction credits for every page cost less than maintaining rules, extract every page and keep batch checks. If you need auditable rules or upkeep is cheaper, keep selectors with holdout repair. List pages usually suit per-page extraction because a rule needs known values for at least two items. Some marketplaces offer structured APIs that need neither selectors nor per-page extraction.

Production safeguards

Start by routing repairs to review or a pull request, saving the HTML each repair used and testing new rules in CI. Automate saving only after tests would catch a bad rule. Watch null rates and row counts per batch, repair once per template rather than once per URL, and spot-check a sample against independent model extraction. Send rounded-only values and every disagreement to review.

How the 2019 selectors degraded
CaptureDescriptionStarsForksWatchers
January 2019Correct13,0054,5301,266
January 2020Correct16.5k5.4k1.3k
February 2021Null19.8k6.2k1.3k
January 2022 to January 2026NullNullNullNull

Captured Wayback Machine results for the 2019 selectors; exact counts later moved to attributes and embedded JSON.

Rule-writing approaches compared
ApproachRunsWrong values writtenSecond-page check
Model rules, CSS only2Counts read 0 in both runsNo
Model rules with JSON paths3NoneNo
Value-first workflow15NoneYes, every rule

Captured test results on one repository page; they demonstrate failure modes, not general model rankings.

Repair decision checklist
CheckWhat it catchesAction on failure
Anchor text presentChallenge, login or error pageEscalate fetch, then stop if still absent
Strict parseAbbreviated or malformed numbersTreat as missing and validate
Fresh-baseline jump checkRule reading a different numberTrigger repair
Holdout matchRule pointing at the wrong elementSend to review, do not save
Batch null and row checksTemplate-wide break or lost rowsOne repair per template

Original decision checklist synthesising the workflow's gates.

Frequently asked questions

Can you web scrape with AI?

Yes. A model can read a page and return described fields as JSON in one call, without selector upkeep. If selector upkeep costs more than extraction credits, use that on every page; otherwise reserve model extraction for fields that fail validation.

Can ChatGPT do web scraping?

It can help with small one-off tasks, especially with web search, but repeatable work needs checked values because a confident answer can still be wrong. Use it to help write the scraper, then verify extracted fields.

How does Healenium work?

It stores each element's DOM path from a Selenium test, then on locator failure picks the most similar current element above a configurable score threshold. For scraping, still validate the value, because a similar element may be the wrong one.

How do you scrape without getting blocked?

Make the whole request resemble a browser: TLS fingerprint, HTTP version, headers, IP reputation and request rate all matter, and a Chrome user agent alone does not fix a mismatched handshake. A managed fetching service can handle headless browsers and proxy rotation and try cheaper configurations first.

Is Selenium outdated?

No. Selenium is actively maintained, and Selenium Manager from version 4.6 can download a missing driver. It suits scripted browser runs; at scale, pair it with managed fetching for blocks and rendering, plus value validation.

{
  "version": 1,
  "rules": {
    "description": {"css": "span[itemprop='about']",
                    "attr": null, "type": "str"},
    "stars": {"css": "a.social-count.js-social-count",
              "attr": null, "type": "int"},
    "forks": {"css": "a.social-count[href$='/network/members']",
              "attr": null, "type": "int"},
    "watchers": {"css": "a.social-count[href$='/watchers']",
                 "attr": null, "type": "int"}
  },
  "history": []
}
{
  "date": "2019-01-05",
  "values": {
    "description": "Example post details.",
    "stars": 13005,
    "forks": 4530,
    "watchers": 1266
  }
}
"stars": {
  "css": "script[data-target=\"react-app.embeddedData\"]",
  "attr": null,
  "type": "int",
  "json_path": [
    "payload",
    "codeViewRepoRoute",
    "overview",
    "sidebarAboutData",
    "stargazerCount"
  ]
}
"price": {"xpath": "//p[@class='price']", "attr": null, "type": "decimal"}
{
  "titles": {
    "description": "Example post details.",
    "type": "list"
  },
  "prices": {
    "description": "Example post details.",
    "type": "list"
  }
}
Capturedescriptionstarsforkswatchers
January 2019correct13,0054,5301,266
January 2020correct16.5k5.4k1.3k
February 2021null19.8k6.2k1.3k
January 2022 to January 2026nullnullnullnull
Page when testednullnullnullnull