Self-healing Selenium scrapers that check values, not just errors
A selector can keep matching after a redesign while quietly returning rounded or wrong data. This guide builds a repair loop around that failure: fetch the real page, parse strictly, compare each record with the last good one, and save a replacement rule only after a second page from the same template confirms it.
Why regeneration alone is not self-healing
Regenerating a test script is not the same as repairing extraction. In the captured comparison, a plain fetcher received real content on only 3 of 14 production pages, while a managed fetch returned the requested page on all 14. A challenge page can also return a successful status, so a model may happily repair the wrong page. Treat page identity and field values as separate checks.
A passing test can still mean nothing
One generated set from a challenge page contained 18 always-true length checks, 4 checks with no condition, 7 checks true on almost any loaded page, and 1 check for a heading that the challenge also had. Counting the word Passed would mark all 30 as passing. That single set proves the failure mode is possible, not how often it occurs. A useful check must be able to fail when the extracted value is wrong.
The silent break: matching, but wrong
Selectors written in 2019 for Selenium's repository page still matched in 2020, but the counts had become abbreviated forms such as 16.5k. A strict integer parser rejects that form; a loose cleanup that strips non-digits would turn it into 165 and store a plausible-looking wrong number. Later, the same selectors returned nothing. Exact counts survived in attributes and embedded JSON even when visible text was rounded. Decision rule: reject abbreviated numbers, search attributes and embedded JSON, and send rounded-only values to review rather than guessing.
Step 1 and 2: rules as data, strict parsing, value validation
Store each rule as data with a selector, optional attribute or JSON path, and a type. Use int for counts and decimal for prices so cents are preserved. Keep the last valid record beside the rules. Validation flags missing required fields and, when the baseline is fresh, numeric jumps over 50 percent. Mark genuinely optional fields so an empty sale price or missing description does not trigger a repair on every run. Set the baseline age limit longer than your run interval.
Step 3: confirm the real page before repairing
Fetch with headless Chrome first, then check for anchor text that only the requested page would contain. If the anchor is absent, escalate through the configured fetching service and check the anchor again; if it is still absent, stop instead of repairing. Choose anchor text a login, error or challenge page from the same site would not show. In containers, the documented Chrome flags for sandbox and shared memory may be needed, but only in that environment.
Step 4: model returns values, code builds rules
Ask the model for values only, never for executable code. Deterministic code then searches embedded JSON first, then element text and value attributes, and builds the closest rule that reads the value exactly. It skips generated class names and classes that are not plain CSS names. For rounded model values, allow a tolerance derived from trailing zeros; for decimals, require an exact cent match. This separation keeps generated code out of the pipeline and makes each rule diffable and auditable.
Step 5: promote only after a holdout confirms
Request the same fields on a holdout page from the same template whose values differ. Save a rule only if it reads the holdout value correctly; otherwise write the field to a review file. In 15 runs from the 2019 rules, no run saved a wrong rule. A second run with valid rules made no model calls. Checklist before promotion: correct page confirmed, strict parse passed, holdout value matched, old rule archived in history, and the new record revalidated.
Adapting an existing Selenium scraper
Move existing selectors into the rule file, keeping login and navigation code and feeding the browser page source to the extractor. XPath rules can coexist with CSS rules produced by repair. Add field descriptions that name the exact location, such as a sidebar rather than a README, and put fallback settings on both the fetch and the model call. When values can change between requests, request page HTML together with extracted values so rules and values come from the same copy.
Model-written rules versus value-first rules, and coding agents
Model-written rules can work when the format allows JSON paths, but in the captured tests CSS-only model rules read zero for all counts, and one model run read null everywhere. The value-first workflow checked every rule on a second page. For breaks needing judgment, a coding agent can edit rules from a review file and must pass saved fixtures without network calls; in captured runs it took longer than routine repair. Use the agent for new templates or moved fields, and the holdout workflow for routine breaks.
When to skip selectors entirely
Compare cost per usable page, not per request, because a blocked response still needs another attempt. If extraction credits for every page cost less than maintaining rules, extract every page and keep batch checks. If you need auditable rules or upkeep is cheaper, keep selectors with holdout repair. List pages usually suit per-page extraction because a rule needs known values for at least two items. Some marketplaces offer structured APIs that need neither selectors nor per-page extraction.
Production safeguards
Start by routing repairs to review or a pull request, saving the HTML each repair used and testing new rules in CI. Automate saving only after tests would catch a bad rule. Watch null rates and row counts per batch, repair once per template rather than once per URL, and spot-check a sample against independent model extraction. Send rounded-only values and every disagreement to review.
| Capture | Description | Stars | Forks | Watchers |
|---|---|---|---|---|
| January 2019 | Correct | 13,005 | 4,530 | 1,266 |
| January 2020 | Correct | 16.5k | 5.4k | 1.3k |
| February 2021 | Null | 19.8k | 6.2k | 1.3k |
| January 2022 to January 2026 | Null | Null | Null | Null |
Captured Wayback Machine results for the 2019 selectors; exact counts later moved to attributes and embedded JSON.
| Approach | Runs | Wrong values written | Second-page check |
|---|---|---|---|
| Model rules, CSS only | 2 | Counts read 0 in both runs | No |
| Model rules with JSON paths | 3 | None | No |
| Value-first workflow | 15 | None | Yes, every rule |
Captured test results on one repository page; they demonstrate failure modes, not general model rankings.
| Check | What it catches | Action on failure |
|---|---|---|
| Anchor text present | Challenge, login or error page | Escalate fetch, then stop if still absent |
| Strict parse | Abbreviated or malformed numbers | Treat as missing and validate |
| Fresh-baseline jump check | Rule reading a different number | Trigger repair |
| Holdout match | Rule pointing at the wrong element | Send to review, do not save |
| Batch null and row checks | Template-wide break or lost rows | One repair per template |
Original decision checklist synthesising the workflow's gates.
Frequently asked questions
Can you web scrape with AI?
Yes. A model can read a page and return described fields as JSON in one call, without selector upkeep. If selector upkeep costs more than extraction credits, use that on every page; otherwise reserve model extraction for fields that fail validation.
Can ChatGPT do web scraping?
It can help with small one-off tasks, especially with web search, but repeatable work needs checked values because a confident answer can still be wrong. Use it to help write the scraper, then verify extracted fields.
How does Healenium work?
It stores each element's DOM path from a Selenium test, then on locator failure picks the most similar current element above a configurable score threshold. For scraping, still validate the value, because a similar element may be the wrong one.
How do you scrape without getting blocked?
Make the whole request resemble a browser: TLS fingerprint, HTTP version, headers, IP reputation and request rate all matter, and a Chrome user agent alone does not fix a mismatched handshake. A managed fetching service can handle headless browsers and proxy rotation and try cheaper configurations first.
Is Selenium outdated?
No. Selenium is actively maintained, and Selenium Manager from version 4.6 can download a missing driver. It suits scripted browser runs; at scale, pair it with managed fetching for blocks and rendering, plus value validation.
{
"version": 1,
"rules": {
"description": {"css": "span[itemprop='about']",
"attr": null, "type": "str"},
"stars": {"css": "a.social-count.js-social-count",
"attr": null, "type": "int"},
"forks": {"css": "a.social-count[href$='/network/members']",
"attr": null, "type": "int"},
"watchers": {"css": "a.social-count[href$='/watchers']",
"attr": null, "type": "int"}
},
"history": []
}
{
"date": "2019-01-05",
"values": {
"description": "Example post details.",
"stars": 13005,
"forks": 4530,
"watchers": 1266
}
}
"stars": {
"css": "script[data-target=\"react-app.embeddedData\"]",
"attr": null,
"type": "int",
"json_path": [
"payload",
"codeViewRepoRoute",
"overview",
"sidebarAboutData",
"stargazerCount"
]
}
"price": {"xpath": "//p[@class='price']", "attr": null, "type": "decimal"}
{
"titles": {
"description": "Example post details.",
"type": "list"
},
"prices": {
"description": "Example post details.",
"type": "list"
}
}
| Capture | description | stars | forks | watchers |
|---|---|---|---|---|
| January 2019 | correct | 13,005 | 4,530 | 1,266 |
| January 2020 | correct | 16.5k | 5.4k | 1.3k |
| February 2021 | null | 19.8k | 6.2k | 1.3k |
| January 2022 to January 2026 | null | null | null | null |
| Page when tested | null | null | null | null |