Self-Reviewing Code Generation Loops for Web Extraction
Generated extraction code can run without errors and still return incomplete or subtly wrong data. A self-reviewing loop treats a clean run as only the first check: the program must execute against real HTML, pass independent verification, and only then be trusted for reuse.
The four stages: Generate, Execute, Verify, Refine
The workflow has four narrow stages. Generate produces a runnable extractor from the page and the required fields. Execute runs it against real HTML and records stdout, stderr, and exit code. Verify checks the output against rules you define independently, such as schema, record count, known values, or page state. Refine returns the failed code plus focused feedback for another version.
Asking the same model whether its code looks correct is a weak check, because it may repeat the same mistaken assumption. Verification should come from outside that reasoning: execution results, deterministic checks, or an independent reference not shown in the generation prompt.
Keep a paper trail for every attempt: the generated source, execution output and exit code, verification result, and a diff after any revision. That record shows whether a repair was a small fix or a rewrite of working code, and it gives a human reviewer something concrete when the loop stops.
The pattern has recognizable research relatives. Reflexion, from Shinn et al. at NeurIPS 2023, feeds earlier attempts back as context without retraining the model. Chen et al.'s Self-Debugging uses program behavior and execution feedback when revising code. A 2026 paper, ASG-LLM, applies a similar generate-execute-repair pipeline to scraping modules, working with more security-sensitive and session-bound workflows than the public-page examples here.
Classify the failure before you refine
Before refining anything, classify the failure. Incomplete input, faulty extraction code, and faulty verification need different fixes.
- Incomplete input: JavaScript rendering, lazy loading, pagination, or infinite scroll can return a successful response with only part of the data. In the captured tests, a plain fetch found 0 product cards while a rendered fetch found 16, and an initial infinite-scroll fetch returned 6 of 48 items until scrolling and waiting for a data-complete signal produced 48 of 48. Rewriting the parser cannot recover records that never reached the DOM, so check page completeness before generation.
- Plausible but wrong output: valid JSON can still miss records or carry wrong prices, titles, or ratings. Compare against an independent reference rather than judging plausibility.
- Verifier failure: an overly strict pre-execution allowlist rejected harmless standard-library use in the tests. Treat checks as software that needs testing, grounded in record counts, schema, known values, page state, and exit codes.
A practical checklist: confirm the HTML is complete, run the extractor, verify against an independent reference, inspect the diff, and cap Refine at roughly two or three attempts before failing loudly for human review.
The captured experiments used GPT-5.6 through OpenAI's Responses API to generate Python extractors with BeautifulSoup. The baseline was Books to Scrape, a public practice site with mostly static HTML. The generated extractor passed on its first attempt and also passed on two holdout pages it had not seen during generation: catalogue page 2, with another 20 books, and the Travel category, with 11 books and a different relative URL depth. The harder cases were the JavaScript-rendered products challenge and the infinite-scroll challenge on Scrapify Data Labs.
Run generated code safely and reuse what passes
Generated code must run somewhere safe. Do not give it unrestricted access to the application machine. A production setup should inspect source before execution, restrict allowed imports and operations, remove secrets from the environment, impose time and output limits, and run inside an isolated container or similar restricted environment. Static checks help catch obvious problems but are not a security boundary on their own, and they can produce false positives.
Once an extractor passes verification, reuse it instead of regenerating for every request. Re-invoke the loop only when verification starts failing again, for example after the target structure changes. This reuse is the main difference from a self-healing scraper, which tries to relocate a broken selector, and from single-request AI extraction, which returns data directly without leaving a reusable program to execute and review.
Cost is part of that trade-off. Both ai_query and ai_extract_rules add 5 credits on top of the normal request cost, so a classic request with JavaScript rendering costs 5 credits, or 10 credits once one of those AI extraction features is added. Because credit pricing can change, confirm the current rates before you budget on these numbers.
When the loop is worth owning
Use a code-generation loop when you expect to reuse the result: repeated extraction from the same page family with stable or semi-stable markup at meaningful volume. For a one-off job, a small batch, or pages with highly variable structure, a single AI extraction request is simpler because you describe the data and receive it without maintaining parser code.
Keep fetching separate from generation and verification. A fetch layer, including an MCP server used by an agent, can supply HTML with rendering, waits, browser controls, proxy, and geo-targeting options, but it does not write, execute, verify, or repair the extractor. The current MCP server exposes several tools relevant to extraction: get_page_html fetches the full page HTML; get_page_text returns the page as text or Markdown; extract_page_data extracts structured data using CSS or XPath rules; get_screenshot captures a page or element screenshot; and get_file fetches files such as images or PDFs. For this loop, get_page_html is the most obvious integration point.
That separation makes it easier to tell whether the problem is the page, the parser, or the checks. The loop itself remains your own application logic, including its execution environment, verification rules, logging, and retry limits.
| Situation | Better fit |
|---|---|
| One-off or small extraction job | Single AI extraction request |
| Pages with highly variable structure | Single AI extraction request |
| Repeated extraction from the same page family | Generated, verified extractor |
| Stable or semi-stable markup at meaningful volume | Generated, verified extractor |
Rule of thumb from the captured guidance; cost balance depends on target, rendering, and volume.
Frequently asked questions
What is a self-reviewing code generation loop for web extraction?
An LLM drafts extraction code, that code runs against real HTML, its output is checked mechanically, and a failed attempt can be revised using execution and verification feedback. Looking correct is not enough; the code must prove itself in execution.
Is a self-reviewing code generation loop something a fetch service does automatically?
No. It is an application pattern you build yourself. A fetch service can supply page HTML, including rendered HTML for JavaScript-heavy pages, but it does not generate, execute, verify, or repair extraction code as a built-in self-review loop.
How many Refine iterations should the loop allow before giving up?
Two or three repair attempts is a reasonable starting point. If it still fails, stop, preserve execution and verification evidence, and surface the case for human review rather than continuing indefinitely.
Does the loop have to be written in Python?
No. Python with BeautifulSoup was used for the experiments described here, but the pattern does not depend on that choice. The same four-stage workflow fits any language that can execute the generated program, parse the target content, and validate the result.
Can a self-reviewing code generation loop scrape pages that require a login?
The pattern itself does not set that boundary, but the fetch setup used here does. It is limited to publicly accessible, pre-login content, so the examples do not target pages or data requiring authentication. Adding a code-generation loop does not change that policy.
How is this different from a self-healing scraper?
A self-healing scraper aims to keep an existing extraction strategy working as markup shifts, typically by relocating an element once a selector stops matching. The loop described here operates on the program itself: it authors source code, executes it, validates the output, and can produce a revised version of that code when a check fails.
{
"title": "product title",
"price": "current product price",
"in_stock": "whether the product is available"
}