Computer Vision Web Scraping with Playwright, YOLO and OCR
A visual scraper reads the rendered page instead of the markup. This guide walks through a working pipeline: Playwright captures a fixed viewport, a UI-trained YOLO model locates interface regions, OCR turns crops into text, and Python pairs fields into validated records before a coordinate click moves to page two. The same fixture is run in two layouts and a spacing variant so the failure modes — a missed detection, an orphan price, a fragile class selector — are visible rather than hidden.
What each stage actually does
The pipeline splits responsibilities cleanly, and mixing them up is the most common source of confusion. Playwright renders the page and can click coordinates; it does not decide which pixels are a product. The UI-trained YOLO model locates regions and assigns interface classes; it does not read a price or understand a product schema. OCR converts a selected crop into characters; it does not know which product owns a price. Validation code pairs fields by row position and rejects bad records; it does not repair a missed detection automatically.
A standard YOLO checkpoint trained on everyday objects should not be expected to recognize buttons or pagination. The example uses the ScreenParser checkpoint, which is trained for interface elements, and its model card warns that results can vary outside its training distribution. Confidence scores and annotated screenshots are saved so predictions can be inspected.
When a visual pipeline is worth the cost
CSS selectors identify nodes in the DOM and are usually the simplest route when the HTML is meaningful, attributes are stable, and controls are accessible. Playwright locators that use roles or visible text can survive class-name changes, so a new CSS class alone is not a reason to add a detector.
Rendered pixels are a reasonable fallback in a handful of narrower cases: content painted into a canvas, content embedded in an image, or repeated card layouts where the DOM structure varies but the visual relationships stay recognizable. A coordinate click may also help when a visible control has no dependable semantic locator. That list is much shorter than "every JavaScript site" — Playwright already renders JavaScript on its own, with no detector involved.
Plan for the entire chain, not only the model: the browser has to render, the detector has to find the right region, OCR has to read it, and your grouping rules have to attach each value to the correct record. A visual scraper might tolerate one kind of HTML change and still fail on a new font, an overlay, or an unexpected mobile breakpoint. Treat the screenshots and intermediate detections as test artifacts you keep, not disposable details.
Set up the reproducible fixture
The test ran on macOS 27.0 (arm64) with Python 3.12.14, Chrome 154.0.8037.57, and Tesseract 5.5.3. The companion requirements.txt pins Playwright 1.55.0, Ultralytics 8.3.203, PyTorch 2.14.0, Pillow 12.3.0, and pytesseract 0.3.13. The script checks the SHA-256 of the ScreenParser checkpoint before starting; the full revision and checksum are in the setup command and companion README.
From the companion directory, after installing Python 3.12, Chrome, and Tesseract, run the setup command. The expected SHA-256 is dbcb4f583ccfdb8100a68e606525c247890a2de4c1a54b14741e0ee29ce0ab88, and the pinned download matched it during verification. Running the compact B case returns exit status 1, since one record goes missing. Add --chrome when Chrome lives at a non-default path. Licensing differs by component: the checkpoint's model card states Apache-2.0, whereas Ultralytics offers its software under AGPL-3.0 or a separate enterprise agreement.
The fixture serves the same records through two layouts. Layout B changes the parent class, the card layout and spacing, and the visual name/price order, while the expected JSON and the DOM order of the child fields stay the same. It does not simulate a responsive breakpoint or every production redesign. Because the data is fixed, you can see which failures come from perception or grouping rather than from the source.
Page one contains Cedar Lamp, Linen Basket, and Stone Mug. Page two contains Oak Desk, Glass Vase, and Wool Throw. The evaluator reads these six records from fixture/expected.json after extraction; the visual scraper does not read that file to find products. The compact variant changes only B's card spacing, so a changed result points to the detector or grouping rather than a different catalog.
Capture a viewport screenshot
The capture step creates a 1280 by 960 viewport and asks Playwright for a viewport-only PNG. The saved image shows three product rows and the Next button; it is the input passed to ScreenParser.
The DOM locator only confirms the fixture has loaded; it does not locate a product or the button. Without that check, a slow page load could look like a detection failure.
Because device_scale_factor=1 and scale="css" are set, one screenshot pixel equals one CSS pixel. The page.mouse.click(x, y) call in Playwright takes coordinates relative to the viewport in CSS units, so a detected box's center can be clicked directly with no rescaling. A full-page capture would additionally contain regions beyond the viewport, and clicking those would first require scrolling and a coordinate conversion.
The script keeps the exact image it hands to YOLO, so a missed detection can be reviewed afterward. On other sites, swap the fixture-specific locator for a condition that confirms the data you need has appeared; Playwright advises against treating networkidle as a general readiness check.
Detect interface regions with a UI-trained model
The detect() function runs the UI model on the screenshot and saves an annotated image. It also records OCR text for boxes predicted as Text and Button.
A 0.10 confidence threshold keeps weak text candidates for this fixture; it is not a general recommendation. Each page-*-detections.json file stores predictions above that threshold with their class, box, confidence, and OCR text. The record builder later drops boxes that do not form valid product rows.
On layout A the detector locates names, prices, and the Next button, and it also marks header and footer text that the record builder discards. ScreenParser did not find a dependable product-card boundary, so the script pairs text boxes by row position. A Text label by itself does not indicate whether a crop holds a name, a price, or page furniture. Compact B later reveals a different failure: the visible name has no box at all.
Read text and validate records
The OCR stage scales each selected crop up to three times its size and reads it with Tesseract's --psm 6 setting. The record builder searches for a price-shaped string, then pairs it with the closest non-price text box whose vertical center is within 26 pixels.
The full function normalizes the price, accepts names matching the fixture's two-to-four-word pattern, and writes page, name, and price. For example, OCR produced Cedar Lamp and $42.00 in layout A; the JSON record is {"page": 1, "name": "Cedar Lamp", "price": "$42.00"}. A real site needs its own price formats, name rules, and representative test pages.
The 26-pixel row tolerance is specific to this fixed viewport. A different font size or zoom level could break pairing even when YOLO detects both fields correctly.
The script saves unpaired text and rejected prices with reasons in run-summary.json. This makes an OCR or grouping error visible instead of silently adding a bad record. In compact B, page two, OCR read $65.00, but the record builder rejected it because YOLO did not detect a name in the same row.
Tesseract is a standalone executable, not merely a Python import. Its installation guide covers the engine and language data, and its quality guide explains how crop size, borders, and page segmentation affect short text.
Click a detected control without a selector
On page one the model tagged the Next control with the Button class, and OCR returned Next page. The script refuses to click unless it finds exactly one button matching that description, then clicks the center of its box.
All three runs satisfied the URL check. Navigation makes the previous boxes stale, so the script grabs a new screenshot afterward. A raw coordinate click skips the actionability checks that Playwright locators perform, and on a live site an overlay might swallow the click. Where a role or text locator reliably pins down the control, prefer it.
Compare visual extraction with selectors
Six expected records span the two pages. One CSS baseline is written for layout A: it starts at .product-card and is deliberately not adapted for B. A second baseline is shared across layouts, targeting article together with its .name and .price children, which both layouts keep. That shared selector is the more even comparison for this fixture.
The shared baseline uses the structure that all fixture versions retain. For this fixture those selectors suffice. The A-specific baseline breaks because it depends on .product-card, which B swaps out. Testing both baselines is what keeps the result honest: reporting only the brittle one would suggest vision is required, when a minor selector tweak already handles the fixture.
The comparison table reports exact record matches against fixture/expected.json. Compact B missed one record; none of the runs produced a false record. The wall clock starts at browser launch and ends after both page captures, model and OCR passes, both selector reads, the coordinate click, and browser close. It excludes model loading and setup. Each case was run once, and the selectors were not timed separately, so these numbers do not compare method speed.
Layout B serves the same products, yet it moves prices to the left and swaps the card container. The shared selector continues to read the name and price fields, which are untouched.
The compact variant changes only B's card height from 120 to 112 pixels and its vertical padding from 24 to 21 pixels. On its second page, ScreenParser detected $65.00 but did not draw a box around the adjacent Wool Throw text. The record builder saved the reason "no name text detected in the same row" for the orphan price. OCR did not get a chance to read the missing name, so changing the price parser would not fix it. Across both compact pages, the visual path read all six prices and five of six names used for records.
In every case, compact B included, the shared selector yields all six records without any model or OCR step. For this fixture, selectors are both simpler and more dependable. Visual detection earns its place when the DOM offers no comparably reliable target. This small controlled test cannot establish a general accuracy rate or a speed advantage on real sites.
Diagnose visual scraper failures
When boxes appear displaced across the board, verify the capture scale, the viewport dimensions, and how far the page has scrolled before touching the model. When the boxes land correctly but the extracted names or prices are wrong, go to the crops themselves and the raw OCR output: cropping too tightly can clip characters, while cropping too loosely can drag in text belonging to the neighboring card. If detections vanish after navigating, determine whether an overlay, a lazy-loading placeholder, or an animation altered what was rendered at capture time. A further pitfall comes from regional number and currency conventions: a parser built around €19.99 may not handle 19,99 €. Read the original text on the page before revising normalization or schema checks. And if a site-specific detector demands repeated retraining whenever the design shifts even slightly, set that ongoing effort against the alternative of a locator or an API you already have access to. Treat claims of self-healing as valid only when a measured result backs them up, not as a slogan borrowed from AI marketing.
Choose the least complicated extraction method
Start with a direct API if the site exposes the needed data and you are authorized to use it. Otherwise, try semantic Playwright locators or stable attributes. Add screenshot detection when the pixels are the most dependable interface you can access and you can test the model, OCR, and validation rules on representative pages. Keep diagnostics and a review path for rejected records. A visual pipeline is justified by its measured success on your pages, not by the word "AI."
A practical decision checklist:
- Does the page expose the data through a documented API you are authorized to use? Prefer it.
- Do stable attributes, roles, or visible text identify the target? Prefer a locator.
- Is the target drawn into a canvas, embedded in an image, or spread across layouts whose DOM differs but whose visual relationships hold? Consider visual detection.
- Can you test the model, OCR, and validation rules on representative pages, and keep diagnostics for rejected records? If not, the visual pipeline is not yet justified.
- Does a site-specific detector need frequent retraining to follow modest design changes? Compare that maintenance cost with a locator or an available API.
| Component | Role in this example | Does not do |
|---|---|---|
| Playwright | Render the page, capture its visible viewport, and click coordinates | Decide which pixels represent a product |
| UI-trained YOLO | Locate visual regions and assign UI classes | Read a price or understand a product schema |
| OCR | Turn a selected image crop into text | Know which product owns a price |
| Validation code | Pair fields by row position and reject bad records | Repair a missed detection automatically |
Roles reflect the fixture used in this example; they are not a general capability claim for any YOLO or OCR tool.
Frequently asked questions
Can a standard YOLO model detect web buttons?
It needs training on the relevant interface classes first. A general object-recognition checkpoint will not work as a ready-made UI detector. This walkthrough relies on the UI-trained ScreenParser checkpoint and still checks each prediction.
Why is OCR needed if YOLO found the box?
A box only marks a region's location and its likely visual class. OCR is what extracts the characters inside that region. Validation code then judges whether those characters make a usable field.
Are image coordinates the same as Playwright click coordinates?
They line up here because the capture covers only the viewport and uses scale="css" at a fixed viewport size. Screenshots taken at device scale or as full-page images need an explicit scale and scroll conversion before any click.
Is visual scraping more reliable than CSS selectors?
It varies by page. A selector may keep working through a visual redesign, while a visual detector may keep working through HTML refactoring. Test each against the changes you anticipate, counting OCR errors and missed boxes, instead of assuming one approach always wins.
Is web scraping legal?
The answer depends on the jurisdiction, data, access method, terms, privacy obligations, and intended use. Public visibility alone is not blanket permission. robots.txt is a crawler instruction standard, not an access authorization system. Review the site's conditions and applicable law before collecting data, especially personal data. See RFC 9309 and GDPR Article 6.
Can a managed screenshot API replace Playwright in this example?
When the task allows it, a managed screenshot API can deliver a rendered image for later detection. The coordinate-click step still requires a live browser session, so this tutorial retains Playwright for that interaction.
python3.12 -m venv .venv
.venv/bin/python -m pip install -r requirements.txt
mkdir -p models
MODEL_REV="f029e565f1206577402e43206454522075be3f72"
MODEL_BASE="https://huggingface.co/docling-project/ScreenParser/resolve"
curl -L --fail --silent --show-error \
"$MODEL_BASE/$MODEL_REV/best.pt" \
-o models/screenparser-best.pt
shasum -a 256 models/screenparser-best.pt
.venv/bin/python scraper.py --layout a --output-dir runs/a
.venv/bin/python scraper.py --layout b --output-dir runs/b
.venv/bin/python scraper.py --layout b-compact --output-dir runs/b-compact
| Layout | Visual records | A-specific CSS | Shared selector | Run wall time |
|---|---|---|---|---|
| A | 6/6 | 6/6 | 6/6 | 4.338 s |
| B | 6/6 | 0/6 | 6/6 | 3.426 s |
| B compact | 5/6 | 0/6 | 6/6 | 3.495 s |
Exact record matches against fixture/expected.json. Wall clock starts at browser launch and ends after both page captures, model and OCR passes, both selector reads, the coordinate click, and browser close; it excludes model loading and setup. Each case was run once and selectors were not timed separately, so these numbers do not compare method speed and cannot establish a general accuracy rate.