Playwright OCR with Tesseract.js: Screenshot Scraping in Node.js
Sometimes the text you need is visible on screen but awkward to reach through the DOM: a canvas widget, a layout split across dozens of wrappers, or a screenshot you were handed rather than a live page. This guide builds a small Node.js pipeline that takes a Playwright screenshot of a product grid, runs Tesseract.js over the image, and uses word coordinates to put each product's name, manufacturer, price, rating, availability, and SKU back into its own card — then parses those lines into JSON. It is a working pattern for one predictable catalogue layout, not a universal scraper.
What the pipeline does
Three stages, each with a clear job:
- Playwright opens the page, waits for the product grid, and captures that element as a PNG at 2× device scale.
- Tesseract.js recognizes the image and returns words with bounding boxes and confidence scores.
- Your JavaScript assigns each word to a product card by checking whether the word's center falls inside the card's box, groups words into lines, and parses the lines into objects.
One honest caveat up front: the card boundaries still come from the DOM. Playwright locates .product elements to learn where each card sits. Only the product information itself is read from pixels. A canvas-only page with no such elements would need a different way to define card regions.
Project setup
Use Node.js 22 or newer with npm. Create a directory, initialize the project, switch package.json to ES modules, and install the two libraries plus a Chromium build for Playwright. You do not need to install the native Tesseract engine separately — Tesseract.js handles that. The exact commands are in the setup command module that accompanies this page.
Your project ends up with node_modules, package.json, package-lock.json, and an index.js where the scraping code lives. Run it with node index.js as you build it up.
Capturing the product grid with Playwright
The demo target is a Northwind Supply catalogue: nine products in a grid, each with a name, manufacturer, price, rating, availability, and SKU. The capture script uses a 1440×1000 viewport with deviceScaleFactor: 2, waits for domcontentloaded, then waits until the first .product inside .grid is visible before shooting.
Three choices matter for OCR quality:
- Screenshot the grid element, not the whole page. Playwright measures the grid itself, so no manual cropping or hardcoded height is needed.
- Use scale: 'device' together with the 2× factor. In the original testing, Tesseract.js sometimes read 4.6 as 4.5 or dropped decimal points at 1×; the 2× capture fixed most of that.
- Disable animations and hide the sticky header during capture. Navigation links overlapping cards were being read as product text.
The finally block closes Chromium even if something throws. In the reference run, the script found nine products and produced a 1752 × 2340 PNG.
Reading the screenshot with Tesseract.js
Create a worker with the English language model, then call recognize() on the PNG. Pass { blocks: true } as the third argument — recent Tesseract.js versions return plain text by default, and you need the detailed block output to get word positions.
The detail tree is blocks → paragraphs → lines → words → symbols. You only need words. Flatten them into one array, and for each word keep:
- text, trimmed
- x0, y0 (top-left) and x1, y1 (bottom-right), each divided by the 2× scale so coordinates are back in CSS pixels
- confidence, rounded
Dividing by the scale is what makes OCR coordinates comparable with Playwright's element positions later. Confidence is a useful flag for suspicious results, but a high score does not prove a word was read correctly. Save the text and words to ocr.json, then terminate the worker to release resources.
What Tesseract.js actually returns
Plain text output can look wrong in a revealing way. In the reference run, the first three product names came back as one line: "Cast Iron Mug Set, Oat Wool Skillet, Large Walnut Side Table, Large". That is not a bug in the usual sense — Tesseract reads horizontally across the image and has no idea three cards sit side by side.
The coordinates tell the real story. Words from different cards share a similar y0 but sit hundreds of pixels apart on x0. So the plain-text mash-up is recoverable: use the boxes to reassign words to their cards.
Other observed quirks: rating stars sometimes became random characters, and crossed-out prices were not always read correctly. Those are acceptable losses if you only need the current price and the numeric rating.
Getting card coordinates and assigning words
Playwright's boundingBox() returns an element's position and size in CSS pixels. Two coordinate systems need reconciling: element boxes are relative to the viewport, while OCR coordinates are relative to the screenshot. Subtract the grid's position from each card's position, and you get card boxes measured in screenshot space.
For each card, store card number plus x0, y0, x1, y1. No column count, card width, or row height needs hardcoding — Playwright reads the real dimensions.
Assignment rule: a word belongs to a card when its center point falls inside the card's box. A second filter drops any OCR result taller than 40 CSS pixels, because Tesseract occasionally tried to read product illustrations as giant characters.
Words then need to become lines. A small helper sorts words by vertical center and groups any whose centers are within 12 CSS pixels into the same row, then sorts each row left to right and joins the text. That 12-pixel tolerance is a heuristic tuned for this catalogue, not a general layout algorithm.
Turning card lines into product JSON
Every card in this catalogue follows the same order: name, manufacturer, pricing-and-rating line, availability, SKU. The parser exploits that order rather than trying to understand shopping sites in general.
Rules used:
- Name: join all lines except the last four, which also handles wrapped names.
- Manufacturer, pricing line, availability, SKU: the last four lines in order.
- Price: the first dollar amount with two decimals, e.g. /\$(\d+\.\d{2})/. The crossed-out price comes after, so it is ignored.
- Rating: a digit 0–5, an optional decimal point, and a final digit at the end of the line. The decimal is optional because Tesseract sometimes read 4.6 as 46; since this catalogue always uses one decimal place, the point can be restored.
- SKU: matched with /NW-\d{4}/i and uppercased.
- If any required field is missing, return null and skip the card rather than emit a partial product. Compare the extracted count against the card count and log failures.
A worked example from the reference run — the first card's raw lines were:
Cast Iron Mug Set, Oat / Fieldhouse / $239.00 $294.80 xxxxs 41 / Low stock - 2 weeks / NW-1001
Parsed result: name "Cast Iron Mug Set, Oat", manufacturer "Fieldhouse", price 239, rating 4.1, availability "Low stock - 2 weeks", sku "NW-1001". The gibberish stars and the slightly wrong crossed-out price are discarded. In the final reference test, all nine products parsed on the first catalogue page.
Three different failures, three different fixes
It helps to separate failure modes, because the fix differs:
- Recognition error. A character is misread — 4.6 becomes 46, a decimal vanishes, stars become letters. Fix: raise capture resolution, hide overlays, and make the parser tolerant where the format is known.
- Structural mis-assignment. Correctly read words land in the wrong card or line. Fix: check the coordinate math (scale division, grid offset) and the line-grouping tolerance.
- Missing information. The text is not visible at all — truncated titles, collapsed sections, content behind a click. No OCR setting recovers it. Fix: reveal the content first, or accept the limit.
A high confidence score only speaks to the first category. It says nothing about whether a word was placed in the right card or whether the source text was ever on screen.
When to use OCR scraping — and when not to
Start with the cheapest reliable source. If the data is in the HTML, use a locator. If the site exposes an API, use it. Reach for screenshots when:
- content is drawn on a canvas or otherwise not exposed as text;
- the markup is so fragmented or volatile that selectors are a maintenance burden;
- you only have screenshots — archived pages, dashboard images, exports from another system;
- you want a visual fallback: try DOM extraction first, switch to OCR when expected elements are absent.
Stay with traditional scraping when the data is already in the HTML, an API exists, you need large volumes quickly, or accuracy is critical (financial figures, for example). Browser rendering plus OCR adds real overhead, and misread digits are a genuine risk.
Rule of thumb: don't make OCR do a job a two-line CSS selector could finish.
Adding an AI step (and its costs)
Two patterns can extend the pipeline:
- OCR + LLM. Feed Tesseract's text and coordinates (or grouped lines) to a language model with a prompt asking for structured JSON, ignoring crossed-out prices and stars, and using null rather than guessing.
- Screenshot + vision model. Skip Tesseract and send the image to a multimodal model that identifies cards and fields in one step.
Trade-offs are real: token or image-processing fees that scale with page count, added latency per request, hardware needs if you run models locally, and unreliable output that still needs validation — models can confuse prices, attach fields to the wrong product, or invent values. There is also no guarantee a vision model reads tiny text better than Tesseract; testing both on the same screenshots is the sensible comparison.
For a stable nine-product catalogue, the JavaScript parser already handled everything without external AI calls. An AI step earns its place when layouts are messy or change often.
Pre-flight checklist
Before running the pipeline on a new page:
- Confirm the data is not already available in HTML or via an API.
- Identify a stable selector for the container and for each card.
- Capture at 2× device scale and screenshot the container element, not the full page.
- Hide sticky headers, cookie banners, and chat widgets during capture.
- Verify that OCR coordinates are divided by the same scale factor used at capture.
- Check that card boxes are offset by the container's position.
- Tune the line-grouping tolerance against real output, not assumptions.
- Log cards that fail parsing and compare counts.
- Spot-check digits and decimal points, especially prices and ratings.
Frequently asked questions
What is Playwright OCR web scraping?
It is a screenshot-based extraction approach: Playwright captures a webpage or element as an image, and an OCR engine such as Tesseract.js reads the text from that image. JavaScript then processes the recognized text — often using word coordinates — into structured data.
How do you use Tesseract.js with Playwright in Node.js?
Open the page in Playwright and take a screenshot. Pass the image to Tesseract.js, which returns text, word coordinates, and confidence scores. Use those results to identify and organize the fields you need.
When is screenshot scraping useful?
When content is hard to extract from HTML, when information is drawn on a canvas, or when only screenshots are available instead of the original page.
What are the main challenges of OCR-based web scraping?
Misread characters, missing decimal points, overlapping text, and difficulty determining which words belong together. Screenshot resolution, font size, and layout all affect accuracy.
Is Tesseract.js better than traditional web scraping?
Not inherently. Tesseract.js suits visual scraping, but CSS selectors and APIs are typically faster and more reliable when the data is already in HTML or a structured format. OCR works best as an alternative or fallback.
Can AI improve Playwright and Tesseract.js scraping?
Yes, in two ways: an LLM can organize OCR output into structured JSON, and a vision model can analyze screenshots directly. Both add processing time, possible API costs or hardware requirements, and the risk of incorrect or invented data.
[
{ "text": "An example comment.", "x0": 16, "y0": 255, "x1": 48, "y1": 266 },
{ "text": "An example comment.", "x0": 315, "y0": 255, "x1": 350, "y1": 266 },
{ "text": "An example comment.", "x0": 615, "y0": 255, "x1": 662, "y1": 266 }
]
[
{
"name": "Cast Iron Mug Set, Oat",
"manufacturer": "Fieldhouse",
"price": 239,
"rating": 4.1,
"availability": "Low stock - 2 weeks",
"sku": "NW-1001"
},
{
"name": "Wool Skillet, Large",
"manufacturer": "Aoyama Works",
"price": 139,
"rating": 4.6,
"availability": "In stock - 2 weeks",
"sku": "NW-1002"
},
{
"name": "Walnut Side Table, Large",
"manufacturer": "Tessa Ruiz Studio",
"price": 64,
"rating": 4.8,
"availability": "In stock - 1 week",
"sku": "NW-1003"
}
]