Skip to content

Rust Web Scraping: Build a Scraper Step by Step

A Rust scraper is a small pipeline: one stage pulls bytes off the network, another turns those bytes into a queryable tree, and a third maps nodes onto your own types. Keeping those stages apart is what makes the code easy to test when a target site changes. This walkthrough builds that pipeline against a static demo page, then extends it to typed records and to pages whose content only appears after client-side scripts run. The tooling is reqwest for HTTP, scraper for CSS-selector parsing, and tokio for async execution.

Quick answer: the three-step Rust scraping loop

Most beginner Rust scrapers follow the same sequence: fetch the HTML, parse it into a searchable document, and extract fields into Rust types. For static or lightly dynamic pages, reqwest + tokio + scraper is enough. When a site renders its content with JavaScript, rotates markup, or blocks plain HTTP clients, many developers move the fetching layer to a scraping API and keep Rust for parsing and data processing. The protected dependency block (code0001) and the minimal example (code0002) accompany this page exactly as supplied.

What you need before you start

You do not need advanced Rust experience, but a few basics make the process smoother: functions, structs, and enums; Cargo and dependency management; a rough understanding of HTTP requests, responses, and status codes; and enough HTML knowledge to read tags, attributes, and nesting. The core tools are reqwest for HTTP, scraper for HTML parsing with CSS selectors, and tokio for async execution. Headless browsers are optional and only needed when the page requires JavaScript.

Set up a Rust scraping project

Install Rust with rustup if it is not already available, then confirm that rustc and cargo respond. Create a new Cargo project, add the dependencies shown in the protected dependency block (code0006), and replace src/main.rs with the first-request example (code0007). Run it with cargo run. If HTML appears in the terminal, the request path works and the remaining work is parsing and looping.

Handle pages that load content with JavaScript

reqwest makes HTTP requests but does not execute JavaScript. If a page builds its product list, prices, or listings after load, a plain request returns only the shell. Three realistic options exist: find the underlying JSON endpoint, use a headless browser such as Playwright or Selenium, or route the request through a scraping service that renders JavaScript and handles proxies and blocking. For beginners, checking for an underlying API is usually the fastest path.

Scrape a static site step by step

The books.toscrape.com demo is a stable target for learning the flow. Fetch the page with a reusable reqwest client and a basic User-Agent, fail fast on non-2xx responses with error_for_status, and keep fetching separate from parsing. Then parse the HTML with scraper, count book cards using article.product_pod, and extract titles from the title attribute rather than truncated visible text. The protected code blocks (code0009 through code0012) show each stage exactly.

Turn scraped strings into structured records

Real scraping produces messy strings, so plan for cleanup: trim whitespace, resolve relative URLs against a base URL, and convert price or rating text into usable types. On the books demo, the useful fields are title, price, link, image, and rating. The protected structured example (code0013 through code0023) defines a Book struct, extracts each field defensively, resolves relative URLs with Url::join, and serializes the result to JSON. The rating helper turns a class such as star-rating Three into Three.

Use a scraping API when the page fights back

Car marketplaces are a common example of pages that load listings with JavaScript, use hashed CSS classes, change markup without warning, and block plain HTTP clients. In that situation, the fetching layer can be moved to a scraping API that returns rendered HTML, while Rust continues to parse and process the result. The protected car-listing example (code0025 and code0026) shows defensive selectors, partial class matching, text heuristics for price and location, and a marker element to confirm rendering. The API request itself is retained privately as a source-exact module and is not published as a live endpoint. A scraping API typically bundles rendering, proxy rotation, retries, and bot handling behind one request; whether a specific provider offers those capabilities, and on what terms, is a decision for the owner to confirm before any such claim appears here.

Python example

dotenvy = "0.15"

Decision checklist: plain Rust or a scraping API?

Use plain Rust with reqwest and scraper when the content is present in the initial HTML, the markup is predictable, the job is small to medium, and you are learning the fundamentals. Consider a scraping API or headless browser when the page requires JavaScript to show data, CSS classes are unstable or hashed, anti-bot protection is aggressive, or the scrape is large enough that IP rotation and retries become a burden. Keep the parsing and data logic in Rust either way.

Tell the difference between HTTP failure, missing elements, and unrendered JavaScript

A non-2xx response means the request itself failed; error_for_status turns that into a Rust error instead of letting you parse an error page. A 200 response with zero matching cards usually means the selector is wrong or the page structure changed. A 200 response with a shell but no listings often means JavaScript has not run. Checking the raw HTML length, saving a debug copy, and looking for a known marker element are practical ways to separate these cases before changing selectors.

Field map for the books demo

The demo page uses a consistent card structure. Title comes from h3 a[title]; price from .price_color; link from h3 a[href]; image from .image_container img[src]; and rating from p.star-rating, where the class name carries the rating word. Links and image paths are relative, so resolve them with Url::join rather than string concatenation. If a required element is missing, skip that card instead of panicking.

Where to go next

Get one small static page working end to end before adding anything else. Once the fetch, parse, and extract stages behave predictably, layer on pagination, concurrent requests, and durable storage such as CSV or a database, and only introduce rendering or proxy handling for targets that genuinely need it. The shape of the pipeline does not change as the job grows; the pieces around it do.

Frequently asked questions

Is Rust good for web scraping?

Yes. Rust is fast, memory-safe, and handles concurrent workloads well, which makes it a strong fit for scrapers that run for a long time or at scale. It also tends to offer better performance and safety than many older scraping stacks.

Can Rust scrape JavaScript pages?

Not directly with a plain HTTP client. reqwest does not execute JavaScript, so JS-rendered content will not appear in the response. The usual options are to target the underlying JSON API, use a headless browser, or fetch rendered HTML through a scraping API.

How do I parse HTML in Rust?

Use the scraper crate. Load the HTML with Html::parse_document, query elements with CSS selectors, and extract text or attributes into Rust data structures.

What is the difference between scraping and crawling?

Scraping extracts specific data from pages, such as titles or prices. Crawling discovers and visits pages by following links. Many projects use both: crawling finds the pages, and scraping pulls structured data from them.

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh