Skip to content

Python Data Extraction with CSS Selectors

When you need a few named fields from a page rather than the whole HTML document, you can ask the request to return those fields directly.

OUTPUT FORMATSHTMLJSONMarkdown

This guide shows how to define extraction rules in Python, verify the selectors before you run the request, and interpret the result.

What extraction rules do

An extraction rule pairs a label you choose with a CSS selector that identifies the element to read. The label becomes the key in the returned data, and the selector tells the service which part of the page to read. This avoids downloading the full response and then parsing it with a separate HTML library when all you need is a small set of fields.

Build the rule map

Pass a dictionary under the extract_rules parameter. Each key is the output label; each value is a CSS selector. For the data extraction documentation page, the title uses the h1 selector and the subtitle uses span.text-[20px].

Verify selectors before sending the request

Open the target page in a browser, open the developer tools console, and run document.querySelector("CSS_SELECTOR") with your selector in place of CSS_SELECTOR. If the call returns an element, the selector matches at least one node. If it returns null, the selector does not match and the extraction rule will not produce a value for that label.

Worked example: title and subtitle

The example below requests the data extraction documentation page and asks for two fields. The output labels are title and subtitle. The selectors are h1 and span.text-[20px]. The expected result is a JSON-shaped payload with those two keys.

Reading the result

A successful response returns a body containing the extracted fields. In the supplied example the body is b'{"title": "Documentation - Data Extraction", "subtitle": "Extract data with CSS selector"}'. The b prefix indicates a bytes object in Python; decode it before treating it as text or JSON.

Distinguish the common failure cases

A selector that matches nothing produces a missing or empty value for that label, not a request failure. A selector that matches several elements may return the first match or a collection depending on the extraction behavior; check the returned shape. A request-level failure is separate: the response is not ok, and the body may contain an error rather than extracted fields. Verify the selector in the browser console first, then check the response status, then inspect the returned body.

When selector-based extraction is the right fit

Use extraction rules when the fields you need are stable, identifiable by CSS selectors, and small in number. If the page requires interaction, pagination, or complex transformation, a full HTML response plus your own parsing step may be more appropriate. The extraction feature is intended to return exact data without a separate post-processing library for the simple case.

Frequently asked questions

How do I check that a CSS selector is correct before using it?

Run document.querySelector("CSS_SELECTOR") in the target page's developer tools console, replacing CSS_SELECTOR with your selector. A returned element means the selector matches; null means it does not.

Why does the extracted result start with a b prefix?

The response content is a bytes object in Python. Decode it to a string before parsing or displaying it as text.

What happens if a selector matches more than one element?

The returned value depends on the extraction behavior for that selector. Check the shape of the returned field and narrow the selector if you need a single specific element.