Skip to content

Ruby data extraction with CSS selectors and extract_rules

A Ruby script can ask the API to return only the fields you name, instead of downloading a full HTML page and parsing it with an extra library.

OUTPUT FORMATSHTMLJSONMarkdown

You send a label-to-selector map as the extract_rules parameter, and the response arrives as a JSON object keyed by those labels. This guide walks through the request shape, how to confirm each selector before you rely on it, and how to tell a selector problem apart from a transport problem.

What extract_rules changes about the workflow

Without extract_rules, a Ruby scraper receives the page content and must locate elements itself, usually with a parser gem. With extract_rules, the selection happens on the API side: you supply a hash where each key is the output label you want and each value is a CSS selector. The response is a JSON object whose keys match your labels.

That changes the shape of the client code. The Ruby side becomes a thin HTTP wrapper: build the query, send the request, print or parse the body. The selector logic lives in the rules hash, which is easy to review and change independently of the transport code.

Request inputs at a glance

The request carries three inputs that matter for this task:

  • api_key — your API credential.
  • url — the page you want to read.
  • extract_rules — a JSON-serialized hash of label-to-CSS-selector pairs.

The example below targets a documentation page and pulls two fields: a title from the h1 element and a subtitle from a span whose class includes the text-[20px] token. The exact request construction is preserved in the accompanying protected code module; treat its field names and endpoint path as fixed.

Verify selectors before you send the request

A selector that looks right in source view can still miss because the page renders differently. Before wiring a selector into the rules hash, open the target page's developer tools console and run document.querySelector("YOUR_SELECTOR"). If it returns the element you expect, the selector is a reasonable candidate. If it returns null, adjust the selector until it matches.

Do this for every label in the hash. A single wrong selector does not necessarily break the whole request, but it will leave that key empty or missing in the response, which is harder to diagnose later.

Reading the response

When the selectors match, the response body is a JSON object keyed by your labels. In the documentation example, the title label resolves to the page heading text and the subtitle label resolves to the span text, producing a compact object rather than a full HTML document.

Because the body is JSON, you can parse it with Ruby's standard JSON library and access fields by the same labels you defined. That keeps downstream code independent of the page's markup structure.

Telling failures apart

Three failure modes look similar in a terminal but need different fixes:

  1. Selector mismatch — the request succeeds but a key is empty or absent. Re-check the selector in the console against the rendered page.
  2. HTTP or transport error — the request itself fails, for example a connection or credential problem. The rescue block in the example prints the error message; read it before changing selectors.
  3. Serialization mistake — the rules hash was not converted to JSON before being placed in the query. Confirm the hash is serialized to a JSON string first.

Separating these three cases saves time, because each points at a different part of the script.

When to use extract_rules and when not to

Use extract_rules when you know the fields you want and the page exposes them through stable CSS selectors. It removes a parsing dependency and keeps the Ruby client small.

Prefer post-processing in Ruby when you need to transform, join or validate values across many pages, or when the markup is too irregular for a single selector per field. In those cases, fetching the page and parsing it locally gives you more control.

Frequently asked questions

Do I need an HTML parsing library in Ruby to use extract_rules?

No. The selection happens on the API side, so the Ruby client only needs to build the request and read the JSON response. A standard HTTP client and the JSON library are enough for the example workflow.

How do I confirm a CSS selector is correct?

Open the target page in a browser, open the developer tools console, and run document.querySelector("YOUR_SELECTOR"). If it returns the element you expect, the selector is a good candidate. Repeat for each label in the rules hash.

Why is a field missing from the JSON response?

The most common cause is a selector that does not match the rendered page. Check the selector in the console first. If the selector matches but the field is still absent, confirm the rules hash was serialized to JSON before being sent.

Can I extract more than two fields?

Yes. Each key in the rules hash is an output label and each value is a CSS selector. Add as many label-selector pairs as you need; the response object will contain one key per label.