Skip to content

Data extraction in Go

This guide shows how to use a scraping API's extract_rules parameter from Go so that selected page elements come back as structured JSON.

OUTPUT FORMATSHTMLJSONMarkdown

You will see the request shape, the CSS selector choices, the expected response, and the error paths that matter when the API returns something other than 200.

Why use extract_rules instead of parsing HTML in Go

The extract_rules parameter lets you name the fields you want and pair each with a CSS selector. The scraping service applies those selectors and returns a JSON object keyed by your labels. That means your Go program does not need to download the full page, parse the HTML, and walk the DOM with a third-party library. You still control which elements are selected, but the extraction step happens remotely and the response arrives already structured.

The Go request shape

The example builds a GET request to the API endpoint, adds three query parameters, and reads the response body. The parameters are api_key, url, and extract_rules. The rules are a Go map that is marshalled to JSON before being added as a query value. The code checks for errors at each stage: rule encoding, request construction, the HTTP call, non-200 status, and body reading. Each failure returns a distinct message so you can tell which step broke.

Worked selector example

The example targets a documentation page and extracts two fields. The title field uses the h1 selector. The subtitle field uses span.text-[20px]. Before putting a selector into your Go code, you can test it in the browser console with document.querySelector("CSS_SELECTOR") on the target page. When both selectors match, the response is a JSON object with title and subtitle keys. In the captured example the values are "Documentation - Data Extraction" and "Extract data with CSS selector".

Request field and control map

Use this checklist to confirm each part of the request before running it:

  • api_key: your API credential, added as a query parameter.
  • url: the full target page address, URL-encoded by the query builder.
  • extract_rules: a JSON object whose keys are your output field names and whose values are CSS selectors.
  • HTTP method: GET in the supplied example.
  • Status check: treat any response other than 200 as an error before reading the body.
  • Body handling: read the full response body and close it with defer resp.Body.Close().
  • Error messages: keep encoding, request-build, HTTP-call, status, and body-read failures separate so debugging is faster.

Distinguishing failure modes

The Go example separates four failure points that are easy to blur together. A json.Marshal error means the rules could not be encoded. An http.NewRequest error means the request itself could not be built. A client.Do error means the HTTP call failed before a response arrived. A non-200 status means the service responded but rejected or could not complete the request. A body-read error means the response arrived but its content could not be consumed. Checking these separately tells you whether to fix your rules, your URL, your network path, or your credential.

Where to go next

The data extraction documentation covers the full parameter set and additional options. For CSS selector syntax, the W3Schools CSS selectors reference is a practical companion. If you want to adapt the example to another page, start by testing selectors in the browser console, then move them into the rules map unchanged.

Frequently asked questions

Do I need an HTML parsing library in Go to use extract_rules?

No. The scraping service applies the CSS selectors and returns a JSON object keyed by your field labels. Your Go code only needs to build the request and read the response body.

What happens if a selector matches nothing?

The supplied example does not describe the response for a missing match. Test selectors in the browser console first, and treat an empty or absent field as a signal to review the selector against the live page structure.

Why check the status code before reading the body?

A non-200 response means the service did not complete the extraction as requested. Returning an error before reading the body keeps the failure message specific and avoids treating an error payload as extracted data.