Skip to content

C# data extraction with CSS selectors and extract_rules

A C# client can avoid parsing a whole HTML document when the API accepts selectors and returns labeled fields.

OUTPUT FORMATSHTMLJSONMarkdown

The request carries three inputs: an API key, a target URL, and an extract_rules value that maps labels to CSS selectors. The response is JSON keyed by those labels, so downstream code reads fields instead of walking a DOM tree.

What the request needs

The supplied C# example builds a query string with HttpUtility.ParseQueryString, then sends it to the API base URL. Three query fields matter here: api_key, url, and extract_rules. The extract_rules value is a JSON object whose keys become the labels in the response and whose values are CSS selectors. In the example, title maps to h1 and subtitle maps to span.text-[20px]. The complete request code is delivered as the protected code module code0001 and should be used exactly as supplied.

Check selectors before sending the request

A selector that works in the browser console is the quickest sanity check. Open the target page's developer tools, switch to the console, and run document.querySelector("CSS_SELECTOR") with the selector you plan to use. If it returns the element you expect, the selector is plausible for extract_rules. This check does not prove the API will see the same DOM, but it catches typos and selectors that match nothing on the page.

Worked example

The supplied example targets the data extraction documentation page. It asks for two fields: title from h1 and subtitle from span.text-[20px]. The returned JSON is {"title": "Documentation - Data Extraction", "subtitle": "Extract data with CSS selector"}. That output shows the label-to-selector mapping working end to end: each key in extract_rules appears as a key in the response, and each value is the text matched by the selector.

Set API_BASE_URL, API_KEY and SITE_BASE_URL. This is an HttpClient alternative to the documented HttpWebRequest implementation, preserving GET /api/v1/ and its api_key, url and extract_rules query fields and title/subtitle output keys. JsonSerializer emits valid JSON. The subtitle class is quoted as an attribute value so brackets in the class name are literal CSS text; check that both selectors match your target HTML.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Text.Json;

string apiOrigin = Environment.GetEnvironmentVariable("API_BASE_URL")
    ?? throw new InvalidOperationException("Set API_BASE_URL");
string apiKey = Environment.GetEnvironmentVariable("API_KEY")
    ?? throw new InvalidOperationException("Set API_KEY");
string siteOrigin = Environment.GetEnvironmentVariable("SITE_BASE_URL")
    ?? throw new InvalidOperationException("Set SITE_BASE_URL");
var rules = new Dictionary<string, string>
{
    ["title"] = "h1",
    ["subtitle"] = "span[class~='text-[20px]']"
};
var parameters = new Dictionary<string, string>
{
    ["api_key"] = apiKey,
    ["url"] = siteOrigin.TrimEnd('/') + "/documentation/data-extraction/",
    ["extract_rules"] = JsonSerializer.Serialize(rules)
};
string query = string.Join("&", parameters.Select(item =>
    Uri.EscapeDataString(item.Key) + "=" + Uri.EscapeDataString(item.Value)));
using var handler = new HttpClientHandler
{
    AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
};
using var transport = new HttpClient(handler);
string result = await transport.GetStringAsync(apiOrigin.TrimEnd('/') + "/api/v1/?" + query);
Console.WriteLine(result);

Decision checklist

Use extract_rules when you already know the labels you want and can express them as CSS selectors. Use raw HTML post-processing when the structure is unknown, when you need attributes rather than text, or when the selector cannot be validated against the page. Before sending a request, confirm that api_key is present, url points to the intended page, and extract_rules is valid JSON with string keys and selector values. After the response, check that every requested label appears; a missing label usually means the selector matched nothing.

Failure distinction

Three failure modes look similar in code but have different causes. A selector mismatch returns JSON without the expected label, or with an empty value. A missing or malformed extract_rules parameter returns an error or an unlabeled response. A transport failure, such as a non-success HTTP status, happens before extraction and should be handled separately from field-level checks. Keeping these checks separate makes the C# client easier to debug.

Field and control map

Request inputs: api_key identifies the account; url is the page to extract from; extract_rules is the JSON label-to-selector map. Response output: a JSON object whose keys match the labels in extract_rules. The C# helper in code0001 wraps HttpWebRequest with automatic GZip and Deflate decompression and reads the response stream to a string. The example uses HttpUtility.ParseQueryString to assemble the query string, so values are URL-encoded before the request is sent.

Frequently asked questions

What does extract_rules accept?

A JSON object mapping labels to CSS selectors. In the supplied example, the value is {'title': 'h1', 'subtitle': 'span.text-[20px]'}, and the response uses the same labels as keys.

How can I check a CSS selector before calling the API?

Run document.querySelector("CSS_SELECTOR") in the target page's developer tools console. If the expected element is returned, the selector is plausible; if not, adjust it before sending the request.

Why is a requested label missing from the response?

The most common cause is a selector that matched nothing on the page. Confirm the selector in the browser console, then confirm that extract_rules is valid JSON and that the label is spelled the same in the request and the response check.

Does the C# example need an external HTML parser?

No. The example sends selectors through extract_rules and reads the returned JSON string directly. That avoids post-processing the full HTML document with a separate parsing library.