HTML Scraping API Documentation
Reference documentation for an HTML scraping API contract: request parameters, rendering and proxy choices, extraction, response formats, status handling and credit accounting. The credit figures describe the reference schedule rather than a verified live subscription offer. Confirm the deployed service and account configuration before making operational requests. This guide explains how to call the HTML scraping endpoint, choose the right rendering and proxy configuration, extract structured data, and interpret costs and status codes. It covers every parameter, the Auto-Mode cost optimizer, proxy tiers, header forwarding, response formats, and the usage endpoint.
Overview
The HTML endpoint accepts a target URL and returns the page content, optionally transformed by JavaScript rendering, AI extraction, CSS/XPath rules, screenshots, or markdown/text conversion. Every request requires an API key and a URL-encoded target URL. The parameter list below groups options by task so you can find the right control quickly.
Getting Started
To scrape a page, send a GET request to the API endpoint with your API key in the Authorization header and the target URL in the url query parameter. The API returns the raw HTML by default. If a URL fails, the system retries for up to 30 seconds; set your client timeout accordingly. Headers and cookies returned by the target site are prefixed with Spb-.
This standard-library Python example makes the same GET request as the documented curl form. Configure the API origin, TARGET_URL and API_TOKEN before use; the token is read from the environment rather than written into the example.
import os
from urllib.parse import urlencode
from urllib.request import Request, urlopen
api_origin = "{{APP_BASE_URL}}"
parameters = urlencode({"url": os.environ["TARGET_URL"]})
request = Request(
api_origin + "/api/v1?" + parameters,
headers={"Authorization": "Bearer " + os.environ["API_TOKEN"]},
method="GET",
)
with urlopen(request, timeout=60) as result:
html_bytes = result.read()
print(html_bytes.decode("utf-8", errors="replace"))API Key
All requests are authenticated with your private API key. Send it as a Bearer token in the Authorization header. Passing the key as an api_key query parameter is deprecated but still works for backward compatibility; it may be removed in a future version.
URL
The url parameter must be the full URL including http or https. You must URL-encode the value; for example, the + character becomes %2B. If you use the Python or Node.js client libraries, they handle encoding for you.
Headless Browser
By default the API fetches pages through a headless browser that executes JavaScript. This is useful for single-page applications built with React, Angular, jQuery, or Vue. To fetch without a headless browser, set render_js=false. The default render_js=true costs 5 credits per request.
JavaScript Execution
Use js_scenario to interact with a page before the HTML is returned. Scenarios are JSON objects with an instructions array; each instruction runs in order. Supported instructions include click, wait, wait_for, wait_for_and_click, scroll_x, scroll_y, fill, evaluate, and infinite_scroll. Stringify the JSON when passing it in a GET request.
Wait for a Fixed Amount of Time
The wait parameter pauses the headless browser for a number of milliseconds between 0 and 35000 before returning the page HTML. Use it when a page needs extra time to render.
Wait for Selector
wait_for accepts a CSS or XPath selector and waits for that element to appear in the DOM. Selectors beginning with / are treated as XPath; all others are CSS. If you use both wait and wait_for, wait_for runs first, then wait, then js_scenario. To control the order yourself, put the waits inside js_scenario.
Wait for Browser
wait_browser tells the browser to wait for a network condition before returning. Accepted values: domcontentloaded (default), load, networkidle0 (no more than 0 connections for at least 500 ms), and networkidle2 (no more than 2 connections for at least 500 ms).
Blocking Ads
Set block_ads=true to skip ads and speed up requests. This parameter is unnecessary when JavaScript rendering is disabled.
Blocking Images and CSS
By default the API blocks images and CSS to speed up requests. Set block_resources=false to load them. This parameter is unnecessary when JavaScript rendering is disabled.
Viewport Width and Height
Use window_width (default 1920) and window_height (default 1080) to change the browser viewport size. These are only useful when render_js=true.
Auto-Mode
With mode=auto, the API tries configurations from cheapest to most expensive and stops at the first success. It only varies render_js, premium_proxy, and stealth_proxy; it never adds page-specific parameters like js_scenario, wait_for, wait, block_resources, country_code, headers, or cookies. Do not pass render_js, premium_proxy, stealth_proxy, or transparent_status_code alongside mode=auto — doing so returns a 400 error. Auto-Mode is only available on GET requests. The credits charged are returned in the Spb-auto-cost response header (0 on failure).
Max Cost
max_cost caps the credit cost Auto-Mode may reach. It must be an integer greater than or equal to 1 and requires mode=auto. For example, max_cost=25 allows the premium-proxy-with-JavaScript tier (25 credits) but not the stealth proxy tier (75 credits). If omitted, Auto-Mode is uncapped.
Proxies
Three proxy tiers are available: premium_proxy, stealth_proxy, and own_proxy. Premium proxies are recommended for hard-to-scrape sites such as search engines, social networks, and e-commerce. Stealth proxies are a stronger pool for the hardest sites. own_proxy lets you use your own proxy provider with the API infrastructure.
Geolocation
Use country_code to choose the proxy country. Geolocation works with both classic and premium proxies, but each has its own supported country list. To use a German premium proxy, set both premium_proxy=true and country_code=de. Stealth proxies also support geolocation. Country codes use ISO 3166-1 format. The table below lists popular examples; the full lists are linked from the source registry.
Stealth Proxy
stealth_proxy=true uses a special proxy pool for the hardest sites. It currently only works when JavaScript rendering is enabled and costs 75 credits per successful call. It supports geolocation. The following are not supported with this option: the infinite_scroll instruction, custom headers and cookies, the timeout parameter, and evaluate_results data when using json_response.
Own Proxy
own_proxy lets you use your own proxy provider with the API. The syntax is <protocol><username>:<password>@<host>:<port>. The protocol is optional; if no port is specified, 1080 is used. Example: https://johndoe:password@my_proxy.com:1234.
Headers
Use forward_headers=true to forward custom headers to the target site. Prefix each header with Spb-; the prefix is trimmed before forwarding. For example, to send Accept-Language: En-US, add Spb-Accept-Language: En-US. The Python and Node.js libraries handle the prefix automatically. To forward only your headers and nothing else, use forward_headers_pure=true; this is only useful when render_js=false.
Response Format
By default the API returns the resource transparently. You can request JSON wrapping, screenshots, markdown, plain text, or structured extraction. Non-HTML resources such as images and PDFs are downloaded transparently; use render_js=false for files and note the 2 MB limit per request.
Data Extraction with AI
Use ai_query to extract information using natural language, and optionally ai_selector to focus the AI on a specific CSS selector. AI extraction parameters (ai_query and ai_extract_rules) add 5 credits on top of the regular API cost. A relevant ai_selector speeds up processing.
CSS / XPath Data Extraction
extract_rules accepts a stringified JSON object mapping keys to CSS or XPath selectors. The simplest format is {"key_name": "css_or_xpath_selector"}. Stringify the JSON when passing it in a GET request.
Screenshot
Set screenshot=true to return a screenshot. Screenshots require render_js=true. When screenshot=true, block_resources is automatically set to false so images and CSS load. To get both a screenshot and HTML, combine screenshot=true with json_response=true. Use wait, wait_for, or wait_browser to control timing.
Screenshot a Particular CSS Selector
By default a screenshot captures only the visible viewport. Use screenshot_selector=<CSS_selector> to capture a specific area of the page.
Screenshot Full Page
Set screenshot_full_page=true to capture the entire page. Use window_width and window_height to change the viewport size before taking the screenshot.
JSON Response
Set json_response=true to wrap the response in JSON. This is useful for third-party tools that only accept JSON, or for intercepting XHR/Ajax requests. The JSON body includes headers, cost, initial-status-code, resolved-url, type, body, and optional fields such as html, screenshot, cookies, evaluate_results, iframes, xhr, js_scenario_report, and metadata.
Return Page Source
Set return_page_source=true to get the original HTML before JavaScript execution. This parameter is unnecessary when JavaScript rendering is disabled.
Return Markdown Content
Set return_page_markdown=true to get the main page content in markdown format, stripped of HTML tags and unnecessary information. This is easier for LLMs to read.
Return Text Content
Set return_page_text=true to get the main page content in plain text, stripped of HTML tags and unnecessary information. This is easier for LLMs to read.
Return Full HTML
Set add_html=true to get the original page HTML alongside a transformed response, returned in the top-level html field. Requires json_response=true. It only has an effect when combined with AI Query, AI data extraction, CSS/XPath data extraction, Return Markdown Content, or Return Text Content; otherwise it is ignored.
Scraping Configuration
Use scraping_config=[Configuration Name] to apply a pre-saved request configuration. This lets you reuse settings across requests without retyping them.
Proxy Mode
The API also offers a proxy front-end for easier integration with third-party tools. Proxy mode only changes how you access the API; request cost, return codes, and default parameters are the same as a standard request. Recommended: disable JavaScript rendering in proxy mode. Configure the proxy domain for your API service before connecting. The endpoint templates are HTTP proxy.<API_DOMAIN>:8886, HTTPS proxy.<API_DOMAIN>:8887, and Socks5 socks.<API_DOMAIN>:8888. Replace <API_DOMAIN> with the configured API domain, preserving each prefix and port. Username: YOUR-API-KEY. Password: PARAMETERS (replace with supported API parameters; start with render_js=False).
Session
Requests sharing a session_id are routed through the same IP address for 5 minutes. Use a random integer between 0 and 10,000,000 each time you need a new session.
Timeout (in ms)
timeout sets the maximum wait in milliseconds, between 1000 and 140000. The default is 140000. Changing it may affect your success rate. There is a 0.5-second margin of error between the timeout you set and the actual maximum duration.
Tag
tag attaches a custom string identifier to a request. The value is returned in the response headers and has no effect on scraping behavior.
Device
device controls the device type that sends the request. Accepted values: desktop (default) and mobile.
The Google Search API scrapes search results pages in real time. You need an API key and a search query. See the dedicated Google API page for details.
POST / PUT
To send a POST or PUT request, send it to the main endpoint with the url parameter and the Authorization: Bearer YOUR-API-KEY header. Data is forwarded transparently to the target page, and headers and cookies are returned.
Credit Cost for Your Requests
Requests cost between 1 and 75 credits depending on the parameters used. The table below shows the base costs. AI features (ai_query, ai_extract_rules) add 5 credits. With Auto-Mode (mode=auto), you are charged only for the configuration that succeeds — one of 1, 5, 10, 25, or 75 credits — and 0 credits if every configuration fails. Use max_cost to cap which configurations Auto-Mode may try.
Usage Endpoint
The /usage endpoint lets you monitor credit consumption and concurrency programmatically. Calls to this endpoint do not increase concurrency, but you can call it at most 6 times per minute. Results are available in real time.
Status Code
The API returns its own HTTP status codes, not the target page's. By default, any 2xx becomes 200; 404, 410, and 413 are passed through; everything else becomes 500. Redirects are followed first. A 500 from the API is not billed. To get the target's status code, use transparent_status_code=true. With that parameter, anything below 500 is billed and there are no retries.
Response Headers
The API returns additional HTTP headers: Spb-cost (request cost in credits), Spb-auto-cost (credits charged by Auto-Mode, only when mode=auto), Spb-initial-status-code (initial status from the scraped page), Spb-resolved-url (resolved URL after redirects), and Spb-request-id (unique request identifier, returned on every response including errors). HTTP/2 lowercases header names, so match case-insensitively. Use the tag parameter to attach your own identifier.
| country_code | Country Name |
|---|---|
| br | Brazil |
| in | India |
| mx | Mexico |
| ... | ... |
| ru | Russia |
| us | UnitedStates |
| gb | UnitedKingdom |
This is a partial list of popular country codes. The full lists for premium and classic proxies are linked from the source registry.
| Code | Billed? | Meaning | Solution |
|---|---|---|---|
| 200 | Yes | Successful Call | |
| 400 | No | Bad request | Incorrect parameters or parameters type. See the message in the response body. |
| 401 | No | Out of credits, or an invalid API key | Check the API key and account entitlement, then resolve any credit or concurrency limit before retrying. |
| 403 | No | The account is banned, or the request was redirected to a blocked destination | The response body says which. This is never the scraped page's own 403. |
| 404 | Yes | Requested URL not found | Provide a valid URL. |
| 410 | Yes | Requested URL gone | Provide a valid URL. |
| 413 | Yes | The target page returned 413 (payload too large) | Check the URL; the page itself refused the request. |
| 429 | No | Too many concurrent requests. | Resolve the account entitlement or capacity limit before retrying. |
| 500 | No | We could not scrape the page, or the target returned a status we do not pass through | Retry; the response body carries the reason. |
These are the API's own status codes, not the target page's. Use transparent_status_code=true to get the target's status.
| Name | Meaning |
|---|---|
| Spb-cost | Request cost in credits. |
| Spb-auto-cost | Credits charged by Auto-Mode. Returned only when mode=auto is used: it equals the cost of the configuration that succeeded, or 0 if every configuration failed. |
| Spb-initial-status-code | The initial status code returned by the scraped page. Useful when the page redirects. |
| Spb-resolved-url | The resolved URL of the scraped page. Useful when the page redirects. |
| Spb-request-id | A unique identifier for your request, returned on every response, including errors. Include the request identifier when reporting a problem; it correlates a request with the corresponding service logs. |
HTTP/2 lowercases header names; match case-insensitively.
Frequently asked questions
What is the difference between render_js and js_scenario?
render_js controls whether the page is fetched through a headless browser that executes JavaScript. js_scenario lets you interact with the page (click, wait, fill, scroll) before the HTML is returned. js_scenario requires JavaScript rendering to be enabled.
How do I get the target page's HTTP status code instead of the API's?
Set transparent_status_code=true. By default the API replaces most target statuses with its own (2xx becomes 200, 404/410/413 pass through, everything else becomes 500). With transparent_status_code=true, you get the target's status, but anything below 500 is billed and there are no retries.
Can I use Auto-Mode with premium_proxy or stealth_proxy?
No. Do not pass render_js, premium_proxy, stealth_proxy, or transparent_status_code alongside mode=auto; doing so returns a 400 error. Auto-Mode chooses those tiers for you. You can still pass page-specific parameters like js_scenario, wait_for, wait, block_resources, country_code, headers, and cookies.
What happens if I use forward_headers_pure with render_js=true?
forward_headers_pure is only useful when render_js=false. With JavaScript rendering enabled, the browser adds its own headers, so pure forwarding is not applicable.
How do I get both a screenshot and the HTML content?
Use screenshot=true together with json_response=true. The JSON response will include both the screenshot (base64-encoded) and the HTML body.
curl "{{APP_BASE_URL}}/api/v1?url=YOUR-URL" \
-H "Authorization: Bearer YOUR-API-KEY"
curl "{{APP_BASE_URL}}/api/v1?url=YOUR-URL" \
-H "Authorization: Bearer YOUR-API-KEY"
curl "{{APP_BASE_URL}}/api/v1?url=YOUR-URL" \
-H "Authorization: Bearer YOUR-API-KEY"
curl "{{APP_BASE_URL}}/api/v1?url=YOUR-URL&render_js=False" \
-H "Authorization: Bearer YOUR-API-KEY"
{
"instructions": [
{"click": "#buttonId"}
]
}
curl "{{APP_BASE_URL}}/api/v1?url=YOUR-URL&js_scenario=%7B%22instructions%22%3A+%5B%7B%22click%22%3A+%22%23buttonId%22%7D%5D%7D" \
-H "Authorization: Bearer YOUR-API-KEY"
curl "{{APP_BASE_URL}}/api/v1?url=YOUR-URL&wait=10000" \
-H "Authorization: Bearer YOUR-API-KEY"
curl "{{APP_BASE_URL}}/api/v1?url=YOUR-URL&wait_for=.loading-done" \
-H "Authorization: Bearer YOUR-API-KEY"
curl "{{APP_BASE_URL}}/api/v1?url=YOUR-URL&wait_browser=load" \
-H "Authorization: Bearer YOUR-API-KEY"
curl "{{APP_BASE_URL}}/api/v1?url=YOUR-URL&block_ads=True" \
-H "Authorization: Bearer YOUR-API-KEY"
curl "{{APP_BASE_URL}}/api/v1?url=YOUR-URL&block_resources=True" \
-H "Authorization: Bearer YOUR-API-KEY"
curl "{{APP_BASE_URL}}/api/v1?url=YOUR-URL&window_width=1500" \
-H "Authorization: Bearer YOUR-API-KEY"
curl "{{APP_BASE_URL}}/api/v1?url=YOUR-URL&window_height=500" \
-H "Authorization: Bearer YOUR-API-KEY"
curl "{{APP_BASE_URL}}/api/v1?url=YOUR-URL&mode=auto" \
-H "Authorization: Bearer YOUR-API-KEY"
curl "{{APP_BASE_URL}}/api/v1?url=YOUR-URL&max_cost=25&mode=auto" \
-H "Authorization: Bearer YOUR-API-KEY"
curl "{{APP_BASE_URL}}/api/v1?url=YOUR-URL&premium_proxy=True" \
-H "Authorization: Bearer YOUR-API-KEY"
curl "{{APP_BASE_URL}}/api/v1?url=YOUR-URL&country_code=de&premium_proxy=True" \
-H "Authorization: Bearer YOUR-API-KEY"
curl "{{APP_BASE_URL}}/api/v1?url=YOUR-URL&stealth_proxy=True" \
-H "Authorization: Bearer YOUR-API-KEY"
curl "{{APP_BASE_URL}}/api/v1?url=YOUR-URL&own_proxy=https%3A%2F%2Fjohndoe%3Apassword%40my_proxy.com%3A1234" \
-H "Authorization: Bearer YOUR-API-KEY"
curl "{{APP_BASE_URL}}/api/v1/?url=http%3A%2F%2Fhttpbin.org%2Fheaders%3Fjson&forward_headers=true" \
-H "Authorization: Bearer YOUR-API-KEY" \
-H "Spb-Accept-Language:En-US"curl "{{APP_BASE_URL}}/api/v1/?url=http%3A%2F%2Fhttpbin.org%2Fheaders%3Fjson&forward_headers_pure=true" \
-H "Authorization: Bearer YOUR-API-KEY" \
-H "Spb-Accept-Language:En-US"curl "{{APP_BASE_URL}}/api/v1?url=YOUR-URL&ai_query=price+of+the+product&ai_selector=%23product-details" \
-H "Authorization: Bearer YOUR-API-KEY"
{"key_name" : "what you want to extract"}
{
"title" : "title of the blog post",
"summary" : "a short summary of the blog post",
}
{
"title" : "How to web ",
"summary" : "We help you get better at web-scraping: detailed tutorial, case studies and writing by industry experts",
}
{
"name": {
"description": "Example post details.",
"type": "string"
},
"categories": {
"description": "Example post details.",
"type": "list"
},
"price": {
"description": "Example post details.",
"type": "number"
},
"in_stock": {
"description": "Example post details.",
"type": "boolean"
},
"shipping_info": {
"description": "Example post details.",
"type": "item",
"output": {
"delivery_time": "estimated delivery in days",
"shipping_cost": "shipping cost in dollars"
}
},
"size": {
"description": "Example post details.",
"type": "list",
"enum": ["XS", "S", "M", "L", "XL"]
},
}
curl "{{APP_BASE_URL}}/api/v1?url=YOUR-URL&extract_rules=%7B%22title%22%3A+%22title+of+the+blog+post%22%2C+%22summary%22%3A+%22a+5+sentences+summary+of+the+blog+post%22%7D" \
-H "Authorization: Bearer YOUR-API-KEY"
{"key_name" : "css_or_xpath_selector"}
{
"title" : "h1",
"subtitle" : "#subtitle",
}
{
"title" : "The ScrapingBee Blog",
"subtitle" : "We help you get better at web-scraping: detailed tutorial, case studies and writing by industry experts",
}
curl "{{APP_BASE_URL}}/api/v1?url=YOUR-URL&extract_rules=%7B%22title%22%3A+%22h1%22%2C+%22subtitle%22%3A+%22%23subtitle%22%7D" \
-H "Authorization: Bearer YOUR-API-KEY"
curl "{{APP_BASE_URL}}/api/v1/?url=YOUR-URL&screenshot=True" \
-H 'Authorization: Bearer YOUR-API-KEY' > ./screenshot.pngcurl "{{APP_BASE_URL}}/api/v1?url=YOUR-URL&json_response=True" \
-H "Authorization: Bearer YOUR-API-KEY"
curl "{{APP_BASE_URL}}/api/v1?url=YOUR-URL&return_page_source=True" \
-H "Authorization: Bearer YOUR-API-KEY"
curl "{{APP_BASE_URL}}/api/v1?url=YOUR-URL&return_page_markdown=True" \
-H "Authorization: Bearer YOUR-API-KEY"
curl "{{APP_BASE_URL}}/api/v1?url=YOUR-URL&return_page_markdown=True" \
-H "Authorization: Bearer YOUR-API-KEY"
curl "{{APP_BASE_URL}}/api/v1?url=YOUR-URL&add_html=True&json_response=True&return_page_markdown=True" \
-H "Authorization: Bearer YOUR-API-KEY"
curl "{{APP_BASE_URL}}/api/v1?url=YOUR-URL&scraping_config=Configuration-Name" \
-H "Authorization: Bearer YOUR-API-KEY"
{
"url_http": "http://YOUR-API-KEY:proxy.scrapingbee.com:8886",
"url_https": "https://YOUR-API-KEY:proxy.scrapingbee.com:8887",
"url_socks5": "socks5://YOUR-API-KEY:proxy.scrapingbee.com:8888",
}
curl "{{APP_BASE_URL}}/api/v1?url=YOUR-URL&session_id=123" \
-H "Authorization: Bearer YOUR-API-KEY"
curl "{{APP_BASE_URL}}/api/v1?url=YOUR-URL&timeout=10000" \
-H "Authorization: Bearer YOUR-API-KEY"
curl "{{APP_BASE_URL}}/api/v1?url=YOUR-URL&tag=your+text" \
-H "Authorization: Bearer YOUR-API-KEY"
curl "{{APP_BASE_URL}}/api/v1/?url=http%3A%2F%2Fhttpbin.org%2Fcookies%3Fjson&cookies=cookie_name_1%3Dcookie_value1%3Bcookie_name_2%3Dcookie_value_2" \
-H 'Authorization: Bearer YOUR-API-KEY'{
"cookies": {
"cookie_name_1": "cookie_value1",
"cookie_name_2": "cookie_value_2"
}
}
curl "{{APP_BASE_URL}}/api/v1?url=YOUR-URL&device=desktop" \
-H "Authorization: Bearer YOUR-API-KEY"
curl "{{APP_BASE_URL}}/api/v1?url=YOUR-URL&device=mobile" \
-H "Authorization: Bearer YOUR-API-KEY"
curl "{{APP_BASE_URL}}/api/v1/google?search=YOUR-SEARCH" \
-H 'Authorization: Bearer YOUR-API-KEY'curl -X "POST" "{{APP_BASE_URL}}/api/v1?url=https:%2F%2Fhttpbin.org%2Fanything" \
-H 'Authorization: Bearer YOUR-API-KEY' \
-H 'Content-Type: application/x-www-form-urlencoded; charset=utf-8' \
--data-urlencode "KEY_1=VALUE_1"curl "{{APP_BASE_URL}}/api/v1/usage" \
-H 'Authorization: Bearer YOUR-API-KEY'{
"max_api_credit": 20000000,
"used_api_credit": 3704332,
"max_concurrency": 200,
"current_concurrency": 1,
"renewal_subscription_date": "2022-04-18T10:05:58.134716"
}
curl "{{APP_BASE_URL}}/api/v1?url=YOUR-URL&transparent_status_code=True" \
-H "Authorization: Bearer YOUR-API-KEY"
| Feature used | API credit cost |
|---|---|
| Classic Proxy without JavaScript rendering | 1 |
| Classic Proxy with JavaScript rendering (default) | 5 |
| Premium Proxy without JavaScript rendering | 10 |
| Premium Proxy with JavaScript rendering | 25 |
| Stealth Proxy without JavaScript rendering | (coming soon) |
| Stealth Proxy with JavaScript rendering | 75 |
AI features add 5 credits. Auto-Mode charges only for the successful configuration and 0 if all fail.