Scrape Algolia Search Results via the Public Search API
Algolia powers search on many sites, and its search endpoints are designed to be called from the browser. That makes the Search API a cleaner target than the rendered results page: you can request JSON directly, read structured hits, and control pagination. This guide walks through locating the public application ID, search-only key and index name, calling the Search API, handling the default 1,000-result pagination cap, and recognizing the common failure modes.
What scraping Algolia actually targets
Algolia is a hosted search service. Sites use its InstantSearch widgets or call its APIs to power product search, documentation search, listings and content discovery. When a search page loads, the browser usually sends a request to an Algolia host and receives JSON back.
A rendered-page approach would load the page, simulate typing, wait for results and parse the final DOM. A more direct approach is to call the same Algolia endpoint the page calls. You get the same records as structured JSON, without browser automation for the query step.
This is different from Algolia's Crawler. The Crawler indexes your own site into an Algolia index so visitors can search it. Scraping Algolia means querying a third-party site's search API to retrieve results that site already exposes to browsers.
Legal and ethical checks before you start
The search-only key a page uses is public by design, but public does not mean unrestricted use. Before querying a site's Algolia endpoint, review its Terms of Service and robots.txt, and confirm that your intended use is allowed. Only use keys meant for client-side search. Never use admin or write-capable keys, even if one is accidentally exposed. Keep request rates modest so you do not add unnecessary load.
A practical pre-flight checklist:
- Read the target site's Terms of Service and robots.txt.
- Confirm the key you found is a search-only key, not an admin key.
- Decide how much data you actually need; avoid exhaustive catalog downloads beyond owner-configured limits.
- Set a request rate you can justify and monitor for 429 responses.
- Store credentials only as long as needed, and refresh them when they expire.
Find the application ID, search key and index name
Every Algolia query needs three identifiers: the application ID, a search-only API key and an index name. There are two practical ways to get them.
Manual discovery in the browser:
- Open the target site and its Developer Tools.
- Switch to the Network tab and filter to Fetch/XHR.
- Run a search on the page.
- Look for a request to a host ending in algolia.net, typically a /query or /queries path.
The application ID appears in the x-algolia-application-id header or query parameter. The search key appears in x-algolia-api-key. The index name is the path segment after /indexes/. Application IDs are usually short uppercase strings; search keys are longer alphanumeric values. The same values are often embedded in a script tag in the page source.
Automated discovery: render the page with a browser-capable scraping service and capture the XHR requests it makes. The captured request URL and body contain the same credentials and the JSON response. This is useful when keys rotate or when you need to repeat discovery across many sites.
The request shape for a single-index search is shown in the protected reference module code0001. A real captured example from the HN Algolia site is in code0002, and the search page URL pattern used for discovery is in code0003.
Query the Algolia Search API directly
With the three identifiers in hand, you can send a POST request to the single-index search endpoint. Authentication can go in query parameters or headers; both forms are shown in the protected reference modules code0007 and code0008. The request body carries the search parameters, such as query, hitsPerPage and page.
The response is JSON. The fields that matter most for scraping are:
- hits: the matching records.
- nbHits: total matching records.
- nbPages: total pages at the chosen hitsPerPage.
- page: the current page index.
Each hit contains the same fields the rendered page would display, so you can select the fields you need and export them. The protected reference module code0013 shows a CSV export example using title, url, author, points, num_comments and created_at.
Paginate, then work around the 1,000-result cap
A single request returns only one page. To collect more, loop the page parameter from 0 to nbPages - 1. The protected reference module code0009 shows the loop pattern.
There is a hard limit to plan for. Algolia caps global pagination through paginationLimitedTo, and the default is 1,000. The relationship is:
nbPages × hitsPerPage <= paginationLimitedTo
So with hitsPerPage set to 10, nbPages tops out at 100. The protected reference module code0010 states this constraint. A concrete illustration from the HN Algolia Item_dev index: a query matched 398,258 records, but Algolia exposed only 100 pages at 10 hits per page. Requesting page 100 returns an empty hits array rather than an error.
To go beyond the cap, split the search into filtered ranges that each return fewer than 1,000 records. The filters parameter is shown in code0011, and an example range sequence is in code0012. For each range, check nbHits; if it approaches 1,000, split further. Paginate each subset, then merge. The same technique works with non-numeric attributes such as category or date, depending on the index.
The Browse API is an alternative for full-index retrieval, but it requires browse permission that most public search-only keys lack, so it often returns 403 Forbidden.
Handle multi-index searches
Some sites search several indexes in one request. They call the multiple-search endpoint instead of the single-index one. The endpoint shape is in the protected reference module code0014.
Authentication is the same. The body changes: instead of a flat payload, send a requests array where each item names an indexName and its search parameters. The protected reference module code0015 shows a two-index example.
The response contains a root-level results array. Each element is the individual search response for one request, with the usual hits, nbHits, nbPages and page fields, plus an index field identifying which index produced it. If a site uses /*/queries and you send a single-query payload, requests commonly fail; that mismatch is a frequent cause of errors.
Failure modes and how to tell them apart
Not every empty or failed response means the same thing. Use this distinction table when debugging.
- Empty hits array with a valid page index: you have exceeded the pagination cap. Reduce the page or split the query into filtered ranges.
- 403 Forbidden: the key lacks the required permission, often because you tried the Browse API with a search-only key.
- 401 or authorization error: the key has expired, is restricted to specific indexes or filters, or was rotated. Re-fetch credentials from the live page.
- 429 Too Many Requests: you are hitting rate limits. Lower concurrency and request volume, or route through rotating proxies.
- Blocked page or CAPTCHA during discovery: the site is protecting the page. Use a browser-capable scraping service to render and capture the XHR request.
- Single-query payload sent to /*/queries: the endpoint expects a requests array. Switch payload shape.
Secured keys may include validUntil, allowed-index restrictions, filters or referer limitations. Retrieve the key from the live page each run rather than hardcoding it, and re-run discovery on authorization errors.
Field and control map
A quick reference for the values you will handle:
- x-algolia-application-id: short uppercase application identifier; appears in headers or query parameters.
- x-algolia-api-key: search-only key; appears in headers or query parameters.
- index name: path segment after /indexes/ in the request URL.
- query: the search term sent in the request body.
- hitsPerPage: page size; combined with nbPages it determines the pagination cap.
- page: zero-based page index.
- filters: filter expression used to split large result sets into sub-1,000 ranges.
- paginationLimitedTo: server-side cap, default 1,000.
- requests: array used by the /*/queries endpoint, each item with indexName and params.
- results: root-level array in the multi-index response.
Use the protected reference modules for exact request and response shapes rather than retyping them from this map.
Frequently asked questions
Is it legal to scrape Algolia search?
It depends on the site, the data and your intended use. The search key a page uses is designed to be public, and querying the same endpoint the front end uses typically retrieves publicly available data. Still, check the target site's Terms of Service and any applicable rules before scraping. This is technical guidance, not legal advice.
How do I find a site's Algolia API key?
Open Developer Tools, go to the Network tab, filter to Fetch/XHR, and run a search on the page. A request to an *.algolia.net host will appear. The key is in the x-algolia-api-key header or query parameter. Also check the page source, since the same values are often embedded in a script tag.
Why can I only get 1,000 results from Algolia?
Algolia caps pagination at 1,000 results by default through paginationLimitedTo. Only the index owner can raise that limit. To retrieve more, split the search into filtered ranges that each return fewer than 1,000 records, paginate each subset, and merge the results. Keep the scope reasonable; exhaustively downloading an entire catalog beyond an owner-configured limit may violate the site's Terms of Service.
What is the difference between the /query and /*/queries endpoints?
The /1/indexes/{index}/query endpoint searches one index. The /1/indexes/*/queries endpoint runs multiple searches in one request and expects a body with a requests array, where each item names an indexName and its search parameters. Sending a single-query payload to /*/queries is a common cause of failed requests.
Do I need a browser to scrape Algolia?
No. You may need a browser once to discover the application ID, search key and index name. After that, you can query the Search API directly with a standard HTTP client, without rendering pages or executing JavaScript.
What if the Algolia key is secured or stops working?
Secured keys can expire or be restricted to specific indexes, filters or other conditions, so a hardcoded key may become invalid. Retrieve the key from the live page each time the scraper runs, and re-run discovery when you see authorization errors.
Is scraping Algolia the same as using Algolia's Crawler?
No. Algolia's Crawler indexes your own site into an Algolia index so you can add search to it. Scraping Algolia means querying a third-party site's search API to retrieve publicly available search results.
{
"query": "javascript",
"filters": "points >= 10 AND points < 15"
}
POST https://<APP_ID>-dsn.algolia.net/1/indexes/*/queries