Skip to content

News SERP results as structured data

One API call can turn a news aggregator page into usable records: headline, outlet, snippet, date, byline, and article link.

OUTPUT FORMATSHTMLJSONMarkdown

Choose structured JSON for pipelines or rendered HTML when you need the full page.

What the data is good for

Use the returned fields to separate three jobs that are often mixed together: pulling terms out of headlines for curation and trend work, watching how tone changes over time, and tracking shifts in political coverage to adjust messaging. Keeping those purposes distinct helps you decide which fields to store and how often to refresh them.

How it works

Create an account to obtain an API key. Install the SDK for your stack — Python, Node, Go, PHP, or Ruby are named options — so each page becomes a single function call without managing proxy rotation or a headless browser yourself. Send one HTTP request containing your key and the news results URL; proxy rotation, fingerprint handling, and JavaScript rendering happen behind the scenes and a parse-ready response returns. Add advanced parameters only when needed: render_js for JavaScript, stealth_proxy for CAPTCHA-heavy targets, country_code for geo-targeting, and ai_extract_rules for natural-language extraction.

Fields and output choices

Every result carries headline, source publication, snippet, publication date, byline, and article URL. For a quick check, test the request in the dashboard builder and copy the generated snippet into your codebase. For custom fields, pass any SERP URL with news results to the endpoint and choose extract_rules for CSS selectors or ai_extract_rules for AI-driven extraction. If your downstream step is an LLM, clean Markdown output is also available for media-monitoring or sentiment workflows.

Useful extensions

AI extraction rules can parse news data without hand-written HTML parsing. Screenshots show how the page renders in a real browser. JavaScript scenarios can click, scroll, and type to handle dynamic content. A specialized Google Search API path exists for Google News data, and n8n automation supports no-code media-monitoring workflows.

Reliability, scale, and cost controls

Two mechanisms work together to lower the chance of IP-based blocks: rotating proxies and rendering pages in a headless browser. Capacity is designed to grow with demand, so throughput can be raised by adding browsers and proxy routes rather than rebuilding the pipeline. Response times vary by target, and JavaScript-heavy pages generally take longer than static ones; measure against your own list of sources before setting service-level expectations. Data handling deserves a separate review: whether personal data is stored, and which privacy regimes apply, should be confirmed against the operator's current documentation and data processing terms rather than assumed. Budgeting also needs care, because credit consumption depends on the options enabled — base requests, JavaScript rendering, and premium proxies each carry different costs. Check the current credit schedule before sizing a run.

Pricing

Commercial terms — monthly price, included credits, concurrency ceiling, and support level — should be read from the current pricing page rather than from any older summary. The same applies to any introductory credit allowance and whether a payment card is needed to start. Tax treatment is another item to verify: listed prices may exclude VAT or other applicable taxes depending on jurisdiction, so confirm the final invoiced amount with the provider before budgeting.

Parameter decision map
ParameterWhen to use itEffect
render_jsPage content appears only after JavaScript runsRuns the page in a headless browser and returns post-render HTML
stealth_proxyTarget blocks aggressively or presents CAPTCHAsRoutes through stealth proxy handling
country_codeResults vary by countryRoutes through a residential IP in that country
extract_rulesYou know the CSS selectors for each fieldReturns parsed JSON mapped to your field names
ai_extract_rulesYou prefer describing fields in natural languageReturns parsed JSON from an AI-driven extraction spec

Parameter names and effects are as described in the source material; confirm against live documentation before implementation.

Frequently asked questions

Will I get blocked when scraping news search results?

Rotating residential proxies and a real headless browser make requests resemble ordinary browsing. If a target becomes aggressive, enable stealth_proxy on the request.

Can I scrape news search results without writing code?

Yes. The dashboard request builder accepts a URL and parameters, then produces a ready snippet in Python, Node, Go, PHP, Ruby, or cURL.

How do I handle JavaScript when scraping news search results?

Set render_js=true. The page runs in a headless browser on the server and the returned HTML reflects the post-render state.

Can I scrape news search results from a specific country?

Yes. Pass country_code to route the request through a residential IP in the chosen country, which helps when results differ by geography.

What's the rate limit for scraping news search results?

Limits are concurrency limits set by your plan rather than a per-target rate limit. Space requests out, or use multiple sessions, if a target throttles aggressive patterns.

How do I get structured JSON from news search results?

Supply extract_rules mapping CSS selectors to field names, or ai_extract_rules with a natural-language specification. Either approach returns parsed JSON.