Skip to content

Crypto News Scraper: One Endpoint, Structured Article Data

Crypto news moves fast, and the pages that carry it are rarely built for machine reading.

OUTPUT FORMATSHTMLJSONMarkdown

A crypto news scraper sits between the publication and your pipeline: you hand it an article URL, it renders the page in a real browser, rotates the IP, and returns the fields you asked for. This guide walks through what such an endpoint actually returns, how to shape a request, what each option costs in credits, and where the legal and ethical lines sit. It is written for developers and data teams who need reliable, repeatable extraction from public crypto news pages.

What a crypto news scraper returns

A typical crypto news article page exposes a predictable set of fields: the headline, the byline, the publish timestamp, the source publication, a dek (the short summary under the headline), the body copy, and the entity tags the publication attaches (tokens, projects, people). Those fields are the raw material for headline collection, market coverage monitoring, and sentiment trend analysis. The endpoint does not invent data — it returns what renders on the public page, structured according to the extraction rules you supply. If a field is missing from the page, it will be missing from the response.

How a request flows from URL to JSON

The workflow has four steps. First, create an account and get an API key with the free starting credits. Second, install the SDK for your stack (Python, Node, Go, PHP, Ruby, or Java) — or skip the SDK and use plain cURL. Third, send the target article URL with your API key as a parameter; the renderer executes the page's JavaScript in a real browser and rotates the IP. Fourth, add advanced parameters as needed: render_js to force JavaScript rendering, stealth_proxy for CAPTCHA-heavy pages, country_code for geo-targeting, and ai_extract_rules for natural-language extraction. The response is the rendered page or the structured JSON your rules define.

Field and control map

Use this map to decide which extraction method fits each article field. CSS selectors (extract_rules) are deterministic and cheap; AI extraction (ai_extract_rules) is flexible but should be reserved for fields whose markup varies across publications.

What each option costs in credits

Credit consumption is driven by the options you switch on, not by the article you target. A plain request starts at one credit. Enabling JavaScript rendering raises a successful request to 5 credits. Premium proxies cost 10 credits when JS rendering is off and 25 when it is on. Stealth proxy sits at 75 credits per request. Credit validity periods and free-tier allowances vary by provider, so treat any published figure as something to verify against the current pricing page before you commit to a volume plan. The same applies to paid plan entry prices and concurrency limits: check them at signup rather than assuming a fixed number.

Worked example: costing a daily headline pull

Suppose you pull 200 crypto news article pages per day and need JavaScript rendering on each. At 5 credits per successful JS-rendered request, that is 1,000 credits per day, or roughly 30,000 credits per month. The Hobby plan at $19/month includes 75,000 API credits, so it covers this workload with headroom for retries. If you switch to premium proxies with JS rendering, the per-request cost rises to 25 credits, and the same 200-page daily pull becomes 5,000 credits per day — about 150,000 per month, which exceeds the Hobby allowance. The lesson: choose the cheapest option that reliably clears the target site's defenses, and measure before scaling.

Distinguishing the three common failure modes

When a request returns nothing useful, the cause is usually one of three things. A blocked response (CAPTCHA page, 403, or empty body) means the site's anti-bot stack rejected the request — try stealth_proxy or premium proxies. A JavaScript-rendered page that returns only the shell means render_js was not enabled or the page needed more wait time — enable render_js and, if needed, a JavaScript scenario that clicks or scrolls before extraction. A selector miss means the page loaded but your extract_rules did not match — inspect the rendered HTML and adjust the CSS selectors, or fall back to ai_extract_rules. Treating these as distinct saves debugging time and credits.

Choosing between CSS selectors and AI extraction

CSS selectors are fast, deterministic, and cheap — use them when the target publication's markup is stable and you know the exact elements. AI extraction (ai_extract_rules) accepts natural-language instructions and returns structured results, which is useful when you are pulling from many publications with different layouts or when the field you want is buried in inconsistent markup. A practical pattern: start with CSS selectors for the headline, byline, and timestamp, and use AI extraction for the dek, body, and entity tags where markup varies most. Measure accuracy on a sample before committing to a volume plan.

Article field to extraction method
Article fieldTypical sourceRecommended method
Headlineh1 or article title elementextract_rules (CSS selector)
Bylineauthor meta or byline elementextract_rules (CSS selector)
Publish timestamptime element or JSON-LDextract_rules (CSS selector)
Source publicationsite name or canonical URLextract_rules (CSS selector)
Dek summarysubheadline or meta descriptionai_extract_rules or extract_rules
Body copyarticle body containerai_extract_rules or extract_rules
Entity tagstag list or topic linksai_extract_rules

Recommended methods reflect the captured extraction options; actual selector reliability depends on the target publication's markup.

Frequently asked questions

Do I need an account to scrape crypto news?

You do need an API key, and that comes from creating an account with the provider. Free-tier allowances differ between services, so check the current signup page for the exact starting credits and whether a card is required. Once you have the key, you send the article URL and the API takes care of rendering, proxy rotation, and returning the response.

Is scraping crypto news legal?

Public-facing pages can often be scraped, but the target site's terms of service define what is actually permitted. Review those terms before deploying, and leave post-login content out of scope. Depending on your jurisdiction and the data involved, GDPR or CCPA obligations may also apply, so confirm your own compliance position rather than assuming a blanket exemption.

What data can I extract from crypto news?

Anything that renders on the public page. Use extract_rules with CSS selectors for deterministic fields like headline, byline, and timestamp, or ai_extract_rules for natural-language extraction when markup varies. The response is structured JSON.

How do I start scraping crypto news?

Create an account, copy your API key, and send a GET request with the target article URL. The renderer executes the page, rotates proxies, and returns the data. Free-tier credit amounts vary by provider, so check the current signup page for the exact allowance available for testing.

Which programming languages can I use to scrape crypto news?

Any language that can make an HTTP request. Official SDKs exist for Python, Node.js, Go, PHP, Ruby, and Java, and plain cURL works too. Pick whatever your stack already uses.

How much does it cost to scrape crypto news?

Free-tier allowances and paid plan entry prices vary by provider, so confirm both on the current pricing page before budgeting. Per-request credit cost depends on the options you enable: 1 credit baseline, 5 for JS rendering, 10 or 25 for premium proxies (without or with JS), and 75 for stealth proxy. Credit validity periods also differ between plans, so check how long purchased credits remain usable.