Wall Street Journal News Scraper
A hosted scraping API for Wall Street Journal pages: hand over a URL, and the service renders the page, deals with bot defenses, and returns either rendered HTML or structured JSON.
The same request pattern can target article pages, market index pages and economic-calendar pages, so one integration can feed several downstream jobs.
Three jobs, one request pattern
Most WSJ collection work falls into one of three buckets. Knowing which bucket you are in determines the fields you should ask for and the extraction method you pick.
- Market indices. Index name, last price, percentage change and historical chart data — the inputs a market-tracking dashboard needs.
- Article summaries. Headline, dek, byline and section tag per piece — the inputs editorial research and content curation need.
- Economic forecasts. Indicator name, prior value, consensus estimate and actual print — the inputs macro research needs.
A request returns rendered HTML, or structured JSON when you supply extract_rules or ai_extract_rules. Article pages on wsj.com are the documented target; the same request pattern is what you would point at index and economic-calendar pages, so validate each page type on your own before relying on it.
Choosing an extraction method
Two mechanisms produce structured output, and they suit different stages of a project.
- extract_rules — CSS selectors mapped to field names. Best when the page layout is stable and you know exactly which elements you want. Deterministic and cheap to reason about.
- ai_extract_rules — a natural-language specification. Best when you are still exploring which fields matter, or when the markup shifts between page types and writing selectors becomes a maintenance burden.
A practical sequence: start with ai_extract_rules to confirm the fields exist and are shaped as expected, then lock the schema down with extract_rules once the layout is understood. Both return parsed JSON in the response body.
Request options that change the outcome
The base call needs only the target URL and an API key. The options below are what you reach for when the base call is not enough.
- render_js — executes page JavaScript before returning content. Needed when article bodies or index widgets are injected client-side.
- stealth_proxy — the strongest anti-bot tier, for pages that block ordinary requests.
- premium_proxy — a middle tier; its credit cost depends on whether JS rendering is also enabled.
- country_code — routes the request through a residential IP in the chosen country, useful when WSJ serves different content by geography.
- Screenshots and extraction rules — capture a full-page image, or pull structured fields with CSS or XPath.
Each option has its own credit cost, quoted per option rather than as a single combined total. The pricing table below lists the plans and their included features; the per-request credit rules are in the FAQ.
A worked credit example
Three discrete per-request credit figures apply to the options that matter most here:
- JS rendering: 5 credits per successful request.
- premium_proxy with JS rendering: 25 credits per request (the source tooltip gives premium proxies as 10 or 25 credits, depending on whether JS rendering is disabled or not).
- stealth_proxy: 75 credits per request.
Base requests start at 1 credit each. No stacking rule is published that combines these into a single per-request total, so treat any combined figure as an illustrative assumption rather than a stated contract.
Using only the stated per-option figures: 1,000 stealth_proxy requests at 75 credits each come to 75,000 credits, which matches the Hobby plan allowance exactly. The same 1,000 requests at the premium_proxy-with-JS rate of 25 credits each come to 25,000 credits, comfortably inside Hobby. Test the cheaper tier first and escalate only when you see blocks.
When a request fails, which failure is it?
Four different problems look similar from the outside but need different fixes.
- Concurrency limit reached. The API rejects the request because you already have the maximum number of in-flight requests for your plan. Fix: queue work, or move to a plan with higher concurrency.
- Target throttling. WSJ itself starts slowing or refusing requests because of aggressive patterns. Fix: space out requests or spread them across multiple sessions. The API does not impose a per-target rate limit, so this is on your pacing.
- Bot block. The page returns a challenge or a stripped-down version. Fix: enable stealth_proxy, and confirm render_js is on if the content is client-rendered.
- Empty extraction. The request succeeds but your JSON fields are null. Fix: your selectors no longer match the page; re-check them or switch to ai_extract_rules while you rebuild the rule set.
First-call checklist
- Create an account and copy the API key. The free tier includes 1,000 credits and does not require a card.
- Pick one representative WSJ URL for each of your target page types — an article, an index page, an economic-calendar page.
- Send each URL with no options first. Note whether the content you need is present in the returned HTML.
- If content is missing, add render_js. If the response is a block page, add stealth_proxy.
- Add extract_rules (or ai_extract_rules) and confirm the JSON fields match your downstream schema.
- Estimate monthly credits using the per-option costs, then choose a plan with headroom for retries.
- Add pacing and retry logic before scaling volume; concurrency limits are per plan, not per target.
Pricing plans
The table below reproduces the published plan tiers, credit allowances, concurrency and feature inclusions. Prices are exclusive of VAT. Credit allowances are valid for one month and do not roll over. Treat the figures as a dated reference and confirm current pricing before committing.
Languages and SDKs
The API is language-agnostic: any HTTP client works. Official SDKs are published for Python, Node.js, Go, PHP, Ruby and Java, and plain cURL is supported. Documentation includes examples in Python, Node.js, cURL and a request builder, so you can copy a snippet, swap in the key and send the request.
| Plan | Price / month | API credits | Concurrent requests | JS rendering | Rotating & premium proxies | Geotargeting | Screenshots & extraction rules | Priority email support | Dedicated account manager | Team management |
|---|---|---|---|---|---|---|---|---|---|---|
| Hobby | $19 | 75,000 | 25 | Included | Included | Included | Included | Not included | Not included | Not included |
| Freelance | $49 | 250,000 | 50 | Included | Included | Included | Included | Not included | Not included | Not included |
| Startup (recommended) | $99 | 1,000,000 | 100 | Included | Included | Included | Included | Not included | Not included | Not included |
| Business | $249 | 3,000,000 | 200 | Included | Included | Included | Included | Included | Included | Included |
| Business + | $599 | 8,000,000 | 400 | Included | Included | Included | Included | Included | Included | Included |
| Enterprise 14 | $999 | 14,000,000 | 500 | Included | Included | Included | Included | Included | Included | Included |
| Enterprise 24 | $1,599 | 24,000,000 | 600 | Included | Included | Included | Included | Included | Included | Included |
| Enterprise 41 (recommended) | $2,399 | 41,000,000 | 700 | Included | Included | Included | Included | Included | Included | Included |
| Enterprise 69 | $3,599 | 69,000,000 | 800 | Included | Included | Included | Included | Included | Included | Included |
| Enterprise 120 | $5,799 | 120,000,000 | 900 | Included | Included | Included | Included | Included | Included | Included |
Published prices are exclusive of VAT. API credits are valid for one month and do not roll over. Requests start at 1 credit each. Figures are a dated reference; confirm current pricing before purchase.
Frequently asked questions
Do I need an account to scrape Wall Street Journal?
No external WSJ account is needed. You sign up for an API key — 1,000 free credits are included and no card is required — then send the URL and the service handles the rest.
Can I scrape Wall Street Journal from a specific country?
Yes. Set country_code on the request to route through a residential IP in that country. This matters when WSJ serves different content by geography.
What's the rate limit for scraping Wall Street Journal?
There is no per-target rate limit. The API enforces concurrency limits per plan instead. If WSJ itself throttles aggressive patterns, space out requests or spread them across multiple sessions.
How do I get structured JSON from Wall Street Journal?
Use extract_rules with CSS selectors mapped to field names, or ai_extract_rules with a natural-language specification. Either way, the response body comes back as parsed JSON.
Which programming languages can I use to scrape Wall Street Journal?
Any language with an HTTP client. Official SDKs exist for Python, Node.js, Go, PHP, Ruby and Java, and plain cURL works too — pick whatever your stack already uses.
How much does it cost to scrape Wall Street Journal?
Signup includes 1,000 free credits with no card required. Paid plans start at $19/month for higher volume and concurrency. Per-request credit cost depends on the options you enable: 5 credits for JS rendering, 25 for premium_proxy with JS rendering, and 75 for stealth_proxy.
Does the API return HTML or JSON?
Both are available. Without extraction rules you get rendered HTML. Add extract_rules or ai_extract_rules and the response carries parsed JSON instead.
Can I capture a screenshot of a WSJ page?
Yes. The screenshot option captures a full-page image, and it is included on every plan tier shown in the pricing table.