RSS Feed Scraper for News Pipelines
RSS is a simple, well-known format, but feeds still break, hide behind anti-bot layers, or need to be merged with rendered pages.
This page covers how a hosted scraping API turns a feed URL into structured article records, which request options change the result and the credit cost, and how to choose between a plain parser and a rendering service.
What the scraper returns
You send a public RSS feed URL with your API key. The service fetches the feed, renders it in a real browser when needed, rotates the IP, and returns either structured JSON or rendered HTML. The article fields available on a feed are title, summary, author, publication date, content link and category tags. If a feed omits one of these, that field is simply absent from the output for that item — the scraper does not invent values.
Three workflows teams build
The common patterns are: (1) aggregating headlines from several feeds into one stream, (2) monitoring when a competitor or industry site publishes new posts, and (3) feeding fresh article text into an LLM pipeline for summarization or topic tagging. Each of these is a read-only pull of public feed data, so the main design question is how often to poll and how to deduplicate by content link or publication date.
How a request is built
The workflow is four steps. First, create an account to get an API key and free test credits. Second, install a client SDK for Python, Node, Go, PHP or Ruby, or call the API directly. Third, send the feed URL with your key as a parameter. Fourth, add advanced parameters only when the feed needs them: render_js for JavaScript, stealth_proxy for aggressive anti-bot targets, country_code for geo-targeting, and ai_extract_rules for natural-language extraction. Each scrape becomes a single function call; proxy rotation and headless browser management are handled for you.
Field and control map
Use this to decide which request options affect which output fields. The left column is what you want back; the right column is the option that most directly affects it.
Worked example: one feed in, JSON out
Suppose you monitor a public news feed that publishes roughly 20 items per day. A single request with default options returns the feed's items as JSON, each with title, summary, author, publication date, content link and category tags. If the feed is served behind a JavaScript challenge, add render_js and the request costs 5 credits instead of the base rate. If the target also blocks datacenter IPs, add stealth_proxy, which costs 75 credits. Over a month at one request per hour, the default path is 720 requests; the stealth path is 720 requests at 75 credits each. That difference is the main reason to test the plain request first and only escalate when you actually see blocks.
Empty feed vs. blocked request
These two failures look similar in a log but need different fixes. An empty result from a well-formed feed usually means the feed itself has no current items, the URL points to an archived or malformed feed, or your parser is reading the wrong element. A blocked request usually returns an error status, a CAPTCHA page, or a challenge body instead of feed XML. Check the raw response first: if it is feed XML with no items, the problem is upstream; if it is HTML or an error, the problem is access, and stealth_proxy is the option to try.
When a plain parser is enough
If the feed is public, well-formed and served as static XML, a standard feed parser in your language will read it without any scraping service. A hosted scraper adds value when the feed is behind JavaScript, rate-limited by IP, geo-restricted, or when you want the same request shape to also handle rendered article pages and AI extraction. The decision checklist: (1) is the feed static XML? use a parser; (2) does it return a challenge or block? use the API with stealth_proxy; (3) do you need fields not in the feed? use extract_rules or ai_extract_rules; (4) do you need geo-specific content? add country_code.
Credit mechanics
Requests start at 1 credit each. Enabling JS rendering costs 5 credits per successful request. Premium proxies cost 10 credits without JS rendering or 25 with it. stealth_proxy costs 75 credits. API credits are valid for one month and do not roll over. The plan table below shows the captured plan tiers and their included credits and concurrency; treat the figures as a dated reference and confirm current pricing before committing.
Scope and legal care
Feeds that anyone can read without logging in are generally fair game for collection, but the site's own terms still govern how you may use that data, so read them before you ship a pipeline. Content placed behind a login sits outside this scope. Whether a particular use is lawful depends on your jurisdiction and the terms involved; the notes here are orientation, not legal advice.
| Output field | Request option that affects it | Notes |
|---|---|---|
| Article title | default request | Returned from feed metadata |
| Summary | default request | Returned from feed description |
| Author | default request | Absent if the feed omits it |
| Publication date | default request | Used for deduplication |
| Content link | default request | Stable dedupe key |
| Category tags | default request | Returned when present in feed |
| Custom fields | extract_rules (CSS selectors) | For fields not in the feed |
| Natural-language fields | ai_extract_rules | AI-driven extraction |
| Geo-specific content | country_code | 200+ countries and territories |
| Blocked targets | stealth_proxy | 75 credits per request |
Field availability depends on what each feed publishes; the scraper does not synthesize missing values.
| Plan | Price /mo | API credits | Concurrent requests | Priority support | Account manager | Team management |
|---|---|---|---|---|---|---|
| Hobby | $19 | 75,000 | 25 | Not included | Not included | Not included |
| Freelance | $49 | 250,000 | 50 | Not included | Not included | Not included |
| Startup (Recommended) | $99 | 1,000,000 | 100 | Not included | Not included | Not included |
| Business | $249 | 3,000,000 | 200 | Included | Included | Included |
| Business + | $599 | 8,000,000 | 400 | Included | Included | Included |
| Enterprise 14 | $999 | 14,000,000 | 500 | Included | Included | Included |
| Enterprise 24 | $1,599 | 24,000,000 | 600 | Included | Included | Included |
| Enterprise 41 (Recommended) | $2,399 | 41,000,000 | 700 | Included | Included | Included |
| Enterprise 69 | $3,599 | 69,000,000 | 800 | Included | Included | Included |
| Enterprise 120 | $5,799 | 120,000,000 | 900 | Included | Included | Included |
Prices exclude VAT. Credits are valid for one month and do not roll over. Figures are a dated reference; confirm current pricing before purchase.
Frequently asked questions
Do I need an account to scrape RSS?
You need an API key from the service, not an account on the target feed. Sign up, get the key and free test credits, then send the feed URL; the service handles the fetch.
Is scraping RSS legal?
Feeds that anyone can read without logging in are generally fair game for collection, but the site's own terms still govern how you may use that data, so read them before you ship a pipeline. Content placed behind a login sits outside this scope. Whether a particular use is lawful depends on your jurisdiction and the terms involved; the notes here are orientation, not legal advice.
What data can I extract from RSS?
You can capture any field the feed exposes to anonymous visitors. Point extract_rules at CSS selectors to name the fields you want, or describe them in plain language with ai_extract_rules and let the service map them; either route returns structured JSON. The standard feed fields are title, summary, author, publication date, content link and category tags.
How do I start scraping RSS?
Sign up, grab your API key, and send a GET request with the target URL. The service renders the page, rotates proxies and returns the data. Free test credits are available.
Will I get blocked when scraping RSS?
Requests leave through rotating residential proxies and a genuine headless browser, which keeps their traffic profile close to an ordinary visitor's. When a target tightens its defenses anyway, switch on stealth_proxy for that request.
How much does it cost to scrape RSS?
Free test credits are available on signup with no card required. Paid plans start at $19/month for higher volume and concurrency. Credit cost per request depends on options: 5 credits for JS rendering, 25 for premium_proxy with JS, 75 for stealth_proxy. Treat these figures as a dated reference and confirm current pricing.