Skip to content

RSS Feed Scraper for News Pipelines

RSS is a simple, well-known format, but feeds still break, hide behind anti-bot layers, or need to be merged with rendered pages.

OUTPUT FORMATSHTMLJSONMarkdown

This page covers how a hosted scraping API turns a feed URL into structured article records, which request options change the result and the credit cost, and how to choose between a plain parser and a rendering service.

What the scraper returns

You send a public RSS feed URL with your API key. The service fetches the feed, renders it in a real browser when needed, rotates the IP, and returns either structured JSON or rendered HTML. The article fields available on a feed are title, summary, author, publication date, content link and category tags. If a feed omits one of these, that field is simply absent from the output for that item — the scraper does not invent values.

Three workflows teams build

The common patterns are: (1) aggregating headlines from several feeds into one stream, (2) monitoring when a competitor or industry site publishes new posts, and (3) feeding fresh article text into an LLM pipeline for summarization or topic tagging. Each of these is a read-only pull of public feed data, so the main design question is how often to poll and how to deduplicate by content link or publication date.

How a request is built

The workflow is four steps. First, create an account to get an API key and free test credits. Second, install a client SDK for Python, Node, Go, PHP or Ruby, or call the API directly. Third, send the feed URL with your key as a parameter. Fourth, add advanced parameters only when the feed needs them: render_js for JavaScript, stealth_proxy for aggressive anti-bot targets, country_code for geo-targeting, and ai_extract_rules for natural-language extraction. Each scrape becomes a single function call; proxy rotation and headless browser management are handled for you.

Field and control map

Use this to decide which request options affect which output fields. The left column is what you want back; the right column is the option that most directly affects it.

Worked example: one feed in, JSON out

Suppose you monitor a public news feed that publishes roughly 20 items per day. A single request with default options returns the feed's items as JSON, each with title, summary, author, publication date, content link and category tags. If the feed is served behind a JavaScript challenge, add render_js and the request costs 5 credits instead of the base rate. If the target also blocks datacenter IPs, add stealth_proxy, which costs 75 credits. Over a month at one request per hour, the default path is 720 requests; the stealth path is 720 requests at 75 credits each. That difference is the main reason to test the plain request first and only escalate when you actually see blocks.

Empty feed vs. blocked request

These two failures look similar in a log but need different fixes. An empty result from a well-formed feed usually means the feed itself has no current items, the URL points to an archived or malformed feed, or your parser is reading the wrong element. A blocked request usually returns an error status, a CAPTCHA page, or a challenge body instead of feed XML. Check the raw response first: if it is feed XML with no items, the problem is upstream; if it is HTML or an error, the problem is access, and stealth_proxy is the option to try.

When a plain parser is enough

If the feed is public, well-formed and served as static XML, a standard feed parser in your language will read it without any scraping service. A hosted scraper adds value when the feed is behind JavaScript, rate-limited by IP, geo-restricted, or when you want the same request shape to also handle rendered article pages and AI extraction. The decision checklist: (1) is the feed static XML? use a parser; (2) does it return a challenge or block? use the API with stealth_proxy; (3) do you need fields not in the feed? use extract_rules or ai_extract_rules; (4) do you need geo-specific content? add country_code.

Credit mechanics

Requests start at 1 credit each. Enabling JS rendering costs 5 credits per successful request. Premium proxies cost 10 credits without JS rendering or 25 with it. stealth_proxy costs 75 credits. API credits are valid for one month and do not roll over. The plan table below shows the captured plan tiers and their included credits and concurrency; treat the figures as a dated reference and confirm current pricing before committing.

Field and control map
Output fieldRequest option that affects itNotes
Article titledefault requestReturned from feed metadata
Summarydefault requestReturned from feed description
Authordefault requestAbsent if the feed omits it
Publication datedefault requestUsed for deduplication
Content linkdefault requestStable dedupe key
Category tagsdefault requestReturned when present in feed
Custom fieldsextract_rules (CSS selectors)For fields not in the feed
Natural-language fieldsai_extract_rulesAI-driven extraction
Geo-specific contentcountry_code200+ countries and territories
Blocked targetsstealth_proxy75 credits per request

Field availability depends on what each feed publishes; the scraper does not synthesize missing values.

Plan reference (captured figures)
PlanPrice /moAPI creditsConcurrent requestsPriority supportAccount managerTeam management
Hobby$1975,00025Not includedNot includedNot included
Freelance$49250,00050Not includedNot includedNot included
Startup (Recommended)$991,000,000100Not includedNot includedNot included
Business$2493,000,000200IncludedIncludedIncluded
Business +$5998,000,000400IncludedIncludedIncluded
Enterprise 14$99914,000,000500IncludedIncludedIncluded
Enterprise 24$1,59924,000,000600IncludedIncludedIncluded
Enterprise 41 (Recommended)$2,39941,000,000700IncludedIncludedIncluded
Enterprise 69$3,59969,000,000800IncludedIncludedIncluded
Enterprise 120$5,799120,000,000900IncludedIncludedIncluded

Prices exclude VAT. Credits are valid for one month and do not roll over. Figures are a dated reference; confirm current pricing before purchase.

Frequently asked questions

Do I need an account to scrape RSS?

You need an API key from the service, not an account on the target feed. Sign up, get the key and free test credits, then send the feed URL; the service handles the fetch.

Is scraping RSS legal?

Feeds that anyone can read without logging in are generally fair game for collection, but the site's own terms still govern how you may use that data, so read them before you ship a pipeline. Content placed behind a login sits outside this scope. Whether a particular use is lawful depends on your jurisdiction and the terms involved; the notes here are orientation, not legal advice.

What data can I extract from RSS?

You can capture any field the feed exposes to anonymous visitors. Point extract_rules at CSS selectors to name the fields you want, or describe them in plain language with ai_extract_rules and let the service map them; either route returns structured JSON. The standard feed fields are title, summary, author, publication date, content link and category tags.

How do I start scraping RSS?

Sign up, grab your API key, and send a GET request with the target URL. The service renders the page, rotates proxies and returns the data. Free test credits are available.

Will I get blocked when scraping RSS?

Requests leave through rotating residential proxies and a genuine headless browser, which keeps their traffic profile close to an ordinary visitor's. When a target tightens its defenses anyway, switch on stealth_proxy for that request.

How much does it cost to scrape RSS?

Free test credits are available on signup with no card required. Paid plans start at $19/month for higher volume and concurrency. Credit cost per request depends on options: 5 credits for JS rendering, 25 for premium_proxy with JS, 75 for stealth_proxy. Treat these figures as a dated reference and confirm current pricing.