Skip to content

Turn any web page into clean Markdown or plain text

Most web pages arrive wrapped in navigation bars, ad slots, tracking scripts, and layout markup that add nothing to a training set.

OUTPUT FORMATSHTMLJSONMarkdown

A Markdown scraper API lets you request the readable part of a page in a format a language model can consume directly. This page explains the two output parameters, what each returns, and how to pick between them for an AI data pipeline.

What the two output parameters do

The API exposes two request parameters that change the response format. return_page_markdown returns the page as Markdown, preserving headings, lists, and links so the document keeps its structure. return_page_text returns only the meaningful text, stripping HTML, ads, and clutter. Both are sent as part of the normal request; you do not need a separate endpoint. The exact request shapes for Python, JavaScript, and PHP are supplied in the protected code modules referenced below.

Choosing Markdown or plain text

Use Markdown when downstream consumers need structure: RAG chunking that respects headings, documentation mirrors, or content that will be re-rendered. Use plain text when you want the smallest token footprint and the model does not need section boundaries. A quick checklist:

  • Need headings and lists preserved? Choose Markdown.
  • Feeding a token-limited context window? Plain text is usually leaner.
  • Re-publishing or diffing content? Markdown keeps links and structure visible.
  • Only extracting sentences for classification? Plain text removes the markup you would otherwise strip yourself.

A worked example

The supplied Python snippet (code0001) installs the client library, creates a client with your API key, and calls client.get('YOUR-URL', params={'return_page_markdown': 'True'}). The response body is the Markdown version of the target page. The sample output (code0007) shows the shape you can expect: a top-level heading, an italic subtitle, a horizontal rule, then sections with bold terms and bullet lists. That structure is what makes the output usable for chunking and retrieval without a separate HTML parser.

Field and control map

The interactive demo on this page exposes the following controls (see the protected form-controls module for exact attributes):

  • Output mode toggle: Markdown Content / Text Content.
  • Language selector: Python, JavaScript, PHP — six combined options covering both output modes.
  • Copy button for the generated snippet.
  • Output panel showing the .md result.

The two request parameters map directly to the toggle: return_page_markdown for Markdown, return_page_text for plain text.

Two different failure modes

It helps to separate two problems that both look like 'the scrape did not work'. The first is access: anti-bot systems block requests at scale, and the fix is proxy rotation and JavaScript rendering rather than output formatting. The second is noise: the request succeeds but the HTML pollutes the dataset, and the fix is exactly the output parameters described here. If your pipeline is failing on access, changing the output format will not help; if it is failing on data quality, the format change is the lever.

Credit cost

AI web scraping requests cost an additional {{OWNER_AI_CREDIT_SURCHARGE}} credits on top of the regular API cost. The plan tooltips also describe per-request credit rules for JavaScript rendering ({{OWNER_JS_RENDERING_CREDIT_COST}} credits per successful request) and premium proxies ({{OWNER_PREMIUM_PROXY_CREDIT_COST}} credits depending on JavaScript rendering); these figures are tooltip-only and should be confirmed against the current plan page before budgeting.

Frequently asked questions

How much does the AI Web Scraping API cost?

AI web scraping requests cost an additional {{OWNER_AI_CREDIT_SURCHARGE}} credits on top of the regular API cost.

How do I get started using AI Web Scraping?

Register an account to receive {{OWNER_FREE_TRIAL_TERMS}} free API calls, then review the AI data-extraction documentation before sending your first request.

Can I use the AI Web Scraping feature for data extraction projects at scale?

Yes. The AI web scraping functionality is a complementary feature to the existing scraping platform, which is built to process millions of requests a day.

How do I contact customer support to help me with my AI Web Scraping project?

Support channels: {{OWNER_SUPPORT_CHANNELS}}. Agents who are themselves scraping experts can help you scale a project.

Can I scrape dynamic JavaScript websites with the AI Web Scraping feature?

Yes. The AI web scraping support is built on top of the existing platform and can use its broad JavaScript rendering support.

# Extracting Data at Scale with ScrapingBee  

*Turn any website into structured, usable data.*

---

## Introduction  

In the world of web automation, **data extraction** is the backbone of insight.  

ScrapingBee simplifies the process — handling **JavaScript rendering**, **proxies**, and **CAPTCHAs**, so you can focus on **collecting clean, structured data**.

Whether you're gathering product prices, blog articles, or real estate listings, ScrapingBee lets you scrape and extract data efficiently with just a few lines of code.

---

## Why Data Extraction Matters  

Businesses rely on extracted data to drive smarter decisions:

- 🛒 **E-commerce Monitoring** — Track product prices and availability.  

- 📰 **News & Content Aggregation** — Collect headlines or summaries.  

- 📊 **Market Research** — Analyze competitors and industry trends.  

- 💬 **Social Listening** — Gather reviews or comments across platforms.  

- 🏘 **Real Estate Listings** — Keep listings up-to-date automatically.  

---

## How ScrapingBee Helps  

ScrapingBee provides **a fast, reliable, and scalable API** that abstracts the complexities of web scraping.

### Key Extraction Capabilities  

- **Automatic HTML Rendering** — Perfect for sites built with React, Vue, or Angular.  

- **Smart Parsing Options** — Extract structured data like JSON or XPath.  

- **Built-in Proxy Rotation** — Avoid IP bans and captchas automatically.  

- **Header & Cookie Control** — Emulate any browser environment easily.  

---

## Example: Extracting Product Data

### 1️⃣ Make the Request

curl "{{API_BASE_URL}}/v1/?api_key=YOUR_API_KEY&url=https://example.com/products&extract_rules=rules.json"
Extracting Data at Scale with ScrapingBee

Turn any website into structured, usable data.

Introduction

In the world of web automation, data extraction is the backbone of insight.

ScrapingBee simplifies the process — handling JavaScript rendering, proxies, and CAPTCHAs, so you can focus on collecting clean, structured data.

Whether you're gathering product prices, blog articles, or real estate listings, ScrapingBee lets you scrape and extract data efficiently with just a few lines of code.

Why Data Extraction Matters

Businesses rely on extracted data to drive smarter decisions:

E-commerce Monitoring — Track product prices and availability.

News & Content Aggregation — Collect headlines or summaries.

Market Research — Analyze competitors and industry trends.

Social Listening — Gather reviews or comments across platforms.

Real Estate Listings — Keep listings up-to-date automatically.

How ScrapingBee Helps

ScrapingBee provides a fast, reliable, and scalable API that abstracts the complexities of web scraping.

Key Extraction Capabilities

Automatic HTML Rendering — Perfect for sites built with React, Vue, or Angular.

Smart Parsing Options — Extract structured data like JSON or XPath.

Built-in Proxy Rotation — Avoid IP bans and captchas automatically.

Header & Cookie Control — Emulate any browser environment easily.

Example: Extracting Product Data

1️⃣ Make the Request

curl "{{API_BASE_URL}}/v1/?api_key=YOUR_API_KEY&url=https://example.com/products&extract_rules=rules.json"