Skip to content

Scrape a Whole Website into Clean Text for LLM Training

Building an AI feature often starts with getting usable text out of web pages. The same collection pipeline can feed a retrieval system, fine-tuning examples, or a domain corpus for continued pretraining, but each use places different demands on volume, format, and freshness. This guide walks through a Python workflow that discovers every page, fetches it, strips away boilerplate, and saves the result in a form your downstream pipeline can actually use.

The short version

Extraction quality matters more than raw speed. A fast crawl that returns navigation, footers, and ads produces a worse dataset than a slower crawl paired with a proper boilerplate remover. Markdown is the default format for most LLM ingestion because it preserves heading hierarchy and link context while avoiding the token cost of raw HTML. Plain text loses structure, and HTML wastes tokens on markup the model does not need.

Pick the path first: RAG, fine-tuning, or continued pretraining

Decide your path before you crawl, because the three paths want different things. RAG: small volume, roughly MBs to low GBs, markdown output, with clean chunk boundaries and fresh content as the priority. Fine-tuning: moderate volume, roughly GBs, markdown or JSONL, with consistency and formatting accuracy as the priority. Continued pretraining: very large volume, tens to hundreds of GBs or more, plain text or markdown at scale, with deduplication and noise removal as the priority. RAG treats scraped text as a lookup index the model never learns, so freshness and retrieval boundaries matter more than volume. Fine-tuning bakes patterns into weights, so consistency across examples matters more than raw size. Continued pretraining is closest to base-model training on a narrower domain, so corpus-level hygiene dominates.

Setup and prerequisites

You need basic Python and general web-scraping familiarity, Python 3 installed, and an editor or IDE. Create a project folder with two files: find_links.py for URL discovery and extract_data.py for downloading and extracting text. Initialize the project and virtual environment with the supplied module code0001, then install dependencies with module code0002. The roles are: requests for HTTP, BeautifulSoup for HTML parsing, lxml as its parser, pandas for tabular handling, trafilatura to isolate main content, and a proxy client module for routed requests when blocks appear. This section is general information, not legal advice; have LLM-training scraping reviewed by a lawyer if it is central to your business.

Respect robots.txt, opt-outs, and PII

Check robots.txt and the target site's Terms of Service before crawling, and keep request rates reasonable so you do not overload the server. robots.txt is a voluntary standard formalized as RFC 9309: it has no independent legal force, but courts have treated compliance or disregard as evidence of good or bad faith, and ignoring it is not by itself a DMCA circumvention violation. AI crawlers now split training from search, so a site may block training use while allowing indexing and citation; check the specific agent names rather than assuming one blanket AI-bot rule or only a generic Disallow. Under the EU AI Act, providers of general-purpose models must publish a sufficiently detailed summary of training content and maintain a copyright policy respecting machine-readable rights reservations under the EU text-and-data-mining exception; those obligations began applying in August 2025, with enforcement provisions taking effect in August 2026. llms.txt is complementary, not a substitute: robots.txt governs access, while llms.txt guides AI systems toward content a site considers most relevant, typically for retrieval and citation, and adding llms.txt does not grant scraping permission. Screen for PII such as names and emails before it enters your dataset, and note that GDPR obligations apply in addition to AI Act requirements where EU residents' personal data is involved.

Find every page via the sitemap

Start with the sitemap. The discovery script in modules code0003, code0004, and code0005 fetches a sitemap URL, parses it with BeautifulSoup, recurses into nested sitemaps, extracts loc, lastmod, and priority from each url entry, and appends rows to a CSV file. If a site has no sitemap or an incomplete one, fall back to crawling internal links from the homepage, or check robots.txt for a Sitemap line pointing to an index you might otherwise miss. Run it with module code0006. If sitemap requests are blocked, route them through a proxy client as described below.

Fetch pages and extract clean text

Load URLs from the CSV with module code0009, which handles missing, empty, or unparsable files and filters out blank or n/a entries. The baseline fetcher in module code0010 skips non-HTML responses, removes script, style, and noscript elements, and returns soup.get_text(). Save results as UTF-8 text with module code0011 and wire everything together with module code0012. That baseline runs, but it has two flaws: requests may be blocked, and get_text() returns the whole page. On the cited Zyte article-extraction benchmark, BeautifulSoup scored F1 0.665 with precision 0.499, meaning nearly half of a plain get_text() return is boilerplate, while trafilatura scored F1 0.958 with 0.938 precision. Those numbers are measured mostly on article-style pages; on the WCXB benchmark of 2,008 annotated pages across seven page types, trafilatura fell from roughly 0.92 on articles to 0.55-0.58 on forums and listing pages. Expect article-like results on docs and blogs, but spot-check forums, product listings, and other non-article layouts yourself before trusting them at scale. Replace only the extraction step, as in module code0013, using trafilatura.extract with markdown output and returning None when no main content is found. A small first-hand comparison over 20 documentation and blog pages, summarized in module code0019, showed trafilatura output only about 3% smaller than get_text() on average, far less than the Zyte benchmark would predict for article pages; treat that as an illustration of real-world variance, not a general benchmark, and prefer the Zyte and WCXB figures for the general case.

Choose the output format

Markdown is the default for RAG because most chunkers split on headings and markdown carries far fewer tokens than HTML for the same content. For fine-tuning, save markdown now and convert to JSONL, one structured record per line, when you assemble training examples. For large-scale continued pretraining, plain text is sufficient because structure is discarded during tokenization anyway and deduplication and filtering take priority. In short: RAG means markdown, fine-tuning means markdown now and JSONL later, continued pretraining means plain text at scale.

Use proxies when direct requests get blocked

When sites block direct requests, replace the plain request call in module code0021 with a proxy-routed call such as module code0022, which shows premium proxy, country, JavaScript rendering, and device parameters. The API can also return markdown or clean text directly, which removes the separate trafilatura step if you prefer not to run your own extraction pass. Store any API token in an environment variable rather than hardcoding it, as shown in modules code0015 and code0016, and add lxml_html_clean with module code0017 if your benchmark setup needs it.

Add concurrency and retries

Most crawl time is spent waiting for HTTP responses, so process pages concurrently. Add a worker limit such as module code0025 and update main() with module code0026, which submits the existing fetch function to a thread pool and collects results as they complete. Keep the worker count reasonable; too many workers can overload the target site or trigger rate limits. For robustness against temporary server errors, timeouts, and network hiccups, install tenacity with module code0027, import it with module code0028, and wrap the request in a retry helper such as module code0029, which retries up to three times with exponential backoff. Update the fetcher to use that helper as in module code0030; the same retry pattern can wrap a proxy client call. Module code0031 shows the full updated script combining concurrency and retries.

Clean the dataset before you use it

Saving text is not the finish line. Deduplication is the highest-value step: hash normalized text with SHA-256 as in module code0032 to drop exact matches, then use MinHash or SimHash for near-duplicates, with datasketch installed via module code0033 and used as in module code0034. Note that the 0.85 threshold in that snippet assumes n-gram shingles, while the snippet shingles on single words, a coarser signal, so you may need a higher threshold tuned against your own corpus to avoid over-flagging distinct pages. Catch surviving boilerplate by counting strings that repeat verbatim across many different documents and removing them stopword-style. Drop documents that are too short, navigation-only with a high link-to-text ratio, or mostly code or markup when that is not what you collect. Screen for PII with pattern matching before data lands in your store, using a dedicated PII-detection library for higher-stakes cases. Store at minimum the source URL, fetch date, and content hash alongside the text, so you can refresh stale documents, audit provenance, and respond to takedown requests. For RAG, split on heading boundaries rather than fixed character counts, since fixed-size chunking cuts sentences and sections arbitrarily. Decide a re-crawl cadence up front, weekly, monthly, or tied to the sitemap lastmod values already collected, because a corpus with no refresh plan rots.

How much data, and when to use an API instead

Teams train custom models for customization to specific tasks or industries, privacy by keeping sensitive data in their own environment, and performance on a narrow domain. Most scraping projects should still focus on fine-tuning an existing model or building RAG rather than training from scratch. Data needs scale sharply: continued pretraining ranges from roughly 1-5 million words for small models up to billions or trillions of words at frontier scale, where 1 billion words is roughly 900 times the entire Harry Potter series; fine-tuning typically needs low thousands to tens of thousands of examples; RAG often needs just a few hundred to a few thousand documents. A self-hosted requests plus trafilatura pipeline is free and genuinely enough for a few thousand pages on cooperative sites. Reach for a managed scraping API when you hit blocks that rotating proxies and fingerprinting would absorb, when pages are JS-rendered and need a headless browser, when you want markdown back without running your own extraction step, or when the crawl must run reliably on a schedule unattended. Other tools in this space include Firecrawl for LLM-ready markdown, Apify as a broader actor marketplace, and ScrapeGraphAI as an LLM-driven extraction layer; the right fit depends on volume, budget, and how much of the pipeline you want to own.

Pre-crawl checklist

Before you crawl: confirm the path (RAG, fine-tuning, or continued pretraining) and its volume and format. Check robots.txt agent names, Terms of Service, and machine-readable opt-outs. Discover URLs from the sitemap, falling back to homepage links or a robots.txt Sitemap line. Extract main content with a boilerplate remover as markdown, not raw get_text(). Add proxies only when blocks appear, keep concurrency modest, and add retries with backoff. Deduplicate exactly and nearly, filter low-quality pages, screen PII, and store URL, fetch date, and content hash with every record. Chunk on headings for RAG, convert to JSONL for fine-tuning, and set a re-crawl cadence before you rely on the data.

Frequently asked questions

Why scrape website text for LLM training?

Domain-specific pages supply the material for three distinct uses: a retrieval corpus for RAG, examples for fine-tuning an existing model, or a larger corpus for continued pretraining. Collecting relevant pages is how you obtain that material at scale.

What is the best way to get clean text from a URL for an LLM?

Fetch the page and pass it through a boilerplate remover such as trafilatura rather than taking everything with a raw get_text() call. In the cited Zyte benchmark, plain BeautifulSoup scored F1 0.665 versus trafilatura at 0.958. Request markdown output so heading structure remains available for later chunking.

Can I use scraped web text for RAG instead of training a model?

Yes, and for many teams it is the better first option. RAG needs far less data, no training run, and can be refreshed by re-crawling. Keep markdown headings so chunks split on real section boundaries, and store the source URL with every chunk so answers can cite it.

Can I scrape any website for LLM training?

Permission is not automatic. Start by reviewing the site's robots.txt and Terms of Service. That file does not create legal force on its own, yet ignoring its preferences may be held against you in a dispute. For models offered in the EU, the AI Act requires respect for machine-readable reservations of rights. Content being publicly reachable does not make every reuse lawful.

What is an LLM scraper?

The term is used two ways: a scraper that collects data from LLM products, or a scraper that uses an LLM to extract structured fields. This guide covers a third job: collecting web text to feed into an LLM for RAG, fine-tuning, or pretraining.

How do I handle duplicate pages in a scraped dataset?

Begin with a hash of each document to remove identical copies, then apply similarity-based methods such as MinHash to catch near-copies that share substantial boilerplate. Keep the hash and collection date alongside each record so later crawls can be compared. Removal should happen before indexing or training.

How much text do I need to train an LLM?

Scale depends on the intended use. Continued pretraining can require anywhere from millions to billions of words. Fine-tuning usually involves thousands to tens of thousands of examples. Retrieval needs the smallest collection, often hundreds to a few thousand documents, because relevant passages are fetched when a query arrives rather than absorbed during training.

Is raw scraped data ready for training?

No. Raw scraped data usually contains navigation menus, ads, duplicate pages, and other noise that survives extraction. Clean, deduplicate, and screen it for PII before use, using exact-hash and near-duplicate methods, quality filters, and metadata stored alongside the text.

Why use proxies for web scraping?

Websites may block repeated requests from the same IP address. Proxies, or a proxy API, help reduce blocks and make large-scale scraping more reliable.

Should I use multithreading when scraping websites?

It can help, within limits. Running several pages in parallel reduces waiting time, but an excessive worker count risks overwhelming the destination or hitting throttling.