Skip to content

CLI Web Scraping: Batches, Crawls, and RAG-Ready Output from the Terminal

Fetching one page is a solved problem. The cost shows up later: a hundred URLs to process, a retry policy to maintain, an export format to keep stable, and a schedule that has to keep running unattended. This guide follows a terminal-based workflow that keeps those operational pieces together, so the same command can start as a single fetch and grow into a batch job, a bounded crawl, or a recurring pipeline that feeds an LLM or a retrieval system.

When a terminal scraping workflow is the right fit

A command-line scraper earns its place when the work is repetitive and the output needs to land somewhere predictable. It is a poor fit when the page requires heavy custom interaction logic that only a full application can express.

Use this checklist before committing to a CLI workflow:

  • The job starts from a list of URLs, a sitemap, or a site to crawl.
  • Output must be saved as files, CSV, NDJSON, Markdown, or text.
  • The job will run more than once, so scheduling and resume matter.
  • Failures should be retried with backoff instead of restarting from zero.
  • The data feeds an app, an agent, a spreadsheet, or a retrieval pipeline.

If most boxes are ticked, the CLI layer removes plumbing that would otherwise be rewritten for every new project.

Batch scraping from a URL list or CSV

Batch mode takes an input file and writes each result into an output directory. Plain text lists work directly; CSV inputs need the column that holds the URL. The two commands below show both shapes. The first also requests Markdown output per page.

The exact command syntax is preserved in the protected command modules that accompany this page; treat those as the authoritative reference for flags and arguments.

Crawling a site or sitemap with sane limits

Crawling starts from a seed URL and follows links. Two limits keep discovery bounded: maximum depth and maximum page count. A sitemap crawl skips link discovery and reads the URL map directly, which suits documentation sites, blogs, and help centers that already publish one.

A practical rule: set the page cap first, then the depth. Depth controls how far the crawler wanders; the page cap controls how much it can spend. Include and exclude patterns narrow the crawl to the sections that matter, for example a docs path while skipping tag and archive pages.

Scheduling recurring jobs

A built-in scheduler turns a crawl into a repeatable workflow. The scheduler relies on the system cron and is treated as an advanced feature because it executes shell commands, so it must be enabled deliberately. That guardrail is worth keeping: automated scraping jobs should be opt-in, not a default side effect of installing a tool.

Extraction: from natural-language queries to path-based rules

Raw HTML is useful for debugging and rarely useful downstream. Three extraction styles cover most needs:

  • A natural-language AI query for fields described in plain words, such as an article title, author, publish date, and summary.
  • Structured extraction rules that return named JSON fields, such as product name, price, and availability.
  • Path-based smart extraction across HTML, JSON, Markdown, CSV, NDJSON, XML, or plain text when the target is a specific element or attribute.

For spreadsheet-friendly output, extract structured fields first and then export them as CSV with flattened columns. That covers the classic table-to-CSV task without writing a new parser each time.

Worked example: a docs site into chunked NDJSON for RAG

Suppose a documentation site needs to become retrieval-ready data. A workable sequence:

  1. Crawl the docs section with a depth and page cap, requesting Markdown output into a directory.
  2. Export the directory as a single NDJSON file so downstream loaders see one stream.
  3. For a single long page, request Markdown with a chunk size and overlap so each chunk can become an embedding record.

The result keeps the source URL attached to each record, which matters when a retrieval answer needs to cite its origin. That fetch-clean-preserve-ship stage is the one most teams end up hand-rolling; here it is a fixed sequence of commands.

Cost guardrails before a job runs

Waste at scale comes from a short list of causes: the same URL queued twice, a crawl that follows every link it finds, and a batch that dies halfway and starts over. Each has a matching control that can be applied before the first request:

  • Check usage and plan concurrency.
  • Run a small random sample of the batch first.
  • Deduplicate URLs so the same page is not paid for twice.
  • Resume an interrupted batch instead of restarting.
  • Add retries and backoff for transient failures.
  • Bound crawls with depth, page caps, and include/exclude patterns.

None of these controls is interesting on its own. Together they turn a job whose cost is unknown into one whose cost is bounded, which is what makes a pipeline safe to leave running.

Fitting the CLI into an existing stack

Because the tool is a plain command, it composes with whatever already runs the project: shell scripts, Makefiles, CI jobs, data pipelines, cron-style automation, and Python workflows. Once output is saved as Markdown, CSV, JSON, or NDJSON, the next step is ordinary: load it into a vector store, hand it to a Python script, commit it as test fixtures, or pipe it into an evaluation run.

With AI coding assistants, the division of labor shifts. The assistant drafts the command, you run it and inspect the files, and the application logic is written against data that already exists rather than against a scraper that may or may not work.

Failure distinction: transient vs configuration errors

Not every failed request deserves a retry. Separating the two categories saves credits and debugging time.

SymptomLikely categoryFirst response
Timeout or connection reset on a single URLTransientRetry with backoff
Repeated failures on the same URL after retriesPersistent target issueInspect the page manually; adjust rendering or headers
Batch stops partway throughInterrupted runResume from the output directory
Crawl returns far more pages than expectedConfigurationTighten depth, page cap, include/exclude patterns
Duplicate rows in outputConfigurationEnable deduplication before the run
Extraction returns empty fieldsRule mismatchRe-check the extraction rules or path selectors

Retries and backoff belong to the first row. The rest are configuration or target problems that retries will not fix.

Field and control map

The table below maps the common flags to their purpose so a command can be assembled without guesswork. Exact syntax lives in the protected command modules.

ControlPurpose
--input-fileSource list of URLs (text or CSV)
--input-columnColumn holding the URL in a CSV input
--output-dirDirectory for per-page results
--output-fileSingle-file output target
--return-page-markdownReturn each page as Markdown
--chunk-size / --chunk-overlapSplit long Markdown into overlapping chunks
--ai-queryNatural-language extraction instruction
--ai-extract-rulesNamed JSON fields to extract
--smart-extractPath-based extraction across formats
--max-depth / --max-pagesCrawl bounds
--include-pattern / --exclude-patternCrawl path filters
--from-sitemapCrawl from a published sitemap
--sampleRandom subset of a batch
--deduplicateRemove duplicate URLs
--resumeContinue an interrupted batch
--retries / --backoffRetry policy for transient failures
--every / --nameSchedule interval and job name
--format / --flatten / --columnsExport shape

Where to go next

Pick one target that has defeated your earlier scripts: a page that only renders with JavaScript, a long product list, a docs site you want as Markdown, or a small corpus for a retrieval pipeline. Run a single command against that target before designing anything larger. If the output is usable, the same command is the seed of the batch, crawl, or schedule you actually need. Setup steps vary by tool and release, so confirm the current install and authentication instructions in the tool's own documentation before relying on any command in production.

Transient vs configuration failures
SymptomLikely categoryFirst response
Timeout or connection reset on a single URLTransientRetry with backoff
Repeated failures on the same URL after retriesPersistent target issueInspect the page manually; adjust rendering or headers
Batch stops partway throughInterrupted runResume from the output directory
Crawl returns far more pages than expectedConfigurationTighten depth, page cap, include/exclude patterns
Duplicate rows in outputConfigurationEnable deduplication before the run
Extraction returns empty fieldsRule mismatchRe-check the extraction rules or path selectors

Categories reflect the operational behavior of the described controls; they are not a claim about any specific error code.

Field and control map
ControlPurpose
--input-fileSource list of URLs (text or CSV)
--input-columnColumn holding the URL in a CSV input
--output-dirDirectory for per-page results
--output-fileSingle-file output target
--return-page-markdownReturn each page as Markdown
--chunk-size / --chunk-overlapSplit long Markdown into overlapping chunks
--ai-queryNatural-language extraction instruction
--ai-extract-rulesNamed JSON fields to extract
--smart-extractPath-based extraction across formats
--max-depth / --max-pagesCrawl bounds
--include-pattern / --exclude-patternCrawl path filters
--from-sitemapCrawl from a published sitemap
--sampleRandom subset of a batch
--deduplicateRemove duplicate URLs
--resumeContinue an interrupted batch
--retries / --backoffRetry policy for transient failures
--every / --nameSchedule interval and job name
--format / --flatten / --columnsExport shape

Control names are shown as described in the accompanying protected command modules; refer to those modules for exact syntax.

Frequently asked questions

What is a CLI, in plain English?

A CLI is a command-line interface: a tool you run from a terminal instead of clicking through a web UI. For developers, the advantage is that commands can be saved, repeated, scripted, automated, and dropped into existing workflows.

What makes this CLI different from a basic CLI for web scraping?

A basic CLI fetches a page and dumps the HTML. This one adds crawling, batch jobs, structured extraction, retries, exports, scheduling, Markdown output, and LLM-friendly formats on top of the fetch step.

Can I use it for LLM web scraping?

Yes. Pages can be returned as Markdown and chunked into NDJSON, which makes the output easier to feed into agents, embedding pipelines, vector databases, and other LLM workflows.

Is it useful for web scraping for RAG?

Yes. It handles the middle layer of RAG data prep: crawling the right URLs, cleaning content, preserving source data, chunking text, and exporting results in formats the rest of the pipeline can consume.

Can it replace my Python scraping scripts?

For many common jobs, yes: scraping URL lists, crawling docs, exporting Markdown, scheduling jobs, or turning scraped data into CSV or NDJSON. For custom application logic, Python still makes sense, but the CLI can handle the fetching and preparation first.