Skip to content

Command-line interface for web scraping, crawling and extraction

Install the CLI, authenticate once, then choose a single scrape, a crawl, a batch run, or an interactive session. This guide maps the main commands to their inputs, outputs and failure modes so you can pick the smallest workflow that fits the job.

Install, authenticate and check credits

Recommended installation uses uv without a virtual environment; the alternative is pip inside a virtual environment. Verify with the version command. Authentication can be interactive for first-time setup, non-interactive for CI/CD and scripts, or via environment variable with no stored file. The CLI also reads .env files in the current directory. Use the show-key command to see the stored location and logout to remove the stored key. Checklist: install, verify version, authenticate, run usage to confirm credits and concurrency before a large job.

Shell quoting: the most common avoidable failure

Always use single quotes around JSON data and path expressions. Double quotes trigger shell expansion: $ expands variables, backslash creates escape sequences, and backticks trigger command substitution. This matters for --ai-extract-rules, --extract-rules, --js-scenario, --smart-extract and --ai-query. URLs should use double quotes when they may contain & or ?. Decision rule: URL in double quotes, JSON or path expression in single quotes, side by side when both appear.

Interactive mode for iteration

Running the bare command opens an interactive REPL with a pinned banner, live credit toolbar, scrollback, tab completion and ghost-text suggestions. It is suited to exploring a site, tuning proxies, testing extraction rules, comparing search queries or building a batch workflow before scripting the final command. Session defaults set with :set apply to later commands and inline flags override them; :show displays active defaults and :unset removes them. Startup flags must precede the bare invocation. Inside the REPL, :view pages the last result with pretty JSON/HTML detection and a raw/pretty toggle, :list shows active schedules, and Ctrl+C stops a running command mid-request. If no key is configured, the prompt becomes a masked API-key input; auth and logout re-prompt in place. For existing output files, the shell prompts, while --overwrite skips the check in the REPL and in CI/CD. Slash commands and key bindings are enumerated in the REPL tables below.

Inside interactive mode, set defaults and override an individual request when needed.

:set country-code=fr
:set premium-proxy=true
scrape "https://example.com"        # runs with --country-code fr --premium-proxy true
scrape "https://example.com" --country-code us   # explicit flag wins
:unset premium-proxy

Scrape one page, with presets and saved configurations

The scrape command fetches a page through the HTML API. CLI flags map one-to-one to API parameters, with underscores becoming hyphens; predefined values accept hyphens or underscores, and boolean flags accept explicit true/false or a bare flag, with omission leaving the API default. Basic choices are printing HTML to stdout, saving to a file with auto-detected extension, enabling JavaScript rendering and premium proxy, extracting with AI, or returning markdown for LLM pipelines. Saved scraping configurations provide a reusable base that inline options override, and configuration names are case-sensitive and limited to alphanumeric characters, hyphens and underscores. Presets only set flags not already set, so explicit values win. CLI-only flags include escalate-proxy for retrying 403/429 responses with premium then stealth proxy, chunk-size and chunk-overlap for NDJSON text/markdown chunks, and force-extension when --output-file has no extension. Exact flag/API-parameter pairs, ranges and credit costs are in the scrape tables below.

Inside interactive mode, save the response to a file and explicitly allow an existing output file to be replaced.

scrape "https://example.com" --output-file result.html --overwrite

Crawl: choose quick, sitemap or project spider

Quick crawl starts from one or more URLs and follows same-domain links, saving numbered files plus manifest.json. Sitemap crawl parses a sitemap, including indexes recursively to depth 2, then crawls discovered URLs. Project spider mode runs a Scrapy spider with middleware and API key injected; there, parameters belong in the request object and CLI API flags are rejected, while infrastructure flags such as concurrency, download-delay and autothrottle still apply. Use include/exclude patterns to restrict followed URLs, save-pattern to visit pages for discovery but save only matching URLs, allowed-domains or allow-external-domains to widen scope, max-depth and max-pages to bound the job, and resume to skip URLs already in manifest.json. Non-HTML or extraction-focused crawls may run a discovery phase with cheap HTML requests before save requests, so credits can exceed the number of saved files; --confirm yes skips the prompt in scripts. Concurrency defaults to auto-detection from the plan limit and is capped by max-pages. Exact crawl parameters with types and defaults are in the crawl parameters table below.

Batch processing: test small, resume safely, enrich CSVs

Batch mode is enabled with --input-file on scrape, google and other scraper commands. Input supports .txt with one item per line and .csv with --input-column naming a column or 0-based index; - reads stdin. Default output is numbered files in batch_<timestamp> with manifest.json. Alternatives are a single CSV or NDJSON stream via --output-format. Before spending credits, --deduplicate normalizes URLs and removes duplicates, and --sample tests N random items. --update-csv enriches the original CSV in place, flattening nested JSON to dot-notation columns. --resume with the same output directory skips completed items, and bare --resume at the top level finds incomplete batches and prints the resume command. Retries default to 3 with exponential backoff multiplier 2. --post-process and --on-complete execute shell commands and require advanced-features setup; post-process does not apply to --update-csv. Deprecated --extract-field and --fields should be replaced by --smart-extract. Exact batch parameters with types and defaults are in the batch parameters table below.

Smart Extract: one path language across formats

--smart-extract performs client-side extraction with a path language and auto-detects JSON, HTML, XML, CSV, NDJSON, Markdown or plain text. It works on single-item runs for scrape, google, fast-search, Amazon, Walmart, YouTube, ChatGPT and Gemini commands. In scrape batches it applies to individual-file and CSV output, not NDJSON; on non-scrape commands batch support is not yet available, so use the deprecated extract-field or jq post-processing there. The language navigates keys, selects indexes or slices, lists keys/values, searches recursively with ...key or glob patterns, expands context with ~N, filters by substring, glob, regex or key property, and combines searches with | for OR or & for AND. Spaces around | and & are required, and the two operators cannot be mixed in one expression. Output can be raw values one per line, block syntax with separate {} pairs per field, or JSON schema mode matching --ai-extract-rules. A practical pattern is to find a price with a regex, expand two levels up to capture the surrounding product card, and emit named fields for downstream use. The path-language syntax, selection, dict, recursive-search, context, filter and operator forms are enumerated in the Smart Extract path tables below, and auto-format detection rules are in the format-detection table.

Export merged results and schedule recurring runs

Export merges numbered batch or crawl files into one file, using manifest.json to annotate records with source URLs. NDJSON is the default: valid JSON passes through with an added _url field and non-JSON is wrapped as content plus _url. TXT separates files with blank lines and # URL prefixes when a manifest exists. CSV expands JSON arrays into rows; --flatten recursively creates dot-notation columns, --flatten-depth raises the default depth of 5 for deeply nested data, --columns keeps only named columns and drops rows missing all of them, and --deduplicate removes identical rows. Schedule creates cron jobs on macOS/Linux to run any CLI command at an interval such as 5m, 1h or 2d, with a minimum of 1 minute; Windows returns a clear cron-unavailable message. Each schedule creates a tagged cron entry, logs to the CLI config logs directory, and tracks metadata in schedules.json. Use --list to inspect, --stop by name or all to remove, and --name to make a job identifiable. Scheduling requires advanced-features setup. Exact export parameters with types and defaults are in the export parameters table below.

Pipelines, scraper commands and gated shell features

Commands compose through stdin/stdout: read URLs from a pipe with --input-file -, stream NDJSON, filter with jq, or chain a search command into scrape. For LLM and RAG ingestion, crawl a documentation site to markdown or use chunk-size with overlap to emit NDJSON chunks with URL, index, total and timestamp metadata. For enrichment, start from a CSV and add live fields with --update-csv; for larger ETL work, scrape in parallel, export with flattening to CSV, and reshape downstream. Save-pattern crawling reduces cost by discovering links cheaply and applying expensive extraction only to matching pages. Scraper API commands cover Google, Fast Search, Amazon product/pricing/search, Walmart product/search, YouTube search/metadata, ChatGPT and Gemini; all support batch input and shared retry/output flags, with smart-extract batch extraction currently limited to scrape. The command-to-API mapping is in the scraper command table below. Shell-executing features --post-process, --on-complete, schedule and REPL !command are gated because scraped content could contain prompt-injection attempts. Enabling requires setting the SCRAPINGBEE_ALLOW_EXEC environment variable, unsafe verification, and optionally the allowed-commands restriction via SCRAPINGBEE_ALLOWED_COMMANDS. Check status and audit with the unsafe command, and revoke with logout or unsafe --disable. Utility commands include tutorial, unsafe, docs, version, usage, auth, logout and help.

The documented shell-execution gate is opt-in; enable it only when you intend to use shell-running features.

export SCRAPINGBEE_ALLOW_EXEC=1

Restrict permitted commands to the tools and script paths you actually intend to allow.

export SCRAPINGBEE_ALLOWED_COMMANDS="jq,head,python3 /path/to/transform.py"

Smart Extract path language reference

The path language is built from a small set of composable forms. Navigate into a structure with .key, or use (key-name) for a literal key that contains dots or spaces. Select items with [0] or [-1] for a single index, [0, 3, 7] to cherry-pick several, or [0:5] for a contiguous slice. Inspect a dict with [keys] or [values]. Search recursively with ...key, ...*glob* or ...(escaped) to find every occurrence at any depth. Expand context with ~N to return the ancestor N levels above a match, then chain further navigation, key listing, filtering or recursive search. Filter values with [=text], [=*glob*], [=/regex/] or [!=pattern]; filter dicts by a property with [key=pattern], [key!=pattern] or [*=pattern]. Combine searches with path1 | path2 for OR or path1 & path2 for AND, with spaces around the operator; the two operators cannot be mixed in one expression. Auto-format detection picks JSON when the response starts with { or [, HTML when it starts with <, XML on <?xml or <rss>, CSV on comma-separated headers, NDJSON on multiple {...} lines, Markdown on # or === headings, and plain text as the fallback.

Presets

Presets apply a predefined set of options and only set flags you have not already set, so any preset value can be overridden. The screenshot preset captures a viewport screenshot and enables --screenshot and --render-js. screenshot-and-html returns a full-page screenshot plus HTML in a single JSON response. fetch performs a fast fetch without JavaScript rendering (--render-js false). extract-links extracts all <a href> links from the page as JSON. extract-emails extracts all mailto: links, and extract-phones extracts all tel: links. scroll-page performs infinite scroll with JS rendering to load lazy content.

Scraper API commands

Each scraper command wraps a specialized API. The google "query" command maps to the Google Search API; fast-search "query" to the Fast Search API; amazon-product ASIN to the Amazon Product API; amazon-pricing ASIN to the Amazon Pricing API; amazon-search "query" to the Amazon Search API; walmart-product ID to the Walmart Product API; walmart-search "query" to the Walmart Search API; youtube-search "query" and youtube-metadata VIDEO_ID to the YouTube API; chatgpt "prompt" to the ChatGPT API; and gemini "prompt" to the Gemini API. Full parameter documentation lives on each API's page — select the CLI tab for command-line examples.

Scrape rendering flags
FlagAPI ParameterDescription
--render-jsrender_jsEnable/disable JavaScript rendering
--js-scenariojs_scenarioJavaScript scenario to execute
--waitwaitWait time in ms before returning (0-35,000)
--wait-forwait_forCSS/XPath selector to wait for
--wait-browserwait_browserBrowser event to wait for: domcontentloaded, load, networkidle0, or networkidle2
--block-adsblock_adsBlock ads on the page
--block-resourcesblock_resourcesBlock images and CSS
--window-widthwindow_widthViewport width in pixels
--window-heightwindow_heightViewport height in pixels

Wait is 0-35,000 ms; wait_browser accepts domcontentloaded, load, networkidle0 or networkidle2.

Scrape proxy and header flags
FlagAPI ParameterDescription
--premium-proxypremium_proxyUse premium/residential proxies (25 credits with JS)
--stealth-proxystealth_proxyUse stealth proxies for hard-to-scrape sites (75 credits)
--country-codecountry_codeProxy country code (ISO 3166-1)
--custom-googlecustom_googleScrape Google domains (true/false). 15 credits per request.
--own-proxyown_proxyUse your own proxy (user:pass@host:port)
-H / --headerCustom headersAdd custom headers (repeatable: -H "Key:Value")
--forward-headersforward_headersForward custom headers to target
--forward-headers-pureforward_headers_pureForward only custom headers

Credit costs: premium_proxy 25 credits with JS, stealth_proxy 75 credits, custom_google 15 credits per request.

Scrape output, screenshot and extraction flags
FlagAPI ParameterDescription
--json-responsejson_responseWrap response in JSON
--return-page-sourcereturn_page_sourceReturn original HTML before JS rendering
--return-page-markdownreturn_page_markdownReturn content as markdown
--return-page-textreturn_page_textReturn content as plain text
--screenshotscreenshotCapture a screenshot
--screenshot-selectorscreenshot_selectorCSS selector for screenshot area
--screenshot-full-pagescreenshot_full_pageCapture full-page screenshot
--extract-rulesextract_rulesCSS/XPath extraction rules as JSON
--ai-queryai_queryNatural language extraction (+5 credits)
--ai-selectorai_selectorCSS selector to focus AI extraction
--ai-extract-rulesai_extract_rulesAI extraction rules as JSON (+5 credits)

ai_query and ai_extract_rules add 5 credits.

Scrape request and configuration flags
FlagAPI ParameterDescription
--session-idsession_idSession ID for sticky IP (0-10000000)
--timeouttimeoutTimeout in ms (1000-140000)
--cookiescookiesCustom cookies
--devicedeviceDevice type: desktop or mobile
--transparent-status-codetransparent_status_codeReturn target's status code and body as-is (true/false)
--tagtagOptional label included in the API response headers
-X / --methodHTTP methodGET, POST, or PUT
-d / --dataRequest bodyRequest body for POST/PUT
--scraping-configscraping_configApply a pre-saved scraping configuration by name

session_id is 0-10000000, timeout is 1000-140000 ms, and -X accepts GET, POST or PUT.

REPL slash commands
CommandDescription
:help / :?Show grouped commands and key bindings
:q / exit / quitExit the REPL
:clearClear the scrollback buffer
:set key=valueSet a session default (e.g. :set country-code=fr) applied to every subsequent command
:set --key valueAlternate :set syntax, useful when pasting shell-style flags
:unset keyRemove a session default (:unset * or :unset all clears all)
:reset / :unset-allClear every session default
:showDisplay current session settings
:listList active scheduled jobs
:viewPage the last command output. After crawl, bare :view auto-opens the crawl log; otherwise it opens the most recent response body. Auto-detects JSON/HTML and pretty-prints; r toggles raw; q / Esc exits.
:view <path>Page an arbitrary file
:view crawlPage the most recent crawl log

Meta-commands are prefixed with : so they do not collide with API commands.

REPL key bindings
KeyAction
TabSingle-match inline complete; multi-match opens popup for commands, flags, and choice values; ghost-text fallback otherwise
Shift+TabCycle the completion popup backwards. With --no-drag-copy (and no popup open), also toggles Scroll ↔ Select mouse mode.
EscDismiss the completion popup
RightAccept the next word of the ghost-text suggestion
EndAccept the entire ghost-text suggestion
Up / DownNavigate command history in single-line mode; move the cursor inside a multi-line paste buffer
PgUp / PgDnScroll the scrollback up / down
Ctrl+Up / Ctrl+DownScroll the scrollback one visual row at a time
Ctrl+Home / Ctrl+EndJump to the top / bottom of the scrollback
Ctrl+WDelete the word before the cursor (also Alt / Option +⌫)
Ctrl+C (running)Stop the current command. A second press within 2s force-kills subprocess-backed commands if they don't exit cleanly.
Ctrl+C (idle)Clear the typed line or multi-line paste buffer; on an already-empty line, exit the REPL
Ctrl+DExit the REPL when no command is running
Alt+SWith --no-drag-copy, toggle Scroll ↔ Select mouse mode
!commandRun a shell command (requires advanced features setup)
Multi-line pastePasting newline-containing text opens an editable buffer; Enter queues the lines and runs them sequentially, Ctrl+J / Alt+Enter adds a newline
ClickClick an underlined existing path to open it in Finder / Explorer / the default app

Key bindings apply inside the interactive REPL.

Crawl parameters
ParameterType / DefaultDescription
targetstring requiredOne or more URLs to start crawling from, or a Scrapy spider name (project mode)
--allow-external-domainsboolean (false)Follow links to any domain instead of same-domain only. Quick-crawl only
--allowed-domainsstring ("")Comma-separated list of domains to crawl (default is same domain as start URLs). Quick-crawl only
--autothrottleboolean (false)Enable Scrapy AutoThrottle to automatically adapt the request rate based on server load
--concurrencyinteger (0)Maximum concurrent requests (0 = auto-detect from your plan's concurrency limit)
--confirm"yes" | "y" | "true" ("")Auto-confirm crawl prompts. Pass yes to skip interactive confirmation in scripts
--download-delayfloat ("")Delay in seconds between requests (Scrapy DOWNLOAD_DELAY)
--exclude-patternstring ("")Regex: skip URLs matching this pattern
--from-sitemapstring ("")Fetch URLs from a sitemap.xml and crawl them (URL or path to sitemap)
--include-patternstring ("")Regex: only follow URLs matching this pattern
--max-depthinteger (0)Maximum link depth when following same-domain links (0 = unlimited). Quick-crawl only
--max-pagesinteger (0)Maximum pages saved to disk (0 = unlimited). Discovery-only fetches for link-following may cost additional credits
--on-completestring ("")Shell command to run after the crawl completes
--output-dirpath ("crawl_<timestamp>")Crawl output folder. Defaults to crawl_<timestamp>
--projectpath ("")Path to a Scrapy project directory. Spider mode only
--resumeboolean (false)Skip already-crawled URLs from a previous run (reads manifest.json in output dir)
--save-patternstring ("")Regex: only save pages whose URL matches this pattern. Other pages are visited for link discovery but not saved

Types and defaults as documented for the crawl command.

Batch parameters
ParameterType / DefaultDescription
--input-filestring requiredPath to input file with one item per line (URL, query, ASIN, etc.). Use - for stdin. Supports .txt and .csv
--backofffloat (2)Retry backoff multiplier (exponential)
--concurrencyinteger (0)Maximum concurrent requests (0 = auto-detect from your plan's concurrency limit)
--deduplicateboolean (false)Normalize URLs and remove duplicates from the input before processing
--extract-fieldstring ("")Extract values from JSON responses using dot-path notation (e.g. organic_results.url)
--fieldsstring ("")Comma-separated top-level JSON keys to include in output
--input-columnstring ("")For CSV input, the column name or 0-based index containing the target values
--no-progressboolean (false)Suppress the progress bar during batch processing
--on-completestring ("")Shell command to run after the batch completes
--output-dirpath ("batch_<timestamp>")Folder for batch output files. Defaults to batch_<timestamp>
--output-filepath ("")Write output to a specific file instead of stdout
--output-format"csv" | "ndjson" ("")Batch output format: csv (single CSV) or ndjson (newline-delimited JSON). Omit this flag for individual files in --output-dir
--overwriteboolean (false)Overwrite an existing --output-file without prompting
--post-processstring ("")Pipe each result through a shell command (e.g. 'jq .title')
--resumeboolean (false)Skip items already saved in --output-dir from a previous run
--retriesinteger (3)Number of retry attempts on transient errors
--sampleinteger (0)Process only N random items from input for testing (0 = all)
--update-csvboolean (false)Fetch fresh data and update the input CSV file in-place with new result columns
--verboseboolean (false)Show response headers and status code for each request

Types and defaults as documented for batch mode.

Export parameters
ParameterType / DefaultDescription
--input-dirpath requiredBatch or crawl output directory to read from
--columnsstring ("")CSV mode: comma-separated column names to include. Rows missing all selected columns are dropped
--deduplicateboolean (false)CSV mode: remove duplicate rows
--flattenboolean (false)CSV mode: recursively flatten nested dicts to dot-notation columns (e.g. buybox.price)
--flatten-depthinteger (5)CSV mode: maximum nesting depth for --flatten. Defaults to 5; increase for deeply nested data
--format"ndjson" | "txt" | "csv" ("ndjson")Output format: ndjson (one JSON object per line), txt (plain text blocks), or csv (flat table from JSON arrays)
--output-filepath ("")Write output to file instead of stdout
--overwriteboolean (false)Overwrite an existing --output-file without prompting

Types and defaults as documented for the export command.

Scrape flag groups at a glance
AreaKey flagsUse it when
Rendering--render-js, --js-scenario, --wait, --wait-for, --wait-browserPage needs JavaScript, a scenario, or a timed/selector wait
Proxy--premium-proxy, --stealth-proxy, --country-code, --own-proxyTarget needs residential/stealth routing, a country, or your own proxy
Output--json-response, --return-page-markdown, --return-page-text, --screenshotYou need JSON wrapping, markdown/text, or an image
Extraction--extract-rules, --ai-query, --ai-extract-rules, --smart-extractYou need selector rules, AI extraction, or client-side path extraction

Summary of documented scrape flag groups; exact pairs in scrape tables above.

Crawl modes compared
ModeStarting pointOutput
Quick crawlOne or more start URLsNumbered files plus manifest.json in output directory
Sitemap crawl--from-sitemap URLNumbered files plus manifest.json in output directory
Project spiderSpider name plus --project pathSpider-defined output with injected middleware and API key

Quick-crawl and sitemap modes accept scrape rendering/proxy/extraction flags; project spider mode rejects CLI API flags.

Batch output formats compared
Batch outputFlagNotes
Individual filesOmit --output-formatNumbered files in --output-dir with manifest.json
Single CSV--output-format csvMerged CSV to stdout or --output-file
NDJSON stream--output-format ndjsonOne JSON object per line, suitable for piping

Smart-extract batch application is limited to scrape individual-file and CSV output, not NDJSON.

Frequently asked questions

How do I provide an API key to the CLI?

Use the interactive prompt for first-time setup, the non-interactive flag for CI/CD and scripts, or the environment variable when no file should be stored. The CLI also reads .env files in the current directory.

Why do JSON and path expressions need single quotes?

Wrap JSON data and path expressions in single quotes. Double quotes allow shell expansion of $, backslash and backticks, which can break JSON. URLs may use double quotes because they can contain & or ?.

What are the three crawl modes?

Quick crawl follows same-domain links from start URLs, sitemap crawl starts from discovered sitemap URLs, and project spider mode runs a Scrapy spider with middleware and API key injected. In project mode, API parameters belong in the request object, not CLI API flags.

What does --smart-extract do?

It replaces deprecated --extract-field and --fields. It auto-detects JSON, HTML, XML, CSV, NDJSON, Markdown or plain text and supports navigation, selection, recursive search, filters, context expansion and combined searches. Batch support is limited as described in the Smart Extract section.

Why are shell-executing features gated?

Shell-executing features are disabled by default because scraped content could contain prompt-injection attempts. Enabling requires the environment variable, unsafe verification, and optionally an allowed-commands restriction; status and audit can be checked with the unsafe command.

curl -LsSf https://astral.sh/uv/install.sh | sh
uv tool install scrapingbee-cli
{
  "max_api_credit": 1000000,
  "used_api_credit": 42150,
  "max_concurrency": 100,
  "current_concurrency": 0,
  "renewal_subscription_date": "2025-07-26T04:57:13.580067"
}