Command-line interface for web scraping, crawling and extraction
Install the CLI, authenticate once, then choose a single scrape, a crawl, a batch run, or an interactive session. This guide maps the main commands to their inputs, outputs and failure modes so you can pick the smallest workflow that fits the job.
Install, authenticate and check credits
Recommended installation uses uv without a virtual environment; the alternative is pip inside a virtual environment. Verify with the version command. Authentication can be interactive for first-time setup, non-interactive for CI/CD and scripts, or via environment variable with no stored file. The CLI also reads .env files in the current directory. Use the show-key command to see the stored location and logout to remove the stored key. Checklist: install, verify version, authenticate, run usage to confirm credits and concurrency before a large job.
Shell quoting: the most common avoidable failure
Always use single quotes around JSON data and path expressions. Double quotes trigger shell expansion: $ expands variables, backslash creates escape sequences, and backticks trigger command substitution. This matters for --ai-extract-rules, --extract-rules, --js-scenario, --smart-extract and --ai-query. URLs should use double quotes when they may contain & or ?. Decision rule: URL in double quotes, JSON or path expression in single quotes, side by side when both appear.
Interactive mode for iteration
Running the bare command opens an interactive REPL with a pinned banner, live credit toolbar, scrollback, tab completion and ghost-text suggestions. It is suited to exploring a site, tuning proxies, testing extraction rules, comparing search queries or building a batch workflow before scripting the final command. Session defaults set with :set apply to later commands and inline flags override them; :show displays active defaults and :unset removes them. Startup flags must precede the bare invocation. Inside the REPL, :view pages the last result with pretty JSON/HTML detection and a raw/pretty toggle, :list shows active schedules, and Ctrl+C stops a running command mid-request. If no key is configured, the prompt becomes a masked API-key input; auth and logout re-prompt in place. For existing output files, the shell prompts, while --overwrite skips the check in the REPL and in CI/CD. Slash commands and key bindings are enumerated in the REPL tables below.
Inside interactive mode, set defaults and override an individual request when needed.
:set country-code=fr
:set premium-proxy=true
scrape "https://example.com" # runs with --country-code fr --premium-proxy true
scrape "https://example.com" --country-code us # explicit flag wins
:unset premium-proxyScrape one page, with presets and saved configurations
The scrape command fetches a page through the HTML API. CLI flags map one-to-one to API parameters, with underscores becoming hyphens; predefined values accept hyphens or underscores, and boolean flags accept explicit true/false or a bare flag, with omission leaving the API default. Basic choices are printing HTML to stdout, saving to a file with auto-detected extension, enabling JavaScript rendering and premium proxy, extracting with AI, or returning markdown for LLM pipelines. Saved scraping configurations provide a reusable base that inline options override, and configuration names are case-sensitive and limited to alphanumeric characters, hyphens and underscores. Presets only set flags not already set, so explicit values win. CLI-only flags include escalate-proxy for retrying 403/429 responses with premium then stealth proxy, chunk-size and chunk-overlap for NDJSON text/markdown chunks, and force-extension when --output-file has no extension. Exact flag/API-parameter pairs, ranges and credit costs are in the scrape tables below.
Inside interactive mode, save the response to a file and explicitly allow an existing output file to be replaced.
scrape "https://example.com" --output-file result.html --overwriteCrawl: choose quick, sitemap or project spider
Quick crawl starts from one or more URLs and follows same-domain links, saving numbered files plus manifest.json. Sitemap crawl parses a sitemap, including indexes recursively to depth 2, then crawls discovered URLs. Project spider mode runs a Scrapy spider with middleware and API key injected; there, parameters belong in the request object and CLI API flags are rejected, while infrastructure flags such as concurrency, download-delay and autothrottle still apply. Use include/exclude patterns to restrict followed URLs, save-pattern to visit pages for discovery but save only matching URLs, allowed-domains or allow-external-domains to widen scope, max-depth and max-pages to bound the job, and resume to skip URLs already in manifest.json. Non-HTML or extraction-focused crawls may run a discovery phase with cheap HTML requests before save requests, so credits can exceed the number of saved files; --confirm yes skips the prompt in scripts. Concurrency defaults to auto-detection from the plan limit and is capped by max-pages. Exact crawl parameters with types and defaults are in the crawl parameters table below.
Batch processing: test small, resume safely, enrich CSVs
Batch mode is enabled with --input-file on scrape, google and other scraper commands. Input supports .txt with one item per line and .csv with --input-column naming a column or 0-based index; - reads stdin. Default output is numbered files in batch_<timestamp> with manifest.json. Alternatives are a single CSV or NDJSON stream via --output-format. Before spending credits, --deduplicate normalizes URLs and removes duplicates, and --sample tests N random items. --update-csv enriches the original CSV in place, flattening nested JSON to dot-notation columns. --resume with the same output directory skips completed items, and bare --resume at the top level finds incomplete batches and prints the resume command. Retries default to 3 with exponential backoff multiplier 2. --post-process and --on-complete execute shell commands and require advanced-features setup; post-process does not apply to --update-csv. Deprecated --extract-field and --fields should be replaced by --smart-extract. Exact batch parameters with types and defaults are in the batch parameters table below.
Smart Extract: one path language across formats
--smart-extract performs client-side extraction with a path language and auto-detects JSON, HTML, XML, CSV, NDJSON, Markdown or plain text. It works on single-item runs for scrape, google, fast-search, Amazon, Walmart, YouTube, ChatGPT and Gemini commands. In scrape batches it applies to individual-file and CSV output, not NDJSON; on non-scrape commands batch support is not yet available, so use the deprecated extract-field or jq post-processing there. The language navigates keys, selects indexes or slices, lists keys/values, searches recursively with ...key or glob patterns, expands context with ~N, filters by substring, glob, regex or key property, and combines searches with | for OR or & for AND. Spaces around | and & are required, and the two operators cannot be mixed in one expression. Output can be raw values one per line, block syntax with separate {} pairs per field, or JSON schema mode matching --ai-extract-rules. A practical pattern is to find a price with a regex, expand two levels up to capture the surrounding product card, and emit named fields for downstream use. The path-language syntax, selection, dict, recursive-search, context, filter and operator forms are enumerated in the Smart Extract path tables below, and auto-format detection rules are in the format-detection table.
Export merged results and schedule recurring runs
Export merges numbered batch or crawl files into one file, using manifest.json to annotate records with source URLs. NDJSON is the default: valid JSON passes through with an added _url field and non-JSON is wrapped as content plus _url. TXT separates files with blank lines and # URL prefixes when a manifest exists. CSV expands JSON arrays into rows; --flatten recursively creates dot-notation columns, --flatten-depth raises the default depth of 5 for deeply nested data, --columns keeps only named columns and drops rows missing all of them, and --deduplicate removes identical rows. Schedule creates cron jobs on macOS/Linux to run any CLI command at an interval such as 5m, 1h or 2d, with a minimum of 1 minute; Windows returns a clear cron-unavailable message. Each schedule creates a tagged cron entry, logs to the CLI config logs directory, and tracks metadata in schedules.json. Use --list to inspect, --stop by name or all to remove, and --name to make a job identifiable. Scheduling requires advanced-features setup. Exact export parameters with types and defaults are in the export parameters table below.
Pipelines, scraper commands and gated shell features
Commands compose through stdin/stdout: read URLs from a pipe with --input-file -, stream NDJSON, filter with jq, or chain a search command into scrape. For LLM and RAG ingestion, crawl a documentation site to markdown or use chunk-size with overlap to emit NDJSON chunks with URL, index, total and timestamp metadata. For enrichment, start from a CSV and add live fields with --update-csv; for larger ETL work, scrape in parallel, export with flattening to CSV, and reshape downstream. Save-pattern crawling reduces cost by discovering links cheaply and applying expensive extraction only to matching pages. Scraper API commands cover Google, Fast Search, Amazon product/pricing/search, Walmart product/search, YouTube search/metadata, ChatGPT and Gemini; all support batch input and shared retry/output flags, with smart-extract batch extraction currently limited to scrape. The command-to-API mapping is in the scraper command table below. Shell-executing features --post-process, --on-complete, schedule and REPL !command are gated because scraped content could contain prompt-injection attempts. Enabling requires setting the SCRAPINGBEE_ALLOW_EXEC environment variable, unsafe verification, and optionally the allowed-commands restriction via SCRAPINGBEE_ALLOWED_COMMANDS. Check status and audit with the unsafe command, and revoke with logout or unsafe --disable. Utility commands include tutorial, unsafe, docs, version, usage, auth, logout and help.
The documented shell-execution gate is opt-in; enable it only when you intend to use shell-running features.
export SCRAPINGBEE_ALLOW_EXEC=1Restrict permitted commands to the tools and script paths you actually intend to allow.
export SCRAPINGBEE_ALLOWED_COMMANDS="jq,head,python3 /path/to/transform.py"Smart Extract path language reference
The path language is built from a small set of composable forms. Navigate into a structure with .key, or use (key-name) for a literal key that contains dots or spaces. Select items with [0] or [-1] for a single index, [0, 3, 7] to cherry-pick several, or [0:5] for a contiguous slice. Inspect a dict with [keys] or [values]. Search recursively with ...key, ...*glob* or ...(escaped) to find every occurrence at any depth. Expand context with ~N to return the ancestor N levels above a match, then chain further navigation, key listing, filtering or recursive search. Filter values with [=text], [=*glob*], [=/regex/] or [!=pattern]; filter dicts by a property with [key=pattern], [key!=pattern] or [*=pattern]. Combine searches with path1 | path2 for OR or path1 & path2 for AND, with spaces around the operator; the two operators cannot be mixed in one expression. Auto-format detection picks JSON when the response starts with { or [, HTML when it starts with <, XML on <?xml or <rss>, CSV on comma-separated headers, NDJSON on multiple {...} lines, Markdown on # or === headings, and plain text as the fallback.
Presets
Presets apply a predefined set of options and only set flags you have not already set, so any preset value can be overridden. The screenshot preset captures a viewport screenshot and enables --screenshot and --render-js. screenshot-and-html returns a full-page screenshot plus HTML in a single JSON response. fetch performs a fast fetch without JavaScript rendering (--render-js false). extract-links extracts all <a href> links from the page as JSON. extract-emails extracts all mailto: links, and extract-phones extracts all tel: links. scroll-page performs infinite scroll with JS rendering to load lazy content.
Scraper API commands
Each scraper command wraps a specialized API. The google "query" command maps to the Google Search API; fast-search "query" to the Fast Search API; amazon-product ASIN to the Amazon Product API; amazon-pricing ASIN to the Amazon Pricing API; amazon-search "query" to the Amazon Search API; walmart-product ID to the Walmart Product API; walmart-search "query" to the Walmart Search API; youtube-search "query" and youtube-metadata VIDEO_ID to the YouTube API; chatgpt "prompt" to the ChatGPT API; and gemini "prompt" to the Gemini API. Full parameter documentation lives on each API's page — select the CLI tab for command-line examples.
| Flag | API Parameter | Description |
|---|---|---|
| --render-js | render_js | Enable/disable JavaScript rendering |
| --js-scenario | js_scenario | JavaScript scenario to execute |
| --wait | wait | Wait time in ms before returning (0-35,000) |
| --wait-for | wait_for | CSS/XPath selector to wait for |
| --wait-browser | wait_browser | Browser event to wait for: domcontentloaded, load, networkidle0, or networkidle2 |
| --block-ads | block_ads | Block ads on the page |
| --block-resources | block_resources | Block images and CSS |
| --window-width | window_width | Viewport width in pixels |
| --window-height | window_height | Viewport height in pixels |
Wait is 0-35,000 ms; wait_browser accepts domcontentloaded, load, networkidle0 or networkidle2.
| Flag | API Parameter | Description |
|---|---|---|
| --premium-proxy | premium_proxy | Use premium/residential proxies (25 credits with JS) |
| --stealth-proxy | stealth_proxy | Use stealth proxies for hard-to-scrape sites (75 credits) |
| --country-code | country_code | Proxy country code (ISO 3166-1) |
| --custom-google | custom_google | Scrape Google domains (true/false). 15 credits per request. |
| --own-proxy | own_proxy | Use your own proxy (user:pass@host:port) |
| -H / --header | Custom headers | Add custom headers (repeatable: -H "Key:Value") |
| --forward-headers | forward_headers | Forward custom headers to target |
| --forward-headers-pure | forward_headers_pure | Forward only custom headers |
Credit costs: premium_proxy 25 credits with JS, stealth_proxy 75 credits, custom_google 15 credits per request.
| Flag | API Parameter | Description |
|---|---|---|
| --json-response | json_response | Wrap response in JSON |
| --return-page-source | return_page_source | Return original HTML before JS rendering |
| --return-page-markdown | return_page_markdown | Return content as markdown |
| --return-page-text | return_page_text | Return content as plain text |
| --screenshot | screenshot | Capture a screenshot |
| --screenshot-selector | screenshot_selector | CSS selector for screenshot area |
| --screenshot-full-page | screenshot_full_page | Capture full-page screenshot |
| --extract-rules | extract_rules | CSS/XPath extraction rules as JSON |
| --ai-query | ai_query | Natural language extraction (+5 credits) |
| --ai-selector | ai_selector | CSS selector to focus AI extraction |
| --ai-extract-rules | ai_extract_rules | AI extraction rules as JSON (+5 credits) |
ai_query and ai_extract_rules add 5 credits.
| Flag | API Parameter | Description |
|---|---|---|
| --session-id | session_id | Session ID for sticky IP (0-10000000) |
| --timeout | timeout | Timeout in ms (1000-140000) |
| --cookies | cookies | Custom cookies |
| --device | device | Device type: desktop or mobile |
| --transparent-status-code | transparent_status_code | Return target's status code and body as-is (true/false) |
| --tag | tag | Optional label included in the API response headers |
| -X / --method | HTTP method | GET, POST, or PUT |
| -d / --data | Request body | Request body for POST/PUT |
| --scraping-config | scraping_config | Apply a pre-saved scraping configuration by name |
session_id is 0-10000000, timeout is 1000-140000 ms, and -X accepts GET, POST or PUT.
| Command | Description |
|---|---|
| :help / :? | Show grouped commands and key bindings |
| :q / exit / quit | Exit the REPL |
| :clear | Clear the scrollback buffer |
| :set key=value | Set a session default (e.g. :set country-code=fr) applied to every subsequent command |
| :set --key value | Alternate :set syntax, useful when pasting shell-style flags |
| :unset key | Remove a session default (:unset * or :unset all clears all) |
| :reset / :unset-all | Clear every session default |
| :show | Display current session settings |
| :list | List active scheduled jobs |
| :view | Page the last command output. After crawl, bare :view auto-opens the crawl log; otherwise it opens the most recent response body. Auto-detects JSON/HTML and pretty-prints; r toggles raw; q / Esc exits. |
| :view <path> | Page an arbitrary file |
| :view crawl | Page the most recent crawl log |
Meta-commands are prefixed with : so they do not collide with API commands.
| Key | Action |
|---|---|
| Tab | Single-match inline complete; multi-match opens popup for commands, flags, and choice values; ghost-text fallback otherwise |
| Shift+Tab | Cycle the completion popup backwards. With --no-drag-copy (and no popup open), also toggles Scroll ↔ Select mouse mode. |
| Esc | Dismiss the completion popup |
| Right | Accept the next word of the ghost-text suggestion |
| End | Accept the entire ghost-text suggestion |
| Up / Down | Navigate command history in single-line mode; move the cursor inside a multi-line paste buffer |
| PgUp / PgDn | Scroll the scrollback up / down |
| Ctrl+Up / Ctrl+Down | Scroll the scrollback one visual row at a time |
| Ctrl+Home / Ctrl+End | Jump to the top / bottom of the scrollback |
| Ctrl+W | Delete the word before the cursor (also Alt / Option +⌫) |
| Ctrl+C (running) | Stop the current command. A second press within 2s force-kills subprocess-backed commands if they don't exit cleanly. |
| Ctrl+C (idle) | Clear the typed line or multi-line paste buffer; on an already-empty line, exit the REPL |
| Ctrl+D | Exit the REPL when no command is running |
| Alt+S | With --no-drag-copy, toggle Scroll ↔ Select mouse mode |
| !command | Run a shell command (requires advanced features setup) |
| Multi-line paste | Pasting newline-containing text opens an editable buffer; Enter queues the lines and runs them sequentially, Ctrl+J / Alt+Enter adds a newline |
| Click | Click an underlined existing path to open it in Finder / Explorer / the default app |
Key bindings apply inside the interactive REPL.
| Parameter | Type / Default | Description |
|---|---|---|
| target | string required | One or more URLs to start crawling from, or a Scrapy spider name (project mode) |
| --allow-external-domains | boolean (false) | Follow links to any domain instead of same-domain only. Quick-crawl only |
| --allowed-domains | string ("") | Comma-separated list of domains to crawl (default is same domain as start URLs). Quick-crawl only |
| --autothrottle | boolean (false) | Enable Scrapy AutoThrottle to automatically adapt the request rate based on server load |
| --concurrency | integer (0) | Maximum concurrent requests (0 = auto-detect from your plan's concurrency limit) |
| --confirm | "yes" | "y" | "true" ("") | Auto-confirm crawl prompts. Pass yes to skip interactive confirmation in scripts |
| --download-delay | float ("") | Delay in seconds between requests (Scrapy DOWNLOAD_DELAY) |
| --exclude-pattern | string ("") | Regex: skip URLs matching this pattern |
| --from-sitemap | string ("") | Fetch URLs from a sitemap.xml and crawl them (URL or path to sitemap) |
| --include-pattern | string ("") | Regex: only follow URLs matching this pattern |
| --max-depth | integer (0) | Maximum link depth when following same-domain links (0 = unlimited). Quick-crawl only |
| --max-pages | integer (0) | Maximum pages saved to disk (0 = unlimited). Discovery-only fetches for link-following may cost additional credits |
| --on-complete | string ("") | Shell command to run after the crawl completes |
| --output-dir | path ("crawl_<timestamp>") | Crawl output folder. Defaults to crawl_<timestamp> |
| --project | path ("") | Path to a Scrapy project directory. Spider mode only |
| --resume | boolean (false) | Skip already-crawled URLs from a previous run (reads manifest.json in output dir) |
| --save-pattern | string ("") | Regex: only save pages whose URL matches this pattern. Other pages are visited for link discovery but not saved |
Types and defaults as documented for the crawl command.
| Parameter | Type / Default | Description |
|---|---|---|
| --input-file | string required | Path to input file with one item per line (URL, query, ASIN, etc.). Use - for stdin. Supports .txt and .csv |
| --backoff | float (2) | Retry backoff multiplier (exponential) |
| --concurrency | integer (0) | Maximum concurrent requests (0 = auto-detect from your plan's concurrency limit) |
| --deduplicate | boolean (false) | Normalize URLs and remove duplicates from the input before processing |
| --extract-field | string ("") | Extract values from JSON responses using dot-path notation (e.g. organic_results.url) |
| --fields | string ("") | Comma-separated top-level JSON keys to include in output |
| --input-column | string ("") | For CSV input, the column name or 0-based index containing the target values |
| --no-progress | boolean (false) | Suppress the progress bar during batch processing |
| --on-complete | string ("") | Shell command to run after the batch completes |
| --output-dir | path ("batch_<timestamp>") | Folder for batch output files. Defaults to batch_<timestamp> |
| --output-file | path ("") | Write output to a specific file instead of stdout |
| --output-format | "csv" | "ndjson" ("") | Batch output format: csv (single CSV) or ndjson (newline-delimited JSON). Omit this flag for individual files in --output-dir |
| --overwrite | boolean (false) | Overwrite an existing --output-file without prompting |
| --post-process | string ("") | Pipe each result through a shell command (e.g. 'jq .title') |
| --resume | boolean (false) | Skip items already saved in --output-dir from a previous run |
| --retries | integer (3) | Number of retry attempts on transient errors |
| --sample | integer (0) | Process only N random items from input for testing (0 = all) |
| --update-csv | boolean (false) | Fetch fresh data and update the input CSV file in-place with new result columns |
| --verbose | boolean (false) | Show response headers and status code for each request |
Types and defaults as documented for batch mode.
| Parameter | Type / Default | Description |
|---|---|---|
| --input-dir | path required | Batch or crawl output directory to read from |
| --columns | string ("") | CSV mode: comma-separated column names to include. Rows missing all selected columns are dropped |
| --deduplicate | boolean (false) | CSV mode: remove duplicate rows |
| --flatten | boolean (false) | CSV mode: recursively flatten nested dicts to dot-notation columns (e.g. buybox.price) |
| --flatten-depth | integer (5) | CSV mode: maximum nesting depth for --flatten. Defaults to 5; increase for deeply nested data |
| --format | "ndjson" | "txt" | "csv" ("ndjson") | Output format: ndjson (one JSON object per line), txt (plain text blocks), or csv (flat table from JSON arrays) |
| --output-file | path ("") | Write output to file instead of stdout |
| --overwrite | boolean (false) | Overwrite an existing --output-file without prompting |
Types and defaults as documented for the export command.
| Area | Key flags | Use it when |
|---|---|---|
| Rendering | --render-js, --js-scenario, --wait, --wait-for, --wait-browser | Page needs JavaScript, a scenario, or a timed/selector wait |
| Proxy | --premium-proxy, --stealth-proxy, --country-code, --own-proxy | Target needs residential/stealth routing, a country, or your own proxy |
| Output | --json-response, --return-page-markdown, --return-page-text, --screenshot | You need JSON wrapping, markdown/text, or an image |
| Extraction | --extract-rules, --ai-query, --ai-extract-rules, --smart-extract | You need selector rules, AI extraction, or client-side path extraction |
Summary of documented scrape flag groups; exact pairs in scrape tables above.
| Mode | Starting point | Output |
|---|---|---|
| Quick crawl | One or more start URLs | Numbered files plus manifest.json in output directory |
| Sitemap crawl | --from-sitemap URL | Numbered files plus manifest.json in output directory |
| Project spider | Spider name plus --project path | Spider-defined output with injected middleware and API key |
Quick-crawl and sitemap modes accept scrape rendering/proxy/extraction flags; project spider mode rejects CLI API flags.
| Batch output | Flag | Notes |
|---|---|---|
| Individual files | Omit --output-format | Numbered files in --output-dir with manifest.json |
| Single CSV | --output-format csv | Merged CSV to stdout or --output-file |
| NDJSON stream | --output-format ndjson | One JSON object per line, suitable for piping |
Smart-extract batch application is limited to scrape individual-file and CSV output, not NDJSON.
Frequently asked questions
How do I provide an API key to the CLI?
Use the interactive prompt for first-time setup, the non-interactive flag for CI/CD and scripts, or the environment variable when no file should be stored. The CLI also reads .env files in the current directory.
Why do JSON and path expressions need single quotes?
Wrap JSON data and path expressions in single quotes. Double quotes allow shell expansion of $, backslash and backticks, which can break JSON. URLs may use double quotes because they can contain & or ?.
What are the three crawl modes?
Quick crawl follows same-domain links from start URLs, sitemap crawl starts from discovered sitemap URLs, and project spider mode runs a Scrapy spider with middleware and API key injected. In project mode, API parameters belong in the request object, not CLI API flags.
What does --smart-extract do?
It replaces deprecated --extract-field and --fields. It auto-detects JSON, HTML, XML, CSV, NDJSON, Markdown or plain text and supports navigation, selection, recursive search, filters, context expansion and combined searches. Batch support is limited as described in the Smart Extract section.
Why are shell-executing features gated?
Shell-executing features are disabled by default because scraped content could contain prompt-injection attempts. Enabling requires the environment variable, unsafe verification, and optionally an allowed-commands restriction; status and audit can be checked with the unsafe command.
curl -LsSf https://astral.sh/uv/install.sh | sh
uv tool install scrapingbee-cli
{
"max_api_credit": 1000000,
"used_api_credit": 42150,
"max_concurrency": 100,
"current_concurrency": 0,
"renewal_subscription_date": "2025-07-26T04:57:13.580067"
}