Skip to content

Download Files with cURL: Commands, Flags and Practical Examples

cURL is a command-line client for transferring data over protocols including HTTP, HTTPS, FTP and SFTP. Its main value for downloads is not just fetching a URL, but controlling redirects, filenames, retries, resumption, authentication and output in scripts. This guide explains the core syntax, the flags that matter most, batch patterns, common failures and the point at which a browser-based scraping service becomes more appropriate. It also covers the errors that look like successes, the checks that catch them, and the cases where cURL alone is the wrong tool.

What cURL is

Client URL, usually shortened to cURL, is both a command-line tool and a library for moving data with internet protocols. It began in 1996 as a small C project by Daniel Stenberg for automating currency-rate retrieval. The name describes its basic operation: supply a URL and it requests the resource at that address. The same tool handles HTTP, HTTPS, FTP, SFTP, IMAP, SMTP and a dozen other protocols without you having to think about the underlying mechanics.

Why it suits downloads

Five characteristics explain its popularity for file transfers: it supports many protocols in one tool; it fits naturally into shell scripts, cron jobs and CI/CD pipelines; it has a small resource footprint; it includes resume, redirect, retry and bandwidth controls; and it is pre-installed on macOS, Windows 10 and later, and most mainstream Linux systems. The underlying libcurl library can also be used from applications through bindings such as pycurl, which gives Python code the same capabilities as the CLI with full programmatic control.

Choosing between cURL, Wget and Python Requests

Use cURL for specific files, API calls and lightweight automation without dependencies. Use Wget when the task is to mirror a site or recursively collect a directory. Use Python Requests when the downloaded material must be parsed, sessions must be managed in code, or several requests form a larger processing pipeline. A useful rule is to ask whether the job is to interact with one endpoint, collect an entire structure, or process the response. In short: cURL gives you control, Wget gives you breadth, and Python Requests gives you processing power.

Uses beyond file downloads

Beyond saving files, cURL is useful for quick REST API tests, inspecting response headers with -I, submitting multipart forms, and tracing DNS, TCP, TLS and HTTP behaviour with -v. Treating it as a general HTTP client, rather than only a downloader, makes those diagnostic uses easier to remember. A useful habit is to reach for cURL first when the question is 'what does this endpoint actually return', and to reach for a scripting language only once the response needs parsing or chaining.

Setup and verification checklist

Install cURL through the system package manager where it is not already present; the supplied installation module is code0001. Confirm the installation with code0002 and check that HTTP, HTTPS, FTP and SFTP appear among the supported protocols. On macOS, a Homebrew installation may be newer than the system version kept by Apple for compatibility; confirm which binary is active with which curl. Optional helpers are Python 3 with pip, jq for JSON processing and GNU parallel for concurrent jobs, shown in code0003. Create a dedicated download directory with code0004, keep CA certificates current with code0005, and use --create-dirs when a script cannot guarantee that the output path already exists.

Core syntax and the production-ready baseline

Every invocation follows the pattern in code0006: options first, then the URL. Code0007 contrasts printing a response to the terminal, saving with the server filename, choosing a custom name, and a more resilient form combining redirects, resume and retries. The practical distinction is that omitting -O sends binary data to the terminal, omitting -L can save a redirect page instead of the target, and omitting --retry makes one network interruption end the transfer. Missing -L is the most common cause of a script that saves 512-byte HTML pages instead of the intended files, so include it unless there is a specific reason not to.

Filenames, paths, redirects, silence and progress

-O keeps the server-provided filename, while -o sets a chosen name. -L follows redirects. -C - resumes from an existing partial file. --retry repeats failed requests, and --retry-delay spaces those attempts. -s suppresses normal output; -sS is usually preferable in scripts because errors remain visible. -# displays an ASCII progress bar. Code0009 gives the simplest save operation, code0010 covers custom and date-based names, and code0011 covers target directories, automatic directory creation and Content-Disposition filenames. A date suffix such as $(date +%Y%m%d) in scheduled jobs prevents each run from overwriting the previous file and leaves an automatic audit trail.

HTTPS, debugging and resume patterns

cURL verifies HTTPS certificates by default. Certificate failures should normally be fixed by updating the CA bundle or supplying the correct bundle with --cacert, as shown in code0008. The -k option should be limited to local development servers with self-signed certificates. For redirects, -IL prints the header chain without downloading the body, which helps identify the final destination. For progress and auditing, --write-out can record downloaded size, speed, elapsed time and HTTP status. Code0015 demonstrates resume with retries, code0016 provides a reusable resume function, code0017 combines the main production flags, and code0018 and code0019 show an optional shell alias. In the resume function, errors are written to stderr (>&2) rather than stdout, so they do not pollute piped output or captured logs.

Authentication, headers, proxies and rate control

Basic authentication uses -u, bearer tokens use an Authorization header, and cookie sessions use -b and -c; these patterns appear in code0020, code0021 and code0022. Credentials should come from environment variables or a restricted-permissions credentials file, not from a script or shell history. Custom headers are covered in code0023, including content negotiation, API versioning and a browser-like User-Agent when a server rejects the default cURL agent. Proxy use is shown in code0024. The distinction between --socks5 and --socks5-hostname matters because the latter also sends DNS resolution through the proxy. Code0025 covers --limit-rate, pauses between requests and exponential backoff after HTTP 429 responses. --limit-rate accepts k for kilobytes and m for megabytes; adding it to public-API and research-repository downloads avoids IP-level blocks that are far more painful to recover from than a slower transfer.

Downloading many files

URL globbing handles predictable sequences, as in code0026; add --fail so missing files in the sequence surface as errors instead of silent gaps. A URL list can be processed with xargs or a shell loop in code0027; the loop gives better per-file naming, resume and error handling. Always quote URLs because characters such as & and ? have shell meanings, a point demonstrated in code0028. Parallel work can use GNU parallel or xargs -P in code0029, starting conservatively at three to eight concurrent connections for public servers. Code0030 adds logging, size checks, failed-file quarantine and concurrency for a production batch workflow.

Error diagnosis and integrity checks

cURL can return exit code 0 for an HTTP error unless --fail is used, so scripts should add --fail or --fail-with-body; code0031 shows the exit-code check. Common responses include 301 or 302 for redirects, 400 for a malformed request, 401 for missing authentication, 403 for forbidden access, 404 for a missing resource, 429 for rate limiting and 503 for an unavailable server. A successful status and non-zero size do not prove integrity, so code0032 verifies a published SHA-256 checksum. Timeouts are controlled in code0033 with --connect-timeout and --max-time, while -v exposes the connection stages. Code0034 separates certificate diagnosis from fixes, and code0035 covers directory, permission and disk-space checks. In corporate networks, SSL errors often come from a transparent proxy that re-signs HTTPS traffic with the company's own CA; the fix is to add that CA bundle with --cacert, not to disable verification.

When cURL is not enough

cURL does not discover links or crawl a site, execute JavaScript, or maintain large-scale scraping infrastructure such as proxy rotation, CAPTCHA handling and distributed retries. Those limits matter for JavaScript-rendered applications, whole-site crawling and very large collections. For whole-site crawling, a dedicated framework such as Scrapy is a common choice. For JavaScript-rendered pages, a headless browser or a scraping API that renders content is needed. At scale, proxy rotation, anti-bot handling, per-domain throttling and distributed retries become an engineering investment that breaks as target sites change. A managed browser or scraping service may be more suitable when maintenance of that infrastructure would exceed the value of the data task.

Integrating cURL with a managed scraping API

For straightforward file downloads, cURL is all you need. When a target adds JavaScript rendering, anti-bot protection or proxy rotation at scale, a managed scraping API can fill the gap. The integration details depend on the provider: the endpoint base is {{OWNER_API_BASE_URL}} and the free-credit allowance is {{OWNER_FREE_CREDITS}}, both to be supplied by the site owner. The workflow is to obtain an API key from the account dashboard, then call the endpoint with the key and target URL as encoded query parameters; the supplied request module is code0036. The --data-urlencode option lets cURL encode parameter values. A quick diagnostic is to request only the HTTP status from the target: a 200 may indicate that plain cURL is sufficient, while 403, 429 or an unexpectedly small 200 response suggests server-side protection.

Complete command reference

Download a file: curl -L -O https://example.com/file.zip. Save with a custom name: curl -L -o myfile.zip https://example.com/file.zip. Resume: curl -L -C - -O https://example.com/file.zip. Retry: curl -L --retry 3 --retry-delay 5 -O https://example.com/file.zip. Silent with errors visible: curl -sS -L -O https://example.com/file.zip. Progress bar: curl -L -# -O https://example.com/file.zip. Headers only: curl -IL https://example.com/file.zip. Basic Auth: curl -u "$USER:$PASS" -L -O https://example.com/file.zip. Bearer token: curl -H "Authorization: Bearer $TOKEN" -L -O https://api.example.com/data.csv. Limit speed: curl --limit-rate 500k -L -O https://example.com/file.zip. Globbing: curl -L -O https://example.com/file[1-10].zip. URL list: xargs -n 1 curl -L -O < urls.txt. Parallel: cat urls.txt | parallel -j 5 curl -L -O {}. Checksum: sha256sum -c file.zip.sha256. Debug: curl -v -L -O https://example.com/file.zip 2>&1 | head -80. Proxy: curl -x http://proxy:8080 -L -O https://example.com/file.zip. Create directories: curl -L --create-dirs -o ./downloads/sub/file.zip https://example.com/file.zip. Production-ready form: curl -L -C - --retry 3 --retry-delay 5 --fail -sS --create-dirs -o "./downloads/$(date +%Y%m%d)_file.zip" https://example.com/file.zip.

cURL download commands
TaskCommand
Download a filecurl -L -O https://example.com/file.zip
Save with a custom namecurl -L -o myfile.zip https://example.com/file.zip
Resume an interrupted downloadcurl -L -C - -O https://example.com/file.zip
Retry on failurecurl -L --retry 3 --retry-delay 5 -O https://example.com/file.zip
Silent downloadcurl -sS -L -O https://example.com/file.zip
Show progresscurl -L -# -O https://example.com/file.zip
Headers onlycurl -IL https://example.com/file.zip
Basic Authcurl -u "$USER:$PASS" -L -O https://example.com/file.zip
Bearer tokencurl -H "Authorization: Bearer $TOKEN" -L -O https://api.example.com/data.csv
Limit speedcurl --limit-rate 500k -L -O https://example.com/file.zip
Globbingcurl -L -O https://example.com/file[1-10].zip
URL listxargs -n 1 curl -L -O < urls.txt
Parallel downloadscat urls.txt | parallel -j 5 curl -L -O {}
Verify checksumsha256sum -c file.zip.sha256
Debugcurl -v -L -O https://example.com/file.zip 2>&1 | head -80
Proxycurl -x http://proxy:8080 -L -O https://example.com/file.zip
Create directoriescurl -L --create-dirs -o ./downloads/sub/file.zip https://example.com/file.zip
Production-readycurl -L -C - --retry 3 --retry-delay 5 --fail -sS --create-dirs -o "./downloads/$(date +%Y%m%d)_file.zip" https://example.com/file.zip

Commands use example.com placeholders and should be adapted to the actual URL and output path.

Frequently asked questions

Does curl work on Windows?

Yes. It has been included with Windows 10 and Windows 11 since 2018, so no separate installation is normally needed. Open Command Prompt or PowerShell and run curl --version to confirm. On older Windows versions, obtain the official binary from curl.se. After installation, the same download commands work across platforms.

How do I speed up curl downloads for large batches?

Run several transfers at once with GNU parallel, for example cat urls.txt | parallel -j 5 curl -L -O {}. Start with 5 concurrent jobs and adjust according to server tolerance. For one large file, --limit-rate can prevent throttling on congested connections.

How do I download multiple files with curl?

Use URL globbing for numbered sequences, such as curl -O https://example.com/file[1-100].zip. For an unordered list, pipe it through xargs: xargs -n 1 curl -L -O < urls.txt. For parallel batch work, use cat urls.txt | parallel -j 5 curl -L -O {}.

How do I resume an interrupted curl download?

Add -C - so cURL detects an existing partial file and continues from the correct byte offset: curl -C - -L -O https://example.com/large-file.zip. Adding --retry 3 provides automatic recovery after failures.

Can curl download files that require authentication?

Yes. Use -u username:password for Basic Auth, -H "Authorization: Bearer $TOKEN" for token authentication, and -b cookies.txt for cookie-based sessions. Read credentials from environment variables instead of hard-coding them in scripts.

Why is my curl download returning an HTML page instead of the file?

Two common causes are a missing -L flag, which leaves cURL stopped at a redirect response, or missing authentication. Run curl -IL [URL] to inspect the redirect chain and response headers without downloading the file.

How do I download a large file with curl without it timing out?

Give the connection phase its own deadline with --connect-timeout, and cap the entire transfer with --max-time, for example: curl --connect-timeout 30 --max-time 3600 -C - -L -O [URL]. Adding -C - lets you pick up a partial file after a timeout instead of beginning again.

When should I use a web scraping API instead of curl?

Consider a managed scraping service when content is rendered with JavaScript, automated requests are blocked, or proxy rotation is needed at scale. Plain cURL suits static files and open APIs; browser rendering, anti-bot handling, and managed proxy infrastructure require additional tooling.

# Ubuntu / Debian
sudo apt install curl

# Fedora / Red Hat
sudo dnf install curl

# CentOS / RHEL
sudo yum install curl

# Arch Linux
sudo pacman -S curl

# macOS (ships pre-installed, but to update via Homebrew)
brew install curl
curl --version
# Organised subdirectories by content type
mkdir -p ~/curl-downloads/{datasets,reports,images,logs}
cd ~/curl-downloads
curl [options] [URL]
# Simplest possible download — prints output to terminal (rarely what you want)
curl https://example.com/file.zip

# Save with original filename — the version you'll use most
curl -O https://example.com/file.zip

# Save with a custom filename
curl -o my-dataset.zip https://example.com/file.zip

# The production-ready version (redirects + resume + retry)
curl -L -C - --retry 3 -O https://example.com/file.zip
# Update your CA certificate bundle
sudo apt update && sudo apt install ca-certificates

# Or point cURL to a specific CA bundle
curl --cacert /path/to/ca-bundle.crt -O https://example.com/file.zip
curl -O https://example.com/report.pdf
curl -o quarterly-report-2026.pdf https://example.com/report.pdf
# With dynamic naming using date
curl -o "report_$(date +%Y%m%d).pdf" https://example.com/report.pdf
# Save to a specific directory (directory must exist)
curl -o ./downloads/reports/report.pdf https://example.com/report.pdf

# Create directory structure automatically if it doesn't exist
curl -o ./downloads/reports/2026/Q1/report.pdf --create-dirs https://example.com/report.pdf

# Use the server's Content-Disposition header for filename (when available)
curl -OJ https://example.com/download?id=12345
# Follow redirects
curl -L -O https://example.com/latest-release.zip

# Follow redirects and see where they lead (useful for debugging)
curl -IL https://example.com/latest-release.zip
# Complete silence — no output at all
curl -s -O https://example.com/file.zip

# Silent but still shows errors (the version I actually use)
curl -sS -O https://example.com/file.zip

# Silent with error logging to file
curl -s -O https://example.com/file.zip 2>> ~/curl-downloads/logs/errors.log
# Simple ASCII progress bar
curl -# -O https://example.com/large-dataset.zip

# Detailed progress with transfer stats
curl --progress-bar -O https://example.com/large-dataset.zip

# Custom stats output — useful for logging download performance
curl -O https://example.com/large-dataset.zip \
  --write-out "\nSize: %{size_download} bytes | Speed: %{speed_download} bytes/s | Time: %{time_total}s\n"
# Resume from where it stopped
curl -C - -O https://example.com/large-dataset.zip

# Resume with retry logic
curl -C - --retry 3 --retry-delay 5 -O https://example.com/large-dataset.zip
#!/bin/bash
download_with_resume() {
    local url=$1
    local output=${2:-$(basename "$url")}
    local max_retries=5

    for attempt in $(seq 1 $max_retries); do
        curl -C - -L --retry 3 --retry-delay 5 -o "$output" "$url"
        if [ $? -eq 0 ]; then
            echo "✓ Download complete: $output"
            return 0
        fi
        echo "Attempt $attempt/$max_retries failed. Retrying in 10s..."
        sleep 10
    done
    echo "✗ Download failed after $max_retries attempts: $url" >&2
    return 1
}

# Usage
download_with_resume "https://example.com/huge-dataset.zip" "dataset.zip"
curl \
  -L \              # Follow redirects
  -C - \            # Resume if interrupted
  --retry 3 \       # Retry up to 3 times on failure
  --retry-delay 5 \ # Wait 5 seconds between retries
  -# \              # Show progress bar
  --create-dirs \   # Create output directory if needed
  -o "./downloads/$(date +%Y%m%d)_dataset.zip" \
  "https://example.com/dataset.zip"
# Inline (avoid in scripts — appears in shell history)
curl -u username:password -O https://example.com/protected-file.zip

# Safer: prompt for password at runtime
curl -u username -O https://example.com/protected-file.zip

# Safest: read credentials from environment variables
curl -u "$API_USER:$API_KEY" -O https://example.com/protected-file.zip
curl -H "Authorization: Bearer $TOKEN" -O https://api.example.com/export/data.csv
# Save cookies after login
curl -c cookies.txt -d "username=user&password=pass" https://example.com/login

# Use saved cookies on subsequent requests
curl -b cookies.txt -O https://example.com/protected/file.zip
# Request a specific content type
curl -H "Accept: application/json" -O https://api.example.com/data

# Send multiple headers
curl \
  -H "Authorization: Bearer $TOKEN" \
  -H "Accept: application/zip" \
  -H "X-API-Version: 2" \
  -o export.zip https://api.example.com/export

# Spoof a User-Agent (useful when servers block default cURL UA)
curl -A "Mozilla/5.0 (compatible; MyBot/1.0)" -O https://example.com/file.zip
# HTTP proxy
curl -x http://proxy.example.com:8080 -O https://example.com/file.zip

# SOCKS5 proxy — better for privacy, supports DNS resolution through the proxy
curl --socks5-hostname proxy.example.com:1080 -O https://example.com/file.zip

# Proxy with authentication
curl -x http://proxy.example.com:8080 -U proxyuser:proxypass -O https://example.com/file.zip

# Rotate through a proxy list in a script
PROXIES=("proxy1:8080" "proxy2:8080" "proxy3:8080")
for url in "${URLS[@]}"; do
    PROXY=${PROXIES[$RANDOM % ${#PROXIES[@]}]}
    curl -x "http://$PROXY" -L -O "$url"
done
# Limit download speed to 500KB/s
curl --limit-rate 500k -O https://example.com/large-file.zip

# Add delay between requests in a loop
for url in "${URLS[@]}"; do
    curl -L -O "$url"
    sleep 2  # 2 second pause between requests
done

# Adaptive rate control — back off on 429 responses
download_with_backoff() {
    local url=$1
    local delay=1
    local max_delay=60

    while true; do
        response=$(curl -s -w "%{http_code}" -L -O "$url")
        http_code="${response: -3}"

        if [ "$http_code" = "200" ]; then
            echo "✓ Downloaded: $(basename $url)"
            return 0
        elif [ "$http_code" = "429" ]; then
            echo "Rate limited. Waiting ${delay}s..."
            sleep $delay
            delay=$((delay * 2))
            [ $delay -gt $max_delay ] && delay=$max_delay
        else
            echo "✗ Failed with HTTP $http_code"
            return 1
        fi
    done
}
# Download files numbered 1 through 100
curl -O https://example.com/dataset/file[1-100].csv

# Download with zero-padding (file001.csv, file002.csv...)
curl -O https://example.com/dataset/file[001-100].csv

# Download across multiple directories
curl -O https://example.com/{reports,datasets,images}/file[1-10].zip

# Download specific named files
curl -O https://example.com/{january,february,march}-report.pdf
# Create your URL list
cat > urls.txt << EOF
https://example.com/file1.pdf
https://example.com/file2.zip
https://example.com/file3.csv
EOF

# Download sequentially with xargs
xargs -n 1 curl -L -O < urls.txt

# Download sequentially in a shell loop — more control over each request
while IFS= read -r url; do
    filename=$(basename "$url")
    curl -L -C - --retry 3 -o "./downloads/$filename" "$url"
    echo "✓ $filename"
done < urls.txt
# Invalid: shell interprets & as a background operator
curl -L -O https://example.com/download?id=123&format=csv

# Correct: shell passes the full URL string to cURL intact
curl -L -O "https://example.com/download?id=123&format=csv"

# Also correct in a loop
while IFS= read -r url; do
    curl -L -C - --retry 3 -o "./downloads/$(basename "$url")" "$url"
done < urls.txt
# GNU parallel — the cleanest approach
cat urls.txt | parallel -j 5 curl -L -O {}

# xargs with parallel flag — works without GNU parallel installed
xargs -P 5 -n 1 curl -L -O < urls.txt

# Parallel with output directory and error logging
cat urls.txt | parallel -j 8 \
    'curl -L -C - --retry 3 --silent --show-error \
    -o "./downloads/$(basename {})" {} \
    2>> logs/errors.log'
#!/bin/bash
# Production batch downloader
URLS_FILE="${1:-urls.txt}"
OUTPUT_DIR="${2:-./downloads}"
LOG_FILE="$OUTPUT_DIR/download.log"
CONCURRENCY=5

mkdir -p "$OUTPUT_DIR/failed"

download_file() {
    local url="$1"
    local filename
    filename=$(basename "$url")
    local output="$OUTPUT_DIR/$filename"

    result=$(curl -L -C - --retry 3 --retry-delay 5 \
        --silent --show-error \
        --write-out "%{http_code}|%{size_download}|%{time_total}" \
        -o "$output" "$url" 2>&1)

    http_code=$(echo "$result" | cut -d'|' -f1)
    size=$(echo "$result" | cut -d'|' -f2)
    time=$(echo "$result" | cut -d'|' -f3)

    if [ "$http_code" = "200" ] && [ "$size" -gt 0 ]; then
        echo "$(date '+%Y-%m-%d %H:%M:%S') | ✓ | $filename | ${size}B | ${time}s" >> "$LOG_FILE"
    else
        echo "$(date '+%Y-%m-%d %H:%M:%S') | ✗ | $filename | HTTP $http_code" >> "$LOG_FILE"
        mv "$output" "$OUTPUT_DIR/failed/$filename" 2>/dev/null
    fi
}

export -f download_file
export OUTPUT_DIR LOG_FILE

cat "$URLS_FILE" | parallel -j "$CONCURRENCY" download_file {}

echo "Done. Check $LOG_FILE for results."
# --fail makes cURL exit with error code 22 on 4xx/5xx responses
curl --fail -L -O https://example.com/file.zip

# --fail-with-body does the same but still saves the response body
# (useful for debugging — you can see the error message)
curl --fail-with-body -L -O https://example.com/file.zip

# Check exit code in scripts
curl --fail -L -O https://example.com/file.zip
if [ $? -ne 0 ]; then
    echo "Download failed — server returned an error"
    exit 1
fi
# Download a file alongside its published checksum
curl -L -O "https://example.com/dataset.zip"
curl -L -O "https://example.com/dataset.zip.sha256"

# Verify the checksum — exits with error if mismatch detected
sha256sum -c dataset.zip.sha256

# In a script — fail loudly on checksum mismatch
if sha256sum -c dataset.zip.sha256; then
    echo "✓ Integrity verified: dataset.zip"
else
    echo "✗ Checksum mismatch — file may be corrupted or incomplete" >&2
    exit 1
fi

# When no published checksum exists, generate and store your own
# on first download, then compare on subsequent pulls
sha256sum dataset.zip > dataset.zip.sha256.local
# Connection timeout — server not responding fast enough
curl --connect-timeout 30 -O https://example.com/file.zip

# Max total time — kill the request if it takes too long overall
curl --max-time 300 -O https://example.com/large-file.zip

# Both together — connect within 30s, complete within 5 minutes
curl --connect-timeout 30 --max-time 300 --retry 3 -O https://example.com/file.zip

# Verbose output for debugging network issues
curl -v -O https://example.com/file.zip 2>&1 | head -50
# See what the actual certificate error is
curl -v https://example.com/file.zip 2>&1 | grep -A5 "SSL"

# Update CA certificates (fixes most "certificate verify failed" errors)
sudo apt update && sudo apt install --reinstall ca-certificates

# Specify a custom CA bundle
curl --cacert /path/to/corporate-ca.crt -O https://internal.example.com/file.zip

# For self-signed certs in dev environments ONLY
curl -k -O https://localhost:8443/file.zip
# Check you have write permission to the target directory
ls -la ./downloads/

# Create directory if it doesn't exist
mkdir -p ./downloads && curl -o ./downloads/file.zip https://example.com/file.zip

# Check available disk space before a large download
df -h . && curl --limit-rate 10m -O https://example.com/huge-dataset.zip
# Basic page scrape — returns rendered HTML
curl -G "{{APP_BASE_URL}}/api/v1/" \
  --data-urlencode "api_key=YOUR_API_KEY" \
  --data-urlencode "url=https://example.com"

# With JavaScript rendering enabled (default)
curl -G "{{APP_BASE_URL}}/api/v1/" \
  --data-urlencode "api_key=YOUR_API_KEY" \
  --data-urlencode "url=https://example.com" \
  --data-urlencode "render_js=true"

# AI-powered data extraction — returns structured JSON, no parsing required
curl -G "{{APP_BASE_URL}}/api/v1/" \
  --data-urlencode "api_key=YOUR_API_KEY" \
  --data-urlencode "url=https://example.com/products" \
  --data-urlencode "ai_query=Extract all product names and prices" \
  --data-urlencode 'ai_extract_rules={"product_name":{"type":"string","description":"Example post details."},"price":{"type":"string","description":"Example post details."}}'
HTTP CodeWhat it meansTypical fix
301 / 302Redirect or file movedAdd -L to follow
400Bad requestCheck URL encoding and parameters
401UnauthorisedAdd authentication (-u or -H "Authorization:...")
403ForbiddenCheck credentials, User-Agent, or IP restrictions
404Not foundVerify the URL, as the file may have moved
429Rate limitedAdd --retry-delay or --limit-rate
503Server unavailableAdd --retry with exponential backoff