Skip to content

Web Scraping With Linux and Bash: curl, jq, and Xidel Workflows

A Unix shell can fetch pages, extract fields, and save results without setting up a full application runtime. This guide explains when that approach fits, which command-line tools cover each stage, and how the captured examples move from a single request to paginated JSON, HTML extraction, crawling, and scheduled runs.

Choose Bash for the job size, not by habit

High-level languages offer broad scraping libraries, but they also bring runtimes, dependencies, and learning overhead. A Unix machine usually already provides a shell and many small utilities, so a short Bash pipeline can be the fastest route for a simple extraction. A JSON endpoint, for example, may be handled quickly by fetching it and filtering the response, without starting a browser-based stack. That speed advantage narrows as requirements grow. Complex session management, parallel work, JavaScript-heavy pages, and database access with careful escaping or prepared statements are better matched to dedicated platforms. Use this decision check before coding: confirm the target returns usable HTML or JSON, the fields are stable enough for selectors or JSON paths, the volume fits sequential shell processing, and the output can be written safely to a file or database. If several answers are no, choose a fuller scraping stack.

Assign each tool a narrow role

Think of the shell as glue between specialized tools. curl sends HTTP requests and returns responses. grep finds matching text, sed can also replace it, and head and tail keep only the first or last lines. jq filters JSON, extracts fields, remaps values, and reshapes documents. Xidel serves the equivalent role for HTML and XML through XPath expressions and CSS selectors. html2text is narrower: it removes tags and returns the plain text from the page body. For debugging jq expressions, jqp presents the query, the original JSON, and the current result together; the captured example uses module code0001 to pipe a subreddit response into jqp and module code0002 for a query that limits output to ten titles. Exit jqp with Ctrl+C. Install curl, jq, and html2text through your distribution's package manager or from their project download pages; grep, sed, head, and tail are standard Unix tools that are typically already present.

From one request to a bounded paginated crawl

The first captured example, module code0003, fetches a public IP address with curl, stores the response in a variable, and writes a labeled line to myip.data; module code0004 shows the expected output shape. The script anatomy matters more than the endpoint: the shebang selects the interpreter, command substitution captures a command's output in a variable, and redirection writes expanded text to a file. For the Reddit example, module code0005 adds a browser-style user-agent header because the default curl identifier can receive a 429 response. Appending .json to the subreddit URL returns JSON without authentication. Module code0007 extracts the pagination token and posting URLs with jq. Module code0008 then loops while tracking the current URL, collected URLs, iteration count, and stop conditions: more than 100 entries, a null pagination token, or five iterations. It deduplicates against the output file before appending new URLs. The practical lesson is to make pagination and termination explicit rather than relying on an open-ended loop.

Inspect HTML first, then automate the selector

HTML scraping starts by inspecting the page structure and choosing the most stable selector. In the Wikipedia actor example, the visible infobox points toward the birth information, while a hidden span with class bday provides a cleaner ISO 8601 date. The page title supplies the name. Module code0010 gives the birth-date XPath expression, and the source also identifies the title selector in prose. Module code0009 lists the ten actor URLs used as input. Module code0011 reads that file line by line, downloads each page with curl, passes the HTML to Xidel, stores the two extracted values in an array, and appends a CSV row. Set the input field separator to newline so multi-value output is split correctly. Before automating, inspect one representative page and confirm that both selectors return the intended name and date; a selector that works on one article may fail on another layout.

Let Xidel crawl, but normalize missing fields

Xidel can also fetch a starting page and follow selected child links, which removes the need for a separate HTTP client and a hand-built URL list. Module code0012 starts from a Wikipedia category, follows category links, extracts each linked page's title and birth date, and writes pairs to CSV. The captured example exposes an important failure mode: an article without the birth-date element breaks the expected pairing. Module code0013 addresses that gap by requesting wrapped JSON from Xidel, normalizing it with jq, and skipping entries whose birth-date value is null. Treat missing fields as a normal crawl outcome, not as an exception to ignore silently; log or count skipped pages so incomplete output is visible.

Schedule only after a manual run is bounded and repeatable

Once a script works manually, schedule it with cron. Install the scheduler appropriate to the distribution, as shown in module code0014, then edit the user's crontab and add the schedule and script path shown in module code0015. That example runs at the top of every hour; cron syntax supports intervals from one minute to once a year. Before enabling a schedule, run the script once by hand, confirm the output file and exit behavior, and use an absolute script path so the job does not depend on the interactive shell's working directory. Keep the loop bounds, deduplication, and missing-field checks from the examples so unattended runs terminate and do not repeatedly append the same records.

Shell tool map for scraping tasks
StageToolUse it whenWatch for
FetchcurlYou need a configurable HTTP request or responseA site may reject the default user agent with 429
Plain textgrep, sed, head, tailYou need matching, replacement, or line limitsRegular expressions can become brittle on changing markup
JSONjq; jqp for debuggingYou need field extraction or reshapingPagination tokens and null values must be handled explicitly
HTML/XMLXidelYou need XPath or CSS-selector extraction and link followingMissing elements can break positional pairing
Visible text onlyhtml2textTags are unwanted and element-level data is unnecessaryIt does not preserve element structure

Tool choice depends on response format and job complexity; JavaScript-heavy or parallel workloads may need a dedicated platform.

Frequently asked questions

When is Bash the wrong choice for scraping?

Bash suits small, well-structured jobs where the shell and preinstalled Unix tools are enough. It becomes harder when a job needs complex session handling, parallel execution, JavaScript-heavy pages, or careful database writes, where dedicated scraping libraries and headless-browser support are usually stronger.

Which shell tool handles which scraping step?

Use curl to fetch, grep or sed to search and replace text, head and tail to limit lines, jq to filter and reshape JSON, Xidel to query HTML or XML with XPath or CSS selectors, and html2text when only the visible page text is needed. jqp can help test jq expressions interactively.

What setup do the examples assume?

The examples assume Bash at /bin/bash and executable scripts. If that path differs, adjust the shebang, set the executable flag, or run the script explicitly with bash and the script filename.

How do you handle a missing field in a crawled page?

Some pages lack an expected element, such as a birth-date span. Xidel cannot directly test for a missing element in the described workflow, so output JSON, normalize it with jq, and skip entries whose value is null.

How can a finished scraper run automatically?

Install the cron scheduler for the distribution, open the crontab with crontab -e, and add a line containing the schedule and the full path to the script. The example schedule runs at the top of every hour.

curl https://www.reddit.com/r/json/new.json | jqp
#!/bin/bash
my_ip=$(curl https://api.ipify.org)
echo "My IP address is: $my_ip" > myip.data
#!/bin/bash

BASE_URL='https://www.reddit.com/r/bash/new/.json'
USER_AGENT='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36'

content=$(curl -H "User-agent: $USER_AGENT" $BASE_URL)
echo $content
{
    "kind": "Listing",
    "data": {
        "after": "t3_ewz3xu",
        "before": null,
        "children": [ LOTS_OF_DATA_HERE],
        EVEN_MORE_DATA
    }
}
#!/bin/bash

BASE_URL='https://www.reddit.com/r/bash/new/.json'
USER_AGENT='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36'
MAX_ITERATION=5
DATA_FILE=reddit.urls


current_url=$BASE_URL
data=()
iteration=1

while :; do
	content=$(curl -s -H "User-agent: $USER_AGENT" $current_url)

	after=$(echo $content | jq -r .data.after)
	urls=$(echo $content | jq -r '.data.children[].data.url')

	for url in $urls; do data+=($url); done

	if [ ${#data[@]} -gt 100 ] || [ "$after" = "null" ] || [ $iteration -ge $MAX_ITERATION ]; then
		break
	fi

	current_url="$BASE_URL?after=$after"

	((iteration+=1))
done

for url in "${data[@]}"; do
	# Check if URL is already present
	if grep -sc $url $DATA_FILE > /dev/null; then
		continue
	fi

	# Append URL
	echo $url >> $DATA_FILE
done
#!/bin/bash
IFS=$'\n'
while read -r line; do
        # Load HTML content
        content=$(curl -s $line)

        # Extract data with XPath expressions into content array
        content=($(echo $content | ./xidel - -s -e '//span[@class="mw-page-title-main"]' -e '//span[@class="bday"]'))

        echo ${content[0]},${content[1]} >> actors.csv
done < actors.urls