Skip to content

Run Concurrent Scraping Requests in Ruby with Threads

A single Ruby script can send more than one scraping request at a time.

OUTPUT FORMATSHTMLJSONMarkdown

Each call gets its own thread, the threads run side by side, and you join them before the program exits so the responses finish printing. The practical ceiling on how many calls are truly in flight is the concurrency limit attached to your plan.

What concurrency buys you

The scraping API is built to handle several scraping operations at once. Because calls can overlap instead of queueing one behind another, a script can work through far more pages in a day than a strictly sequential loop would allow — the exact scale depends on the plan you are on, ranging from hundreds up to thousands or even millions of pages per day. The controlling factor is your concurrency limit: the higher that limit, the more calls you can keep active in parallel, and the faster the overall scrape completes.

The request shape

Each call is a plain HTTP GET to the API endpoint with two query values: your API key and the target URL you want fetched. The response body is the HTML of that target page. In the example the two targets are a blog index and a documentation page, and each is wrapped in its own thread so the second does not wait for the first.

Checklist before you raise the thread count

  • Confirm your plan's concurrency limit; that number, not the thread count, decides how many calls run in parallel.
  • Keep the API key out of committed code; load it from an environment variable.
  • Join every thread, otherwise the script can exit before responses are printed.
  • Treat each thread's failure separately: the example rescues errors inside the request function and prints a message, so one bad URL does not stop the other thread.
  • Watch for rate-limit responses when you push past your limit; back off rather than adding more threads.

Field and control map

The request function takes two inputs and the script adds two threads plus a join step. Use this map to see where each piece belongs:

  • API key — sent as the api_key query value; identifies your account.
  • Target URL — sent as the url query value; the page the API should fetch.
  • Thread 1 — Thread.new wrapping the request function for the first target.
  • Thread 2 — Thread.new wrapping the request function for the second target.
  • join — called on each thread; blocks until that request finishes.

Worked example

Suppose you want the HTML of two pages on the same site. You write one function that builds the request from the API key and a URL argument, then start two threads, each calling that function with a different URL. Both requests are in flight together; the script waits on the first, then the second, and prints both bodies before ending. If the first URL returns an error, the rescue block prints the failure message and the second thread still completes.

Frequently asked questions

Does creating more threads raise my concurrency limit?

No. The limit comes from your plan. Threads let you use the limit you have; they do not extend it.

What happens if one request fails?

The example catches the error inside the request function and prints a message, so the other thread keeps running.

Why call join on each thread?

join makes the main program wait for that thread to finish, so the response body is printed before the script ends.