Skip to content

Web Scraping with Ruby: Faraday, Nokogiri, Selenium and Capybara

This guide builds a Ruby scraper in stages. It starts with a simple HTTP request, adds HTML parsing, then moves to browser automation for JavaScript-driven pages, and finishes with a parallel product scraper that saves results to CSV. Each stage uses the same underlying idea: fetch or load a page, locate the relevant elements, extract only the fields you need, and store them in a usable form.

Choose the right Ruby tools for the job

Ruby offers several useful libraries for extraction work. Faraday provides an adapter-based HTTP client with middleware support. Nokogiri parses HTML and XML and supports CSS selectors and XPath. HTTParty is aimed at readable, object-like requests. Mechanize is suited to navigating pages and handling forms, links, and cookies. Capybara and Selenium are used when a real browser is needed. Other tools in the ecosystem cover crawling, metadata extraction, URL handling, parallel execution, and fast JSON processing. Choose based on the page behaviour rather than collecting every gem at once.

Set up a small Ruby project

You need basic Ruby knowledge, including running scripts and installing gems, plus Ruby 3.x, a code editor, and a terminal. No Rails setup is required. Create a project folder and initialise Bundler to generate a Gemfile, then create the main script file. The supplied setup modules are code0001, code0002, and code0003.

Make a first HTTP request with Faraday

Faraday is used for the first request because it is flexible and extensible. Add the gem in module code0004 and install dependencies with module code0005. Module code0006 makes a GET request and prints the status, headers, and the start of the body. Run it through Bundler using module code0007. The expected shape of the output is shown in module code0008. A successful response confirms that the project, dependencies, and network request are working before parsing is added.

Customise Faraday requests and middleware

For JSON APIs or compressed responses, Faraday can be configured with a connection object, default headers, and middleware. Module code0009 adds gzip support and module code0010 installs it. Module code0011 creates a connection with JSON headers and a custom user agent, enables gzip handling, performs a GET request, and then sends a POST request with a JSON body. The important distinction is that middleware changes how requests and responses are processed without changing the basic GET and POST calls.

Parse HTML with Nokogiri

Once HTML has been fetched, Nokogiri turns the raw response into a document that can be searched. Add it using module code0012 and install it with Bundler; native extensions may make installation take longer. Module code0013 shows the simplified sample page structure. Module code0014 fetches the page, parses it, and extracts the heading, first paragraph, link text, and link destination. Safe navigation is used so missing elements do not cause the script to fail. The key methods are finding one element, reading its text, and reading an attribute such as href.

Scrape a real movie list with Nokogiri

Inspect the page in browser developer tools before writing selectors. For the Rotten Tomatoes example, the relevant list is under the media-lists container, and the first dynamic poster list represents movies in theatres. Each movie is represented by a tile-dynamic element. Inside a tile, the poster comes from the image element, the title is inside a button, and critics and audience scores are in separate score slots. Module code0015 limits the example to five movies and prints the poster, title, and any available scores. This inspection-first approach prevents guessing selectors from the visible layout alone.

Handle blocked or difficult requests

Some sites throttle requests, serve different content to automated clients, require JavaScript, or present challenges. A scraping API can be used as an alternative request path, but its base URL, credential, pricing, and service terms must be configured by the owner through {{OWNER_API_BASE_URL}}, {{OWNER_API_KEY}}, and {{OWNER_SCRAPING_API_TERMS}}. The request-routing pattern is represented by protected module code0016. After the response returns, the same Nokogiri parsing logic can be reused. Keep the API-specific change separate from the extraction logic so selectors do not need to be rewritten.

Add retries and save results to CSV

Temporary failures can still occur even when a managed request service is used. Module code0017 adds retry support. Protected module code0018 registers retry middleware before gzip handling. The retry settings control the maximum number of retries, the first interval, random variation in the delay, and the backoff factor. Randomised exponential backoff reduces the chance that many clients retry at exactly the same moment. For CSV output, module code0019 adds the CSV gem, which is needed explicitly from Ruby 3.4 onward. Module code0020 writes the movie title, poster URL, critics score, and audience score with headers. The resulting file can be opened in a spreadsheet application or passed to another system.

Use Selenium and Capybara for dynamic pages

Static parsing is insufficient when content is created by JavaScript, loaded lazily, or hidden behind interaction. Selenium controls a real browser, while Capybara provides a more readable Ruby interface for visiting pages and interacting with elements. Module code0021 adds both gems. A matching browser and driver are also required: Chrome uses ChromeDriver and Firefox uses GeckoDriver. The macOS setup modules are code0022 and code0023. On Windows, install the browser and driver and add the driver location to the system PATH. The driver must be discoverable by Ruby before the browser automation will start.

Automate clicks, popups, and screenshots

Module code0024 opens the Rotten Tomatoes homepage, scopes the search to the media list, finds the first movie link, opens its detail page, waits for the trailer control, and clicks it when present. Scoping the search avoids matching unrelated elements elsewhere on the page. Checking for the trailer control before clicking prevents a failure on pages that do not offer one. Module code0025 dismisses a cookie consent overlay when it appears. Module code0026 saves a screenshot of the current page. These checks reflect a common browser-scraping pattern: wait for expected elements, handle overlays, and verify optional controls before interacting with them.

Run browser automation in headless mode

A visible browser is useful during development but unnecessary for many automated runs. Headless mode keeps the browser functional without displaying a window, which is helpful for background work, CI, and remote servers. Module code0027 registers a headless Firefox driver, visits the homepage, finds a movie link, saves a screenshot, and opens the detail page. The same interaction code can be used after the driver changes; only the browser launch configuration differs.

Build a parallel Micro Center product scraper

The final example searches Micro Center for microphones. Module code0028 defines the target site, product limit, concurrency limit, screenshot directory, CSV filename, logger, headless Firefox setup, and a prompt that rejects an empty search term. Module code0029 opens the homepage, clicks the search field before entering the query, submits the form, waits for the results indicator, and saves a screenshot. Clicking the field first matters because the form may not accept input until it has been focused. Module code0030 collects a limited number of product links from the results grid and resolves relative links against the site root.

Scrape product pages in parallel and save CSV

Module code0031 distributes product URLs through a thread-safe queue. Each worker creates its own browser session, visits a product page, extracts the title and price, saves a screenshot, and places the result in another queue. Results are sorted by their original index after all workers finish. Limiting concurrency is important because sending too many simultaneous requests can provoke blocking. Module code0032 writes the title, price, and URL to CSV. Module code0033 combines the complete scraper, and module code0034 shows the Bundler command used to run it. Enter a term such as podcasting to test the search, product collection, parallel visits, screenshots, and CSV output.

A practical decision checklist

Start with Faraday when the response already contains the data. Add Nokogiri to parse that response. Add retries when temporary failures are likely, and CSV output when results need to be retained. Move to Selenium and Capybara when JavaScript, interaction, or browser state is involved. Use headless mode for unattended runs, screenshots for debugging, and bounded concurrency for parallel work. That progression keeps the scraper as simple as the target page allows while leaving room to handle more complex sites.

Static parsing or browser automation?
SituationRecommended approachWhy
Data is present in the HTML responseFaraday plus NokogiriFast and simple; no browser overhead.
Content loads with JavaScriptSelenium plus CapybaraA real browser executes scripts and waits for rendered elements.
The task requires clicks, forms, or scrollingSelenium plus CapybaraBrowser automation can reproduce user interaction.
Temporary network or server failures occurFaraday retry middlewareRetries with randomised backoff handle transient errors.
Many independent pages need visitingBounded parallel browser sessionsParallelism improves throughput while a concurrency limit reduces blocking risk.
Results must be retained or sharedCSV outputA portable format works with spreadsheets and downstream tools.

The choice depends on whether the required content is present in the initial response or appears only after browser execution and interaction.

Frequently asked questions

When is Nokogiri enough and when do I need a browser?

Use a plain HTTP request plus an HTML parser when the needed markup is already present in the response. Move to a real browser when content appears only after JavaScript runs, or when you must click, scroll, type, dismiss a popup, or wait for elements to load.

What does Capybara add to Selenium?

Selenium drives the browser at a low level. Capybara sits on top of it and provides a Ruby-oriented way to visit pages and interact with visible elements. In this guide they are used together, with Capybara handling page actions and Selenium providing the browser engine.

Why run Selenium without a visible browser window?

Headless mode runs the same browser automation without opening a visible window. It suits background jobs, CI, and remote servers, and screenshots can still be saved for debugging.

How should temporary request failures be handled?

Add retry middleware for temporary failures such as timeouts, dropped connections, or server errors. The example retry settings distinguish the number of retries, the initial interval, added randomness, and exponential backoff.