Skip to content

Parse HTML in Ruby with Nokogiri: Extract Text and Handle Pagination

When a site offers no usable API, or its API leaves out fields shown on the page, reading the HTML directly can fill the gap. This guide shows a practical Ruby workflow with Nokogiri: fetch a page, locate rows with CSS selectors, pull text from cells, total repeated values, and walk through every result page.

What Nokogiri does

Nokogiri is a lightweight Ruby gem for working with HTML and XML documents. It can parse, build and search documents, and it provides DOM, SAX and push parsing styles. For everyday scraping, the most useful part is selector-based search: CSS3 selectors handle most element lookups, with XPath available when a selector is awkward. The example in this guide uses a public practice page listing NHL team statistics since 1990. The goal is simple: accept a search term, find matching teams, and report each team's total wins.

Install the documented Nokogiri gem before running the Ruby examples.

gem install nokogiri

Inspect the page before coding

Start by exploring the page in a browser before writing extraction code. Search for a term and note how the address changes, because that pattern becomes the request URL. Then open the developer console, use the element picker, and hover over a result row to see its markup. In the example, each team sits in a table row marked with the team class, the name cell carries the name class, and the wins cell carries the wins class. That inspection turns into three selectors: tr.team for all rows, .name for the first name cell inside a row, and .wins for the wins cell. A useful pre-flight checklist is: confirm the search URL pattern, confirm one row's markup, confirm the pagination control, and confirm whether content appears without JavaScript.

Fetch and parse the first page

The workflow has four stages. First, set up a project folder and install the gem. Second, load the Nokogiri and open-uri libraries, where open-uri provides a convenient wrapper around net/http for fetching a URL. Third, read a search term from standard input and pass the constructed URL through Nokogiri::HTML5 to obtain a document. Fourth, select rows, read cell text, and print results. Protected modules code0002 through code0006 carry the exact setup, imports, prompt and fetch steps, so they should be reused unchanged rather than rewritten.

Set SEARCH_TERM to the team search term. This uses Nokogiri::HTML from the combined example, encodes q/page_num and closes each downloaded stream after parsing. The sandbox must contain the documented tr.team and .name elements.

require "nokogiri"
require "open-uri"
require "uri"

def load_team_page(term, number)
  address = URI("https://www.scrapethissite.com/pages/forms/")
  address.query = URI.encode_www_form("q" => term, "page_num" => number)
  URI.open(address.to_s) { |stream| Nokogiri::HTML(stream) }
end

search_term = ENV.fetch("SEARCH_TERM")
first_page = load_team_page(search_term, 1)
first_page.css("tr.team").each do |row|
  puts row.at_css(".name").text.strip
end

Select rows and aggregate values

The css method returns every element matching a selector, which suits collecting all team rows at once. Inside the loop, the at method returns only the first match, which suits the single name and wins cells within one row. Reading content gives the text, strip removes surrounding whitespace, and to_i converts a wins value to a number. A hash keyed by team name accumulates totals: an unseen team starts at zero, and a repeated team adds its wins to the existing total. This aggregation matters because the same team can appear across multiple rows or pages. Protected modules code0007 and code0008 contain the exact extraction and aggregation logic.

Run after the first-page example. The .name and .wins selectors and integer win conversion follow the documented workflow; inspect cell values before aggregating.

def add_wins(document, totals)
  document.css("tr.team").each do |row|
    name = row.at_css(".name").text.strip
    wins = row.at_css(".wins").text.to_i
    totals[name] += wins
  end
end

totals = Hash.new(0)
add_wins(first_page, totals)
totals.each { |name, wins| puts "#{name} => #{wins}" }

Walk every page

A single page rarely holds every match. The pagination links live inside a list marked with the pagination class, and the page count is the number in the second-to-last link, because the final link points to the last page without showing its number. Once that count is known, loop from page 1 to the count and add a page_num parameter to each request. Page 1 uses the same parameter form as later pages, so no special first-page branch is needed. Keep one teams hash outside the loop so totals accumulate across all pages, then print the finished hash after the loop ends. Protected modules code0010 through code0015 carry the exact pagination steps.

Use load_team_page, first_page, search_term and add_wins from the previous examples. Replace the single-page totals block with this block so the first page is counted once. Page numbers come from the documented page_num links rather than a brittle second-to-last-link position.

last_page = first_page.css(".pagination a").map do |link|
  address = URI.join("https://www.scrapethissite.com/pages/forms/", link["href"].to_s)
  query = URI.decode_www_form(address.query.to_s).to_h
  query.fetch("page_num", "1").to_i
end.max || 1
last_page = [1, last_page].max

totals = Hash.new(0)
1.upto(last_page) do |number|
  page = number == 1 ? first_page : load_team_page(search_term, number)
  add_wins(page, totals)
end
totals.each { |name, wins| puts "#{name} => #{wins}" }

Know the limits

Nokogiri fits static or server-rendered pages where the needed markup is present in the fetched HTML. It does not run JavaScript, so content generated only after script execution will be missing from the parsed document. For those pages, a browser-driving tool is the better fit. As a decision rule: if the data is in the initial HTML, Nokogiri is lightweight and direct; if the data appears only after JavaScript runs, switch approaches. The core pattern from this guide — fetch, select with CSS, read content, aggregate, paginate — transfers to any similarly structured table.

Selector method map
MethodReturnsUse it when
cssAll matching elements as a collectionCollecting every team row at once
atFirst matching element onlyReading the single name or wins cell inside one row
contentText inside the elementTurning a selected cell into a readable value

Based on the worked example's selector usage.

Pre-flight checklist
CheckWhat to confirm
Search URLThe query parameter that carries the search term
Row markupRow, name and wins classes used by the selectors
PaginationWhere the page count appears and the page parameter name
RenderingWhether the data exists without JavaScript execution

Pre-flight checklist derived from the example page's structure.

Frequently asked questions

What is Nokogiri?

Nokogiri is a lightweight Ruby gem for parsing, building and searching HTML and XML. It offers DOM, SAX and push parsers, plus XPath and CSS3 selector search.

How do I extract text from selected elements?

Use a CSS selector with the css method to collect matching elements, use at when only the first match inside a parent is needed, then read its text with content and trim surrounding whitespace with strip.

How can pagination be handled?

Inspect the pagination links, take the page number from the second-to-last link, then loop from page 1 through that count and request each page with its page_num parameter.

Can Nokogiri scrape JavaScript-driven pages?

No. Nokogiri cannot execute JavaScript, so pages whose content depends on JavaScript execution need a browser-automation approach instead.