How to extract a table's content in Ruby
HTML tables are one of the most common ways structured data appears on the web, and pulling their rows out by hand is tedious.
This guide shows a Ruby workflow that sends a page URL plus a table CSS selector to a table extraction endpoint and returns the table already parsed, so you skip writing your own HTML post-processing step.
How the Ruby table extraction flow works
The approach relies on a table extraction feature that accepts a CSS selector inside a set of extract_rules. Instead of returning raw HTML for you to parse, the service returns the table's contents in a structured form. The Ruby script below builds a GET request with Net::HTTP, passes the target URL and the rules as query values, and prints the response body. The example targets a demo page of NASDAQ top-100 stock prices, where the relevant table is identified by the selector .BasicTable-table in the source narrative, while the runnable example uses .table. Treat the selector as the one thing you must confirm against your own target page before running anything.
Set APP_BASE_URL, API_KEY and DEMO_BASE_URL, where the demo origin serves /table_content.html. The documented .table selector and table_json/table_array output names are unchanged. Both the rule object and its query-string value are encoded before the GET /api/v1/ request.
require "net/http"
require "uri"
require "json"
require "openssl"
rules = {
"table_json" => {"selector" => ".table", "output" => "table_json"},
"table_array" => {"selector" => ".table", "output" => "table_array"}
}
target = ENV.fetch("DEMO_BASE_URL").sub(%r{/+\z}, "") + "/table_content.html"
endpoint = URI(ENV.fetch("APP_BASE_URL").sub(%r{/+\z}, "") + "/api/v1/")
raise ArgumentError, "Use an HTTPS API origin" unless endpoint.scheme == "https"
endpoint.query = URI.encode_www_form(
"api_key" => ENV.fetch("API_KEY"),
"url" => target,
"extract_rules" => JSON.generate(rules)
)
response = Net::HTTP.start(endpoint.hostname, endpoint.port,
use_ssl: true, verify_mode: OpenSSL::SSL::VERIFY_PEER) do |connection|
connection.request(Net::HTTP::Get.new(endpoint))
end
puts "HTTP #{response.code}"
puts response.bodyRequest field map: what you send and what you get back
Before running the script, it helps to know which inputs drive the request and which outputs you can expect. The table below is a reader aid, not a replacement for the API contract delivered separately.
- api_key — your credential for the extraction endpoint.
- url — the page that contains the table you want.
- extract_rules — a JSON object whose keys are your own labels and whose values contain a selector and an output mode.
- selector — the CSS selector that identifies the table element.
- output — either table_json (objects keyed by column header) or table_array (rows as arrays).
- Response body — a JSON object whose keys match your extract_rules labels, each holding the parsed table.
Because the rules are serialized to JSON before being sent, keep the hash keys stable so you can map responses back to the right table.
Choosing between table_json and table_array
The two output modes answer different downstream needs, so pick based on what consumes the data next.
- Choose table_json when you want each row as an object keyed by the table's column headers. This is convenient when you will load rows into a database, feed a JSON API, or reference columns by name in Ruby code.
- Choose table_array when you want each row as a plain ordered array. This is convenient for spreadsheet-style pipelines, CSV writers, or any consumer that already knows the column order.
You can request both in the same call by giving them separate rule keys, which is what the example does. That way you can compare the two shapes against the same source table before committing to one.
Worked example: what the response looks like
The captured response for the demo table shows both shapes side by side. In the JSON form, the first row appears as an object with keys such as SYMBOL, NAME, PRICE, CHANGE, and %CHANGE, holding values like AMD, Advanced Micro Devices Inc, 94.82, -3.98, and -4.03. In the array form, the same row appears as ["AMD", "Advanced Micro Devices Inc", "94.82", "-3.98", "-4.03"]. The exact response module is delivered alongside this page; the numbers above are a dated reference from that captured output, not a live quote, so re-run the request to get current values.
A practical reading of the difference: if your Ruby code later does row["PRICE"], you want table_json. If it does row[2], you want table_array. Mixing the two expectations is the most common cause of a NoMethodError or a nil when you first wire the response into your own code.
Checklist before you run the script
Work through these in order; each one maps to a distinct failure mode.
- Confirm the CSS selector matches the table on the live page. A selector that matches nothing yields an empty or missing table, not an HTTP error.
- Confirm the target URL is reachable and returns HTML, not a redirect to a login or a JavaScript-only shell. A transport failure surfaces as an HTTP error in the rescue block.
- Confirm your api_key is present and valid. An authentication failure is distinct from a selector miss.
- Confirm the extract_rules hash is serialized with to_json before being sent; sending a Ruby hash directly will not produce the expected query value.
- Confirm the output mode matches your downstream consumer (see the previous section).
- Confirm you are reading the response body as JSON, not as a raw string, if you plan to index into rows.
If the request succeeds but the table is empty, the selector is the first thing to check. If the request fails before any response, check the URL and credential.
Frequently asked questions
What CSS selector should I use for a table?
Use the selector that uniquely identifies the table element on the target page. In the captured example the table is identified by .BasicTable-table in the narrative and .table in the runnable code, so always verify the selector against the live page before running the request.
When should I use table_json instead of table_array?
Use table_json when downstream code refers to columns by name, such as row["PRICE"]. Use table_array when downstream code refers to columns by position, such as row[2]. You can request both in one call by giving them separate rule keys.
Why is my extracted table empty even though the request succeeded?
An empty table with a successful HTTP response usually means the CSS selector did not match any element on the page. Check the selector against the rendered HTML, and confirm the page is not returning a login screen or a JavaScript-only shell in place of the table.