Skip to content

Ruby: Build Your First HTML Scraping API Request

This guide shows how to call a third-party HTML scraping API from Ruby.

OUTPUT FORMATSHTMLJSONMarkdown

You will build a small function that sends a target URL and an API key as query values, reads the HTTP status code and response body, and writes the returned HTML to a file. The examples use Ruby's standard net/http library and the addressable gem for safe query encoding.

What you are building

You will write a Ruby function that makes a GET request to the scraping API's HTML endpoint. The request carries two query values: your API key and the URL you want to fetch. The function returns the response object so you can inspect its status code and body. In the worked example, you then save the body to a local HTML file.

Before you start, decide how you will handle the response:

  • Check the status code first. A 2xx code means the request succeeded; anything else needs investigation.
  • Read the body only after confirming the request completed. The body may contain an error message rather than page HTML.
  • Keep the API key out of source control. Load it from an environment variable or a local configuration file.
  • Encode the target URL as a query value rather than concatenating it into the path.

1. The general structure of an API request

Every request follows the same shape: build a URI with the API key and target URL as query values, create an HTTP client, send a GET request, and return the response. The exact Ruby script for this general structure is supplied as a protected module and accompanies this page; it uses net/http with SSL verification enabled and rescues StandardError to report a failed request.

Once the function returns, you can inspect the response object. The public contract module below shows how to print the status code and body:

{{code0002}}

A useful distinction: the rescue block catches transport-level failures such as DNS errors or timeouts, while a non-2xx status code is still a completed HTTP response. Check the status code before treating the body as page HTML.

Set APP_BASE_URL to the HTTPS API origin, API_KEY to your key and TARGET_URL to the page to fetch. This standard-library example encodes query values and keeps certificate verification enabled.

require "net/http"
require "uri"
require "openssl"

def fetch_html(target)
  endpoint = URI(ENV.fetch("APP_BASE_URL").sub(%r{/+\z}, "") + "/api/v1/")
  raise ArgumentError, "Use an HTTPS API origin" unless endpoint.scheme == "https"
  endpoint.query = URI.encode_www_form("api_key" => ENV.fetch("API_KEY"), "url" => target)
  request = Net::HTTP::Get.new(endpoint)
  Net::HTTP.start(endpoint.hostname, endpoint.port,
    use_ssl: true, verify_mode: OpenSSL::SSL::VERIFY_PEER) do |connection|
    connection.request(request)
  end
end

response = fetch_html(ENV.fetch("TARGET_URL"))
puts "HTTP #{response.code}"
puts response.body

2. Your first API request

The worked example fetches a blog page through the API and saves the returned HTML to blog.html. The full Ruby script is supplied as a protected module and accompanies this page. It uses Addressable::URI to set the api_key and url query values, sends a GET request with net/http, prints the status code, and writes the response body to a file.

Field map for the request:

  • api_key: your API credential, passed as a query value.
  • url: the target page you want the API to fetch, passed as a query value and encoded by Addressable::URI.

After running the script, open blog.html to confirm the response body contains the page HTML you expected. If the status code is not 2xx, inspect the body for an error message before saving it as HTML.

Use fetch_html from the preceding example and set SITE_BASE_URL to the website origin. This saves that website’s /blog/ response body to blog.html; inspect the HTTP status and body before assuming it is the expected page.

target = ENV.fetch("SITE_BASE_URL").sub(%r{/+\z}, "") + "/blog/"
response = fetch_html(target)
puts "HTTP #{response.code}"
File.write("blog.html", response.body)