Skip to content

Elixir Web Scraping with Crawly and Floki: A Price Spider Walkthrough

Elixir is a practical choice when a scraping job needs to run many concurrent requests and stay stable over time. This guide walks through a small price-tracking spider built with Crawly, a scraping framework for Elixir, and Floki, an HTML parsing library. You will create a supervised project, define a spider, pull a price out of a product page, and then extend the spider so it discovers product links from a search results page. The example uses Amazon graphics-card listings; the same pattern applies to any site whose markup you can inspect.

What this walkthrough builds

The target is a price collector: a spider that visits product pages, reads the displayed price, and writes each result to a file. That output is the raw material for a price-comparison or alert service. The first version visits a fixed list of URLs. The second version starts from a search results page, harvests product links, converts them to absolute URLs, and queues them as new requests.

Before starting, confirm you have:

  • An Elixir toolchain with Mix available.
  • A project directory you can write to.
  • A target site whose HTML you are allowed to crawl, and whose terms and robots rules you have reviewed.
  • Selectors that match the current page markup — product pages change, so verify the price selector against a live response before trusting a run.

Where scraping fits

Scraping is useful when a site has no API or exposes only part of its data. Typical jobs include merging records from several sources into one dataset, watching competitor prices and stock levels, and assembling prospect lists from industry sites. The graphics-card example is a price-monitoring case: collect listed prices across products so they can be compared later.

Project setup

Create a supervised Mix project. The supervision tree matters because the spider will spawn and manage several processes.

The project generation command is shown in the setup command block. After the project exists, add Crawly and Floki to the dependency list in mix.exs; the exact dependency entries are in the dependency block. Then fetch them with the dependency-fetch commands. Crawly provides the spider framework; Floki parses and manipulates HTML so you can pull values out of a response.

Next, create a config directory and a config.exs file. The configuration block sets two middlewares: a UserAgent middleware with a browser-style user agent string, and a WriteToFile pipeline that writes results into /tmp with the jl extension. Sending a browser-like user agent reduces the chance of being refused, and rotating proxies or managed request services can help further when volume grows.

Spider callbacks you must implement

A Crawly spider is a behaviour with three callbacks:

  • init/0 runs once and returns the spider's starting state, including start_urls.
  • base_url/0 runs once and returns the site root; Crawly uses it to keep the spider on the target site and discard unrelated links.
  • parse_item/1 runs for each response and returns a Crawly.ParsedItem struct containing items and requests.

The first spider lives in lib/price_spider/spiders/basic_spider.ex and is shown in the basic spider block. Its parse_item/1 returns an empty ParsedItem, so the run proves the plumbing works without extracting anything yet.

Run it and read the output

Start an interactive session with iex -S mix and call the engine with your spider module, as in the start-spider command block. The debug log shows a manager, a request storage worker, and four workers starting for the spider.

What happens under the hood: Crawly schedules a request for each start URL, and each response is handed to parse_item/1. Because the basic spider has no parsing logic, it returns an empty ParsedItem. That is expected, not an error.

Extract a price with Floki

Fetch a product page directly with Crawly.fetch/1 to inspect the raw response, then parse the body with Floki.parse_document/1. The selector used here is .a-box-group span.a-price span.a-offscreen, applied with Floki.find/2 and read with Floki.text/1. In the captured run the extracted value was "$1,039.99".

Wire that into parse_item/1 so each item carries a price and the request URL, as in the completed basic spider block. Run the spider again with the same engine call. The WriteToFile pipeline writes a file named after the spider plus a timestamp into /tmp.

Field map for the parsed item:

  • items: one map per extracted record; here price and url.
  • requests: follow-up URLs to crawl; empty in the fixed-URL version.

If the run finishes but the file is missing, check the pipeline configuration and the folder path. If the file exists but prices are empty, the selector no longer matches the page — re-inspect the markup rather than assuming the spider failed.

Extend the spider to follow search results

The second spider, in lib/price_spider/spiders/amazon_spider.ex, starts from a search results URL instead of individual products. Its parse_item/1 does four things differently:

  • Collects product links with Floki.find("div.s-result-list a.a-link-normal") and Floki.attribute("href").
  • Turns each relative link into an absolute URL with URI.merge via the build_absolute_url/2 helper.
  • Converts each URL into a request with Crawly.Utils.request_from_url/1 and returns them in the requests field.
  • Adds a name field from span#productTitle alongside price and url.

Run it with the engine call for the Amazon spider. The output file then contains one JSON object per product, including name, price, and url. The captured sample rows show prices such as $1,786.99, $1,522.99, and $1,199.99 for specific graphics cards at the time of the original run; treat those figures as a dated reference and check current listings before relying on them.

What to add next

The two spiders cover the core loop: fetch, parse, queue, write. Crawly also offers request spoofing (simulating a different user agent or IP), item validation through pipelines, duplicate request and item filtering, concurrency control, and robots.txt enforcement. Add these as the job grows rather than all at once.

Crawling responsibly matters. Review the target site's terms and robots rules, keep request rates reasonable, and avoid collecting personal data you do not need. Crawly's documentation includes a section on ethical aspects of crawling that is a sensible starting reference.

Frequently asked questions

Do I need the --sup flag for a Crawly project?

Yes, in this walkthrough. The --sup flag generates an OTP skeleton with a supervision tree, which the price_spider project needs because it spawns and manages several processes.

Why does my spider return an empty Crawly.ParsedItem?

The basic spider's parse_item/1 is written to return empty items and requests. That is expected until you add Floki parsing logic. If you have added parsing and still get empty items, the CSS selector likely no longer matches the page.

What is the difference between items and requests in Crawly.ParsedItem?

items holds the records you extracted from the current response. requests holds follow-up URLs Crawly should crawl next. The search-results spider uses requests to queue product pages and items to store each product's name, price, and URL.

Where do the results get written?

The configuration in this walkthrough uses the WriteToFile pipeline with folder /tmp and extension jl. The file is named after the spider plus a timestamp.

{"url":"https://www.amazon.com/ASUS-Graphics-DisplayPort-Axial-tech-2-9-Slot/dp/B096L7M4XR/ref=sr_1_30?keywords=3080+video+card&qid=1650820592&rnid=2941120011&s=pc&sprefix=3080+video%2Caps%2C107&sr=1-30","price":"$1,786.99","name":"ASUS ROG Strix NVIDIA GeForce RTX 3080 Ti OC Edition Gaming Graphics Card (PCIe 4.0, 12GB GDDR6X, HDMI 2.1, DisplayPort 1.4a, Axial-tech Fan Design, 2.9-Slot, Super Alloy Power II, GPU Tweak II)"}
{"url":"https://www.amazon.com/ASUS-Graphics-DisplayPort-Military-Grade-Certification/dp/B099ZC8H3G/ref=sr_1_28?keywords=3080+video+card&qid=1650820592&rnid=2941120011&s=pc&sprefix=3080+video%2Caps%2C107&sr=1-28","price":"$1,522.99","name":"ASUS TUF Gaming NVIDIA GeForce RTX 3080 V2 OC Edition Graphics Card (PCIe 4.0, 10GB GDDR6X, LHR, HDMI 2.1, DisplayPort 1.4a, Dual Ball Fan Bearings, Military-Grade Certification, GPU Tweak II)"}
{"url":"https://www.amazon.com/GIGABYTE-Graphics-WINDFORCE-GV-N3080VISION-OC-10GD/dp/B098TZX3NT/ref=sr_1_23?keywords=3080+video+card&qid=1650820592&rnid=2941120011&s=pc&sprefix=3080+video%2Caps%2C107&sr=1-23","price":"$1,199.99","name":"GIGABYTE GeForce RTX 3080 Vision OC 10G (REV2.0) Graphics Card, 3X WINDFORCE Fans, LHR, 10GB 320-bit GDDR6X, GV-N3080VISION OC-10GD REV2.0 Video Card"}