Skip to content

Scrape a product catalog with Scrapy, from first selector to stored items

This guide builds two scrapers for the same kind of target: first, a single e-commerce product page, then a catalog-wide crawl. You will move from interactive selector testing to a spider, then add cleaning, validation, and optional database storage. The running example extracts a product URL, title, price, and image URL.

What Scrapy adds over a one-off script

Scrapy is an opinionated Python framework for scraping at scale. Unlike a single request-and-parse library, it supplies conventions for scheduling requests, following links, extracting fields, validating results, and exporting them. That structure helps larger crawls stay maintainable, though it also means a steeper initial learning curve than a one-off script.

Use this decision checklist before writing code:

  • Is the target one page or many pages? One page favors a basic spider; many pages require a discovery strategy.
  • Is there a sitemap? If yes, start there. If no, define link-following rules.
  • What fields define a useful record? Name them before choosing selectors.
  • How will records be cleaned and stored? Decide on loaders, pipelines, and an output format early.
  • Does the page need JavaScript rendering? If so, plain Scrapy may not see the data.

The example output for one product is a single JSON record containing the product URL, a price of 20.00$, the title Taba Cream, and an image URL. That small result is the contract every later component must preserve.

Set up the project and learn the file map

Create an isolated environment first, then install Scrapy and generate the project skeleton. The generated files have distinct jobs:

  • items.py defines the shape of extracted data.
  • middlewares.py can alter the request or response lifecycle, for example by rotating user agents or delegating requests.
  • pipelines.py processes, validates, or persists items.
  • settings.py holds project-wide configuration.
  • spiders/ contains classes that define crawling and parsing behavior.
  • scrapy.cfg holds main project settings.

See protected modules code0001, code0002, code0003, and code0004 for the exact setup commands and generated layout. Treat the virtual environment as part of the setup, not an optional extra, because it prevents conflicts with system packages.

Command example

pip install Scrapy

Test selectors in Scrapy Shell before writing the spider

The interactive shell is the fastest way to confirm that a page loads and that a selector returns the intended text. Fetch the product URL, inspect the status and headers, and optionally view the saved response in a browser. Note that the browser view may be incomplete because external resources and CORS behavior are not fully reproduced.

Two practical checks matter here. First, Scrapy may request robots.txt; a missing file can produce a 404, and an existing file is obeyed by default unless the relevant setting is changed. Second, test both XPath and CSS forms for the price so you can see that they return the same value. Protected modules code0005 through code0014 contain the exact shell configuration, fetch examples, response inspection, and selector tests.

A useful habit is to record three things for each field: the selector, the raw value, and the cleaned value. For price, the raw value includes a currency symbol; the cleaned value should be numeric. That distinction drives the loader and pipeline design below.

Python example

ROBOTSTXT_OBEY = False

Build a basic spider for one product

A spider starts from start_urls, turns them into requests, receives responses, and passes each response to parse. The parse method can return a dictionary, an item, a request for another callback, or an iterable. That flexibility supports multi-stage crawls: categories first, then pagination, then product pages.

For the single-product example, define a Product item with product_url, price, title, and img_url fields, then create a spider with a name, allowed domains, and start URLs. Its parse method populates the item from the response and returns it. Protected modules code0015, code0016, code0017, and code0018 show the exact item, spider generation, spider body, and JSON export command.

Choose the spider type by discovery method:

  • Basic Spider: use when you already know the URLs to fetch.
  • CrawlSpider: use when links must be followed according to rules.
  • SitemapSpider: use when a sitemap already lists URLs.
  • CSVFeedSpider or XMLFeedSpider: use when URLs come from tabular or XML feeds.

The basic spider is enough for one product. For a catalog, discovery becomes the main problem, which is addressed later.

Clean messy fields with item loaders

Real pages vary. A title may appear in a heading on one layout and in the document title on another, and prices may be formatted inconsistently. Item loaders address both problems by trying several selectors for one field and applying input or output processors to clean the result.

In the example, the title can be found through two XPath expressions. A loader tries them in order, while a default output processor keeps only the first match. A separate input processor strips the dollar sign from the price. Protected modules code0020 and code0021 show the exact loader usage and processor setup.

Use loaders when the main challenge is extraction variability or formatting. Use pipelines when the challenge is business validation or storage. That separation keeps parse methods shorter and makes failures easier to locate: a missing selector is an extraction issue, while an out-of-range price is a validation issue.

Validate and store items with pipelines

A pipeline receives each item through a process_item method. It can normalize a value, reject an item, or store it. In the example, the price pipeline removes non-numeric characters, converts the result to a float, and accepts only values within an arbitrary tutorial range. Items without a valid price are dropped. Protected modules code0022 and code0023 show the exact pipeline and its registration.

Pipeline order matters. Lower numeric priorities run earlier, so validation should run before storage. That ordering prevents a database from receiving records that the validator would have rejected.

For storage, the example adds a MySQL pipeline. It reads connection settings from the crawler, opens a connection when the spider starts, inserts each item, commits, and closes the connection when the spider finishes. Protected modules code0024 through code0029 cover the connector installation, database and table creation, settings, pipeline class, and registration. The database setup is assumed to exist already; only the database, table, and credentials need to be created or supplied.

Python example

ITEM_PIPELINES = {
	'product_scraper.pipelines.PriceValidatorPipeline': 100
}

Scale from one page to a catalog

Once one page works, scale discovery. First check for a sitemap, commonly at the site root. A sitemap may include home pages, articles, and products, so filter it. In the example, only URLs containing the products path are sent to the product parser. Protected modules code0030, code0031, and code0032 show the sitemap entries, filtered spider, and CSV export command.

If no sitemap exists, use a crawl spider. Start from the products listing page and define a rule that follows links matching the product pattern, sending each match to the product parser. Protected module code0033 shows the exact rule setup. This avoids manually maintaining a queue, tracking visited URLs, or managing threads.

Before a large run, keep rate limits and bot detection in mind. Some sites actively block scrapers, so a technically correct spider can still fail at the access layer. Plan for polite pacing, retries, and, where appropriate, a scraping service that handles proxies or browser rendering.

Conclusion

Scrapy is strongest when a crawl needs structure: defined items, repeatable parsing, validation, and export. Start with the shell to prove selectors, move to a basic spider for one page, add loaders for messy fields, and add pipelines only when validation or storage is required. For many pages, let a sitemap or crawl rules handle discovery instead of building that logic yourself.

The next areas to explore are middlewares, exporters, and extensions. They extend the same pattern used here: keep extraction, processing, and delivery as separate, testable stages.

Scrapy stage map: where to debug first
StageQuestion it answersFailure signal
ShellDoes the page load and does the selector match?Empty selector result or unexpected status
SpiderWhich URLs are fetched and which callback parses them?Wrong pages fetched or parse never called
Item loaderWhich selector wins and how is the value cleaned?List instead of single value, or currency symbol remains
PipelineIs the item valid and where should it go?Item dropped or invalid record stored
Discovery spiderHow are catalog URLs found?Sitemap includes irrelevant URLs or crawl follows wrong links

Use this map to decide which component to change when a crawl breaks.

Frequently asked questions

How do I set up a Scrapy project safely?

Use a dedicated virtual environment, install Scrapy, and generate a project with the startproject command. The generated layout separates items, middlewares, pipelines, settings, and spiders so each concern can evolve independently.

How should I test selectors before building a spider?

Open the interactive shell, fetch the page, inspect the response, and test XPath or CSS selectors against the live response before writing them into a spider. This catches selector and robots.txt issues early.

What is the difference between a spider, item loader, and pipeline?

A spider defines where to start and how to parse responses. Item loaders help try multiple selectors and clean values; pipelines validate, normalize, or store each item after extraction.

Which spider should I use for a whole catalog?

Prefer a sitemap spider when a sitemap lists the relevant URLs. When no sitemap exists, a crawl spider can follow links matching rules you define. Both avoid hand-rolling queues, duplicate tracking, and threading.

Why might Scrapy miss data on JavaScript-heavy pages?

Scrapy does not execute JavaScript by default, so pages that render data only after JavaScript may not expose it in the initial response. You will need a JavaScript-capable approach for those sites.

$ scrapy shell --nolog
[s] Available Scrapy objects:
[s]   scrapy     scrapy module (contains scrapy.Request, scrapy.Selector, etc)
[s]   crawler    <scrapy.crawler.Crawler object at 0x108147eb8>
[s]   item       {}
[s]   settings   <scrapy.settings.Settings object at 0x108d10978>
[s] Useful shortcuts:
[s]   fetch(url[, redirect=True]) Fetch URL and update local objects (by default, redirects are followed)
[s]   fetch(req)                  Fetch a scrapy.Request and update local objects
[s]   shelp()           Shell help (print this help)
[s]   view(response)    View response in a browser
In [1]:
[scrapy.core.engine] DEBUG: Crawled (404) <GET https://clever-lichterman-044f16.netlify.app/robots.txt> (referer: None)
[scrapy.core.engine] DEBUG: Crawled (200) <GET https://clever-lichterman-044f16.netlify.app/products/taba-cream.1/> (referer: None)
>>> response.status
200

>>> response.headers
{b'Content-Length': [b'3952'], b'Age': [b'100'], b'Cache-Control': [b'public, max-age=0, must-revalidate'], b'Content-Type': [b'text/html; charset=UTF-8'], b'Etag': [b'"24912a02a23e2dcc54a7013e6cd7f48b-ssl-df"'], b'Server': [b'Netlify'], b'Strict-Transport-Security': [b'max-age=31536000; includeSubDomains; preload'], b'Vary': [b'Accept-Encoding']}
[
  {
    "product_url": "https://clever-lichterman-044f16.netlify.app/products/taba-cream.1/",
    "price": "20.00$",
    "title": "Taba Cream",
    "img_url": "https://clever-lichterman-044f16.netlify.app/images/products/product-2.png"
  }
]