Competitor Price Tracking With Web Scraping: A Python Build Guide
Manual price checks do not keep up with stores that reprice several times a day. This guide walks through a Python pipeline that fetches competitor product pages, pulls out price and stock, appends a snapshot on every run, and raises a Slack or email alert when a competitor genuinely undercuts you. It covers Amazon and Walmart through dedicated parsers, other retailers through a generic HTML fetch, and the extraction choices in between.
What a competitor price tracker actually does
A price tracker is more than a one-off scrape. It watches a defined set of competitor product pages on a schedule, records the price, stock status and any promotion, keeps that record so changes become visible, and acts when a number crosses a line you set.
Four recurring obstacles stand between you and that record, and all of them rest on a single prerequisite: a correct pairing between each competitor listing and your own SKU, so that every comparison is genuinely like-for-like. The mapping step below handles that prerequisite.
- Anti-bot defenses. Large retailers and many Shopify or WooCommerce stores sit behind bot-management systems. Datacenter IPs are only part of the exposure; TLS fingerprints from plain Python clients can be flagged before the page loads, so rotating IPs alone does not fix a fingerprint problem.
- JavaScript rendering. Prices are often written into the page client-side after the initial HTML arrives, so parsing raw HTML can return a page with no price in it.
- Selector drift. A selector that worked in March can return nothing in June after a redesign. The scraper does not crash; it logs nulls and alerts go quiet.
- Scheduling, storage and alerting. A script run once is not a tracker. A tracker runs on a schedule, keeps history, and notifies only when something changes.
Those four map onto one pipeline: map SKUs, fetch, extract, store, alert. The guide builds the deterministic version you own, because it is easier to debug and it produces a price history. An agent-driven alternative exists through a hosted MCP server, but the pipeline is the controllable starting point.
Step 1: Map competitor products to your SKUs
Start with a flat CSV that ties each competitor product to one of your own SKUs. This unglamorous step largely decides whether your alerts are signal or noise.
Four columns are enough: your SKU, the source (which decides how you fetch it), the identifier, and your current price so the tracker can compare. For marketplaces, store the stable identifier rather than the full URL. Amazon ASINs and Walmart item IDs survive slug and title changes; the pretty URL does not.
Map at the variant level, not the product name. A search for a wireless mouse can return two Logitech models that share a brand and category but sit at different price points. Comparing them directly is a bug. Match on SKU, GTIN or model number, and keep a match-confidence flag if your catalog is fuzzy. A GTIN-first, then model-code, then fuzzy-title order works because near-identical titles can hide different products.
Step 2: Fetch the product page
Once products are mapped, the next job is fetching the pages. There are two paths: a generic HTML fetch for most retailers, and dedicated parsers for Amazon and Walmart that skip HTML parsing entirely.
For an arbitrary retailer, route the request through a scraping API that runs a headless browser and a managed proxy pool. Your own client only talks to the API endpoint, so the retailer sees a browser-grade request rather than your Python TLS signature. That removes the fingerprint problem from your side, though the automation itself can still be detected by systems that watch for browser-driving calls.
The non-marketplace examples in this guide use public practice stores so they can be run as-is. Real competitors will have messier markup, missing structured data, regional variants and tougher defenses.
Proxy tiers should be chosen per target rather than always reaching for the strongest. JavaScript rendering is on by default and handles client-side prices. A premium residential tier covers most sites behind Cloudflare or DataDome. A stealth tier uses a separate pool for the hardest targets. Prices and availability are often region-specific, so geotargeting matters for accurate tracking.
Some prices appear only after a delay or an interaction, such as selecting a variant or dismissing a region modal. A wait-for-selector step blocks until an element is in the DOM, and a JavaScript scenario can run a sequence of browser actions first: wait for a selector, scroll, wait again. Scenarios also support click, fill and infinite scroll, enough to reach a price behind a size picker or a see-price-in-cart button. They run in the headless browser at the normal rendering rate.
Only publicly accessible product pages are in scope, meaning pages visible in an incognito window without logging in. Scraping behind login credentials is prohibited by the provider's terms.
Credit cost by proxy configuration
Credit cost scales with the proxy tier, and the gap is large enough that the choice is a budget decision. The table below is the captured per-request credit schedule; treat it as a dated reference and confirm current numbers before sizing a plan. A request is billed when it succeeds (including a target 404, since a discontinued product still returns a page), so a blocked 403, 429 or 5xx fetch does not consume credits.
At scale this is the line that matters. Tracking roughly 500 SKUs once a day lands in the range of 75k to 225k credits a month depending on tier. The free allotment is for testing, not for running a catalog.
Amazon and Walmart: use the dedicated parsers
Amazon and Walmart are the two targets where the generic path is the wrong tool. Both run aggressive defenses, render key fields client-side, and restructure HTML often. Dedicated parsers return structured JSON in one call, with no HTML parsing and no selectors to maintain.
For Amazon, the product is passed as a query holding the 10-character ASIN, not as a url or asin parameter. The response is parsed JSON trimmed to the fields a tracker cares about: price, price_strikethrough (the list price the discount is measured against), currency and stock. The full payload also carries images, delivery, category, sales rank, variations, featured merchant and a product-details block, but tracking usually needs only the price and stock fields.
Walmart works the same way with the item ID passed as product_id. Its response shape is flatter than Amazon's. Amazon reports stock as a string such as In Stock, while Walmart uses an out_of_stock boolean. Amazon uses reviews_count, Walmart uses rating_count. The parsers save selector maintenance, but you still normalize their fields into one shape.
A separate Amazon pricing endpoint returns the offers on a listing, each with condition, seller and price, so you can see the full competitive picture when third parties and Amazon itself compete on one ASIN. It is keyed by asin rather than query.
Check that the offer is comparable before you trust the number
A marketplace listing's headline price is whatever currently wins the Buy Box, which can be a used unit, a third-party seller, or a different-size variant than the one you sell. The M185 example shows the trap: its Buy Box also carries a Used - Very Good offer at $7.72, and the product has a 4-pack variant under a different ASIN. Comparing your new single unit against either produces a false undercut alert.
Before trusting a number, pin the condition to New, decide whether you accept any third-party seller or only the first-party offer, confirm the variant is the single unit you sell, and store the seller and condition next to the price so a bad comparison is auditable instead of invisible. A used unit at $7.72 is not an undercut of your new $14.99.
Watch for store- and region-specific pricing too. Walmart serves different prices by store. Walmart exposes store_id to pin a specific store for like-for-like comparison, but it was slow and only intermittently returned a price in testing, so verify it works for your targets before relying on it. Amazon exposes domain for international tracking. Two more parameters come with caveats: Walmart's delivery_zip returns empty fields (prefer store_id), and Amazon's zip_code does not reliably move the price, so confirm a location parameter changes the number before using it.
Choosing a proxy tier per target
The right tier depends on the target's defenses. The table below maps common cases to a method and a credit cost; it is a dated reference, so confirm current pricing before committing a budget.
Pulling an Amazon page through the stealth tier and hand-parsing it burns 75 credits per request and still leaves you owning selector upkeep against a site that reshuffles its markup. The dedicated parser costs a fraction of that and returns already-parsed JSON. When the target is Amazon or Walmart, make the dedicated parser the default.
Dedicated parsers exist for Amazon, Walmart, Google, YouTube and ChatGPT, plus a fast search endpoint. For every other retailer you are on the generic path, which means extracting the price yourself.
Step 3: Extract the price
Once you have HTML from the generic path, you need the price out of it. The approach depends on how many distinct sites you track.
CSS and XPath selectors are the traditional route and can run server-side with an extraction-rules parameter, with no extra credits and no BeautifulSoup needed for simple cases. A more durable selector-based approach is JSON-LD: most retailers embed a schema.org/Product block for search engines, and that structured data changes far less often than the visual layout. Retailers often nest the Product node inside an @graph array alongside breadcrumb and organization nodes, so walk the graph rather than assuming the top-level object is the product. Some sites list @type as an array, so handle both forms.
Selector drift is the quiet killer. A selector keeps working until a redesign renames the class it targets; then it matches nothing, no exception is thrown, and a null lands in your history. Writing selectors is quick, but keeping them correct is not, and the effort grows with every additional site you monitor.
The alternative is AI extraction with a typed schema, where a markup change does not break the field the way a selector would. The schema supports string, number, boolean, list and item types, so a price can come back as a number rather than a string with a currency symbol to strip. AI extraction adds credits per request.
AI extraction is not deterministic. In testing, price and in-stock stayed stable on a clean product page, but currency flipped between the clean ISO code and the bare symbol from one run to the next on the same request and page. On busier pages it degrades further: sometimes only the title comes back and the price drops entirely, or in-stock reads true for a product whose own JSON-LD says out of stock. A query description does not reliably help and can make the currency field worse.
There is also an outright failure mode: on some pages it frequently returns HTTP 200 with a plain-text sorry message instead of JSON, and still bills the full credits. That is why a real tracker retries and raises rather than trusting the JSON parse.
One knob does help. An AI selector scopes extraction to a slice of the page instead of the whole DOM. Pointing it at the product summary stabilized the currency in testing. The catch is that it is only as good as the selector you give it; point it at a block that is not there and the call fails outright. It improves AI extraction but does not make it deterministic, so keep retry-and-validate.
Use the dedicated parsers for Amazon and Walmart: they are deterministic and cheaper. For the long tail of small retailers where a selector per site is not worth maintaining, AI extraction is lower-maintenance, but validate the output by rejecting nulls and sanity-checking against the previous snapshot. Where a site exposes clean JSON-LD, that is often the most reliable of the three.
In a real tracker you do not choose once. A generic fetcher cascades them in order of reliability: a per-site selector if set, then JSON-LD, then a price meta tag, with AI extraction as the fallback, validating whatever comes back.
Telling failures apart
The three extraction paths fail in different ways, and knowing which one you are looking at shortens debugging.
- Selector drift fails silently. The selector returns nothing, the tracker logs a null, and no error is raised. A change-based check never fires because the wrong value is consistently wrong.
- HTTP errors fail loudly. A 401, 403 or 422 returns an error body rather than a price, so raise on the status code instead of parsing it as data.
- AI extraction can fail deceptively. It may return HTTP 200 with a non-JSON body and still bill the request. Retry a bounded number of times, then raise so one bad row does not take down the whole run.
A useful guard is to alert on coverage loss, not just log it. Fire an alert when a share of your targets stop returning a price in a single run, and flag any price more than roughly 50% different from the previous snapshot. Those are almost always a broken selector or a misread rather than a real price move. A dead-man's switch keeps a tracker that goes silent from failing unnoticed for months, though a value that is consistently wrong but present will not trip it. Reconciling the first run by hand catches that.
Step 4: Schedule, store snapshots, and alert
A tracker is a scraper that runs on a schedule, keeps history, and tells you when something moved.
Scheduling options run from most infrastructure to least. Cron on a small VM is fine if you already run a box; one crontab line runs it every six hours, and the working directory matters because the tracker reads and writes its history file relative to it. GitHub Actions is free for public repos with no server to manage, but the runner is ephemeral, so unless you commit the history file back at the end of the job, every run starts from a blank history and change detection never works. That is the git-scraping pattern. Serverless cron on Cloudflare Workers, Vercel Cron or Modal runs the tracker with no box to patch and is usually lower friction than a long-lived VM. No-code schedulers using native n8n, Zapier or Make connectors call the API on a schedule and drop results into a sheet or Slack channel, which suits a non-engineer owner.
For storage, append-only flat files handle more than you would expect. Appending each run's snapshots to CSV, JSONL or Parquet gives you price history. Daily snapshots of a hundred SKUs accumulate to roughly 100,000 rows over three years, which a single flat file still absorbs comfortably. A database makes sense when you outgrow flat files or want concurrent writers; SQLite is fine for a single-writer cron job and Postgres when you have several. The tracker does not care which, so keep this part stack-agnostic.
You do not need a database server to analyze history. DuckDB runs SQL directly over CSV or Parquet files. Start with current standings and reduce to the latest snapshot per SKU first; skip that step and you rank every historical row instead of today's price. The reason to keep history is the questions a single fetch cannot answer, and they are window functions rather than more scraping. Who repriced since we last looked is one query, and it is the same shape as a sale detector, a repricing-cadence report, or a first-time-below-us alert.
Alerting is what separates a live tracker from a passive archive. Each run, line the newest competitor price up against your own, and when it falls under yours by more than a set margin, push a message to a Slack incoming webhook. A 5% default margin absorbs cent-level rounding so you are not pinged over noise. As written, that fires on every run where a competitor is under your threshold. To page only on a new undercut, compare each price to that SKU's previous snapshot before posting. Add the in-stock and same-offer checks too, so a competitor that has been cheaper all week, or whose Buy Box is a used or multipack unit, does not trigger a false alert.
Step 5: Assemble the working tracker
The full path runs from a list of competitor targets to an alert when one undercuts you. It reads the targets file, dispatches each row to the right endpoint, normalizes every response into one typed snapshot, appends history, and flags undercuts.
A worked example using the three-row targets file produced this output with live prices:
- MOUSE-001, amazon, USD 13.99 against our 14.99: undercut 6.7%
- MOUSE-002, walmart, USD 13.83 against our 14.99: undercut 7.7%
- HOODIE-009, generic, USD 69.0 against our 75.0: undercut 8.0%
Each row gets one typed snapshot and an undercut flag. The generic row succeeded because the generic fetcher retries; AI extraction hit its billed failure mode on most attempts for that row. On an unlucky run it exhausts its retries and the tracker skips that SKU instead of crashing, so a bad fetch costs one row rather than the whole run.
Do not read the marketplace numbers as fixed. Across calls, Walmart returned $13.83, $13.52 and $9.88 for the same item ID because it serves variable pricing. The parser is reliable; the price is not.
Before trusting the first run, open two or three of the scraped pages in a browser and check the number matches, especially generic-path SKUs. A drift guard catches a price that moves but not a selector or AI that grabbed a related-items, per-unit or subscription price on run one and then returns the same wrong number every run after. A consistently wrong extraction never trips a change-based check; a human looking at that first run is what catches it. Reconcile a few by hand, then schedule it and let it run.
One-at-a-time fetching is adequate for a handful of rows but becomes the bottleneck at catalog scale, because every call spends seconds waiting on the network. Spread the work across a thread pool sized to your plan's concurrency limit. In testing, five Amazon product requests sent one at a time took about 20 seconds, while the same five sent concurrently finished in about 6 seconds, roughly 3x faster. The speedup grows as you raise the worker count toward your plan's concurrency cap, so stay under that cap because the API returns HTTP 429 when you go over it.
Where the data goes next: agents and MCP
A Slack ping is the simplest action and often enough. But the snapshot table often feeds something automated: a repricing agent that drafts a recommendation, an anomaly check that separates a real price move from a scrape error, or a tool an assistant calls when someone asks who is undercutting us this week. The structured JSON the dedicated parsers and AI extraction return drops into a prompt, a function call or a vector store with no HTML-parsing layer in between.
That has two halves. First, a hosted MCP server lets an agent do the scraping itself. Connect it to the endpoint and it gets tools for fetching HTML, screenshots and files, plus the dedicated Amazon, Walmart and Google extractors, with nothing to deploy. Second, you can publish your own tracker's history as a tool in the same style, letting the assistant answer questions about data you have already gathered. An undercuts tool and a recent_changes tool both return live rows from the same file the tracker writes, with no dashboard and no glue layer.
The tracker is the data layer, and what reads it is up to your stack. Where this guide deliberately stops is the action. Turning a price signal into an automated price change is repricing logic involving margin, minimum advertised price and brand. It stops at a trustworthy signal, and the decision stays yours.
Pitfalls and habits that keep a tracker alive
A few habits separate a tracker that runs for a year from one that rots in three months.
- Throttle your own requests. Even when the API handles concurrency, put a short delay between requests and do not refresh more often than your repricing decisions need. Scraping politely helps keep you off blocklists.
- Normalize currency and number formats, and do not trust the dollar sign. A European price string of 1.299,00 is not 1.299. Strip symbols, normalize the decimal separator, and store a float plus a currency code. A bare dollar sign could be USD, CAD, AUD or MXN, so compare only within the same currency and prefer an explicit code over a guessed symbol. The dedicated Amazon and Walmart parsers already return numeric prices and an explicit currency code; selector and AI paths need this step.
- Settle on one price basis and write it down. Product pages usually expose a live price and a crossed-out list price side by side (Amazon returns price and price_strikethrough). Choose the one your team actually reprices against, document the choice, and avoid quietly mixing the two in the same history.
- Read personalized prices as a benchmark, not an exact figure. Certain retailers tailor prices to an account or session, so the anonymous public price you collect may differ from what a signed-in shopper is shown. That is normally acceptable: the anonymous price is the correct competitive reference and the only one the provider's terms let you gather.
- Monitor the monitor. Alert on coverage loss, not just log it. Log loudly when extraction returns nothing, or when a price is more than roughly 50% different from the previous snapshot. The worst failure is the silent one, where a subset of SKUs returns null for weeks while a dashboard sits flat.
- Keep proof for MAP monitoring. When you are a brand policing resellers for minimum advertised price breaches, the tracker doubles as a compliance record, so save proof next to each figure. A screenshot parameter stores the page image as evidence.
- Scrape only public product pages, never behind a login or past an access control.
A second surface: how AI assistants price you
Your tracker watches competitor product pages, but that is not the only place your price shows up. A growing number of buyers ask an assistant what the cheapest option is before opening a product page, or instead of it. So there is a second thing to watch: how AI assistants price and rank you. An assistant can quote a stale price, miss you entirely, or name a competitor as cheapest. That is a different signal from product-page scraping. The product page is the ground truth you reprice against; the AI surface is what the buyer sees.
A ChatGPT endpoint with live search captures that surface in a single call. Run it across your products and competitors, diff the answers over time, and alert when the assistant drops you, quotes a stale price, or ranks a competitor cheapest. The answer is non-deterministic and geo-noisy: the same query can return US prices one time and off-geo prices the next, even with a country code set. That inconsistency is what buyers get, which is why it is worth watching. It is also heavy, costing roughly 15 credits and running slowly, so run it daily or weekly rather than on the six-hour price cron.
Build vs buy: when to skip the build
Building wins at small-to-mid scale. Buying wins at large scale with no engineering time to spare.
Build when you are tracking up to a few hundred SKUs, want control over alerting and thresholds, or want the snapshots in your own data warehouse. Subscription pricing for commercial tools typically ranges from the low hundreds to low thousands per month depending on vendor and SKU volume, a recurring bill that usually costs more than the initial build plus ongoing API credits and occasional selector upkeep. Buy when you are tracking thousands of SKUs across hundreds of competitors, need a polished UI for non-technical buyers, and have no one to maintain a scraper. Tools such as Prisync, Skuuudle and Competera package this kind of price scraping behind a UI and a support contract, a fit for a different kind of buyer.
A scraping API belongs to the build camp. Its HTML API, AI extraction and dedicated parsers handle the same fetch-and-parse work those packaged tools bundle, while you retain control of everything above the fetch layer.
Where this is heading
The pipeline above is today's baseline, not the end state. Three shifts are worth building toward.
The architecture is going agent-shaped. With a hosted MCP server wired in, an agent can do the scraping and query your price history through tools. The shift ahead is the rest of the loop, compare, decide and alert, becoming an agent's job too rather than a hand-wired pipeline a person maintains. Alongside build and buy, delegating the whole loop to a computer-use agent is now a third option. If you build now, expose the pieces as tools, not just a script, so an agent can drive them later.
Bots are starting to identify themselves, not just hide. The anti-bot arms race is only half the story. The other half is Web Bot Auth, an IETF draft built on HTTP message signatures, where an agent signs each request with a cryptographic key and the site verifies it. Adoption has moved quickly: card networks announced support for agentic commerce, a major cloud firewall added support, and a major CDN says it validates it at its edge. It pairs with the pay-per-crawl idea to suggest a web where legitimate crawlers authenticate and pay for access. Today you still need browser-grade fingerprints; medium-term, an identified-and-paid lane is likely to matter.
Regulators moved from the scraping side to the pricing side. The live legal question is shifting from can I scrape public prices, which is more settled than it once was, to is my algorithmic pricing legal. The FTC's surveillance-pricing study and New York's algorithmic-pricing disclosure law target personalized pricing, setting an individual's price from their own data, not competitor price-matching, which is generally treated as lawful. But if your tracker feeds an automated repricer, that is the side regulators are now watching, so keep the why-this-price-changed trail your snapshots already give you.
| Configuration | Credits per request |
|---|---|
| Classic proxy, no JS (render_js=false) | 1 |
| Classic proxy + JS (default) | 5 |
| Premium proxy + JS | 25 |
| Stealth proxy | 75 |
| AI extraction add-on (ai_extract_rules) | +5 |
Dated reference from the captured source; verify current credit costs before sizing a plan.
Frequently asked questions
Is web scraping legal for tracking competitor prices?
Collecting publicly visible pricing data is broadly allowed in the US and EU provided you honor robots.txt, avoid circumventing technical access controls, and stay out of login-gated pages. Whether you may republish what you collect is a distinct legal matter. For your own situation, get advice from a lawyer.
How often should I scrape competitor prices?
Match the frequency to your own repricing decisions, not to the competitor. Many retailers now run algorithmic repricers that can move prices multiple times a day, so daily has shifted toward a few times a day for competitive categories like electronics. Fast-moving ones such as flights, hotels and flash sales justify hourly. But more frequent scraping costs more credits and only matters if you can act on it. There is no point scraping hourly if you reprice weekly.
Can I scrape Amazon and Walmart prices?
Yes, using dedicated Amazon and Walmart APIs that return parsed JSON. The Amazon product endpoint costs 5 credits in light mode and 15 with full rendering. Walmart is 10 light and 15 full. Both target only publicly accessible product pages, with no logged-in or business-account-only pricing.
What is the best tool to track competitor prices?
The deciding factor is engineering capacity. What you pay a vendor for is largely the interface and the support agreement, not the underlying fetching. If an engineer is available, a Python script paired with a scraping API tends to be the cheapest route that still scales. If not, a subscription tool supplies a polished UI and a maintainer.
How do I deal with sites that block scrapers?
Route requests through a scraping API that uses residential or stealth proxies plus a headless browser, so anti-bot fingerprinting is solved on the provider's side rather than yours. A premium proxy covers most Cloudflare, DataDome and HUMAN sites, while the stealth tier is reserved for the toughest. For Amazon and Walmart, the dedicated parsers handle those defenses for 5 to 15 credits.
How do I store and visualize the scraped price data?
Append snapshots to a CSV, Parquet file, or a table keyed by timestamp, SKU, competitor and price, then query it with DuckDB straight over those files, no warehouse needed. Chart it with whatever your stack already has, or hand the file to an LLM to summarize the week's moves.
Can I do this without writing code?
Yes. Built-in n8n, Make and Zapier connectors plug the API into a no-code workflow. You trade some flexibility for faster setup.
{
"asin": "B004YAVF8I",
"title": "Logitech M185 Wireless Mouse, 2.4GHz with USB Mini Receiver ...",
"brand": "Logitech",
"price": 13.99,
"price_buybox": 13.99,
"price_strikethrough": 17.99,
"currency": "USD",
"stock": "In Stock",
"rating": 4.5,
"reviews_count": 42635,
"is_prime": true
}
{
"id": "15545702811",
"title": "Logitech Silent Wireless Mouse, Blue/Gray, Walmart Exclusive",
"price": 13.83,
"currency": "USD",
"out_of_stock": false,
"rating": 4.5,
"rating_count": 4844,
"seller_name": "Walmart.com",
"gtin": "...",
"sku": "..."
}
claude mcp add --transport http scrapingbee "{{MCP_BASE_URL}}/mcp?api_key=YOUR-API-KEY"
| Configuration | Credits per request |
|---|---|
| Classic proxy, no JS ( render_js=false ) | 1 |
| Classic proxy + JS (default) | 5 |
| Premium proxy + JS | 25 |
| Stealth proxy | 75 |
| AI extraction add-on ( ai_extract_rules ) | +5 |
| Target | Use Case/Method | Credits |
|---|---|---|
| Independent / smaller retailer | Classic + JS | 5 |
| Shopify / WooCommerce store behind Cloudflare | Premium + JS | 25 |
| Amazon or Walmart | Dedicated Amazon / Walmart API | 5 / 10 (light) |
| Hardest anti-bot, no dedicated parser | Stealth | 75 |
Dated reference from the captured source; confirm current pricing and tier behavior for your targets.