Scrape Walmart Product Pages with Python
Walmart product pages are useful for two common jobs: watching a price and watching stock. The page you see in a browser and the HTML your script receives are not always the same document, though, and that difference is the main thing this guide works through. You will fetch a product page with Python, extract four fields, and see why one of those fields has to come from an embedded JSON payload rather than a visible tag. A short decision table then helps you choose between a plain script and a managed scraping API when blocking starts to interfere.
What you will build
The target is a small Python script that takes a Walmart product URL and returns four values: the product name, the current price, the image URLs, and the description text. The example product used throughout is an HP 11.6-inch Chromebook with the Walmart item URL ending in 592161882. The same approach applies to any other product page once you adjust the URL.
Before writing code, decide which fields you actually need. Price and stock checks usually need only the price and availability signals, while catalog work needs the name, images and description as well. Extracting fewer fields means fewer selectors to maintain when the page changes.
Fetch the product page with requests
Create a working folder and a file named walmart_scraper.py, then install the two libraries used here: requests for the HTTP call and beautifulsoup4 for parsing. The exact shell and install commands are preserved in the accompanying code modules.
A minimal fetch sends a GET request to the product URL and prints the response text. If the response begins with a page titled "Robot or human?", Walmart has served a captcha instead of the product. That is a blocking response, not a parsing bug, and no amount of selector tuning will fix it.
The first mitigation is to send a browser-like User-Agent header with the request. You can inspect the headers your own browser sends through its developer tools network panel and copy the User-Agent value into a dictionary passed to requests.get. This sometimes returns the real product HTML, but it is not a durable answer: a header alone does not defeat a system built to detect scripted traffic.
Install the two Python dependencies before running the examples.
python -m pip install requests beautifulsoup4Set WALMART_PRODUCT_URL to the product URL you want to inspect. This saves the returned HTML; a successful HTTP request can still return a challenge page.
from pathlib import Path
import os
import requests
product_url = os.environ["WALMART_PRODUCT_URL"]
browser_header = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:104.0) Gecko/20100101 Firefox/104.0"
page_response = requests.get(product_url, headers={"User-Agent": browser_header}, timeout=30)
page_response.raise_for_status()
Path("product.html").write_text(page_response.text, encoding="utf-8")Field map: where each value comes from
The table below is a reader aid, not a reproduction of the page markup. It records which selector or JSON path produced each field in the worked example, so you can re-check the page when a value stops appearing.
Extract the name, price and images
Parse the response with BeautifulSoup, then locate the product name in an h1 element whose itemprop attribute is name. On the example page this returns the string HP 11.6" Chromebook, AMD A4, 4GB RAM, 32GB Storage, Black 16W64UT#ABA.
For images, find every div with the data-testid attribute set to media-thumbnail, then read the src of the nested img tag that has loading set to lazy. Collect those URLs into a list. For price, find the span whose itemprop attribute is price and read its text.
The general method is the same each time: open the product page in a browser, right-click the value, choose inspect, and identify the nearest stable attribute that isolates it. Prefer semantic attributes such as itemprop and data-testid over generated class names, which change more often.
Run this against the saved product.html. These selectors describe the documented page structure; missing values are a reason to inspect the HTML, not proof the product has no name, price or images.
from pathlib import Path
from bs4 import BeautifulSoup
document = BeautifulSoup(Path("product.html").read_text(encoding="utf-8"), "html.parser")
name_node = document.select_one("h1[itemprop='name']")
price_node = document.select_one("span[itemprop='price']")
image_nodes = document.select("div[data-testid='media-thumbnail'] img[loading='lazy']")
product_fields = {
"name": name_node.get_text(strip=True) if name_node else None,
"price": price_node.get_text(strip=True) if price_node else None,
"images": [node["src"] for node in image_nodes if node.has_attr("src")],
}
print(product_fields)Why the description needs __NEXT_DATA__
The description looks easy in the inspector: it sits inside div elements with a class of dangerous-html. Search the HTML your script received, though, and that class is absent. The inspector is showing a processed DOM after JavaScript has run, not the raw response.
Walmart builds its pages with Next.js and ships the displayed information as a large JSON object inside a script tag with the id __NEXT_DATA__. Parse that script's text with json.loads, then read the short description from props.pageProps.initialData.data.product.shortDescription and the long description from props.pageProps.initialData.data.idml.longDescription. Both values arrive as HTML fragments, so run each through BeautifulSoup once more to get plain text.
The same JSON object also holds the name, the price string and the image info, so once you have it parsed you can source every field from one place. That is often more stable than mixing tag selectors with JSON paths.
Continue from the parsed document in the preceding example. Read the embedded JSON as data; do not execute it. The documented nested keys must be checked again if the page structure changes.
import json
from bs4 import BeautifulSoup
state_node = document.select_one("script[id='__NEXT_DATA__']")
if state_node is None:
raise ValueError("The saved page has no expected product-state script")
state_data = json.loads(state_node.get_text())
product_data = state_data["props"]["pageProps"]["initialData"]["data"]
short_html = product_data["product"]["shortDescription"]
long_html = product_data["idml"]["longDescription"]
descriptions = {
"short": BeautifulSoup(short_html, "html.parser").get_text(" ", strip=True),
"long": BeautifulSoup(long_html, "html.parser").get_text(" ", strip=True),
}
print(descriptions)Plain script or managed scraping API?
For a one-off check, a plain requests script is usually enough. The trouble starts when the same script runs on a timer: repeated automated hits tend to draw blocks, and swapping the User-Agent string alone rarely clears them. Getting past that stage means adding proxy rotation and captcha handling, which is its own upkeep.
For recurring work, a managed API is another way to organize proxy rotation, captcha handling and rendering. Test those capabilities against your request volume and the responses you receive. Its extract rules let you declare selectors and outputs, and the request can run through a real headless Chrome instance so Next.js finishes rendering before the result comes back. Here the extract rules still need post-processing, because the description lives inside a script tag rather than in ordinary markup. The managed route pays off most when the data you want already sits in standard HTML tags.
Failure checklist
When a field comes back empty, identify which of these three situations you are in before changing code.
- Captcha response: the returned HTML starts with a page titled "Robot or human?". The request was blocked. Adjust headers, slow down, or move to a managed API.
- Selector miss: the page loaded but the h1, span or div you targeted is absent. The markup changed; re-inspect and update the attribute.
- JSON miss: the __NEXT_DATA__ script is missing or a key path has moved. Print the parsed JSON keys before indexing into them, and confirm the script tag exists in the raw response.
Keeping these three apart saves time, because each has a different fix and only one of them is about parsing.
Scaling beyond a single page
Once the single-page script works, the next step is usually a list of product URLs and a schedule. A framework such as Scrapy adds pause and resume, data filtering, proxy rotation, multiple output formats and remote operation. It can also be combined with a managed scraping API so that captcha responses do not stall a crawl. Start with the single-page version, confirm the four fields are correct, and only then add the scheduling layer.
| Field | Where it comes from | Notes |
|---|---|---|
| Name | h1 element with itemprop="name" | Returns the full product title as text |
| Price | span element with itemprop="price" | Read the element text |
| Images | div with data-testid="media-thumbnail", nested img with loading="lazy" | Collect the src of each image into a list |
| Short description | props.pageProps.initialData.data.product.shortDescription | HTML fragment; parse again for plain text |
| Long description | props.pageProps.initialData.data.idml.longDescription | HTML fragment; parse again for plain text |
| Name, price, images (JSON route) | props.pageProps.initialData.data.product.name / priceInfo.currentPrice.priceString / imageInfo | Alternative single-source path once __NEXT_DATA__ is parsed |
Selectors and JSON paths reflect the example product page at the time the walkthrough was written. Walmart's markup and payload keys can change; re-inspect before relying on any path.
Frequently asked questions
Why does my script get a page titled "Robot or human?" instead of the product?
That title is the captcha page. Walmart detected the request as automated and refused to serve the product. Sending a browser-like User-Agent sometimes helps, but repeated scripted requests will be blocked again; rotating proxies and captcha solving are the usual next step.
The description class exists in the browser inspector but not in my HTML. Why?
The inspector shows the DOM after JavaScript has run. The raw response instead carries the data in a script tag with the id __NEXT_DATA__, as a JSON object. Parse that script and read the description from the product and idml keys.
Can I get the name and price from the JSON instead of selectors?
Yes. The same parsed object exposes product.name, product.priceInfo.currentPrice.priceString and product.imageInfo, so a single JSON source can supply every field.
Do I need a scraping API for a single product page?
Not necessarily. A plain requests script with a User-Agent header is enough for occasional checks. A managed API becomes worthwhile when you need scheduled runs, because it handles proxy rotation and captcha solving for you.