Scrape Zillow Real Estate Data at Scale with Python
This guide builds a Python workflow for collecting Zillow search-result cards and enriching each property with detail-page information. It covers inspecting page structure, paging through results, handling missing fields, and saving a clean JSON dataset for market research.
What the dataset is for
Zillow listing pages are useful beyond a single purchase decision. Repeated, structured snapshots can show how asking prices, inventory mix and property characteristics change across neighbourhoods. The same fields can feed a market dashboard, a valuation comparison, or an internal property application.
Use this workflow when you need repeatable collection rather than one-off copying:
- Start from a city or ZIP-code search URL
- Walk result pages in order
- Parse each property card
- Visit the detail page for estimate and history fields
- Write one record per listing to JSON
- Re-check selectors after any site layout change
Inspect the page before writing selectors
Open a search results page, inspect a property card in the browser developer tools, and note which tags carry each field. In the captured structure, address, price, bedroom count, bathroom count, square footage, status and listing agent are exposed through card-level attributes and classes, while the detail page holds the estimate and history table.
Treat selectors as fragile. Layout and class names change over time, so a scraper that worked previously can silently return empty fields. A quick manual inspection before a large run saves a failed collection.
Decision check before coding:
- Confirm the base search URL for the target city or ZIP code
- Confirm the page-number pattern for page two and beyond
- Confirm each card field still maps to a visible element
- Decide which fields are required and which may legitimately be missing
Setup and request handling
The project uses Python with BeautifulSoup and the lxml parser for HTML parsing, plus a managed scraping API client for requests. Install the parser dependencies first, using protected module code0002, then configure the API client with {{OWNER_SCRAPING_API_KEY}}.
Anti-scraping controls described for this kind of site include CAPTCHA challenges, IP-based blocking and hidden honeypot elements. A managed API can rotate proxies and assist with CAPTCHA handling, while request options such as stealth_proxy and wait_browser help requests use a proxy pool and wait for the page to finish loading. Those options are shown in protected module code0007.
Failure distinction worth keeping clear: a non-200 response means the request did not succeed and parsing should not proceed; a 200 response with missing card elements usually means the selectors no longer match the current layout. Handle the first by retry or request configuration, and the second by re-inspecting the page.
Page through search results
Define a main function that accepts a base URL, a page count and an output filename, as shown in protected module code0005. Loop from page one to the requested count and build each page URL by appending the page number, following protected module code0006. The captured example pattern adds a segment such as /2_p/ for the second page.
After a successful response, parse the HTML with BeautifulSoup and the lxml parser, following protected module code0008. Find every listing card with the card selector in protected module code0009, then process each card independently so one incomplete listing does not stop the whole run.
Extract card fields with safe fallbacks
For each card, extract the detail URL first, using protected module code0010. Address comes from the address element in code0011 and price from the price element in code0012. Bedroom and bathroom counts use CSS selectors in code0013, square footage uses code0014, and status plus listing agent use code0015.
Apply a consistent missing-value rule: when an element is absent, store N/A rather than raising an error. Skip cards marked with a timeout render marker, because their fields are unreliable. This keeps the output shape stable for downstream analysis.
Field map for the search-result card:
- URL: link to the property detail page
- Address: street address shown on the card
- Price: asking price shown on the card
- Beds and Baths: room counts from the card detail list
- Size (sqft): square footage from the card detail list
- Status: sale or listing status text
- Listing By: agent or agency text
Enrich each listing with estimate and history
Card data alone misses two valuable signals: the Zestimate and the price-history table. A separate detail function, shown in protected module code0016, requests the listing URL, reads the estimate element, then walks each history row and records date, event and price. Rows are accepted only when they contain three cells, and prices without a dollar sign are stored as N/A.
Call that function only when a detail URL was found; otherwise store N/A for both detail fields, following protected module code0017. This separation matters because detail requests multiply total request volume, so they should be optional if a project only needs card-level data.
Save, run and validate the output
Append each completed record to a list, then write the full list to JSON with indentation, following protected module code0018. Run the scraper by supplying a base URL, page count and filename, as in protected module code0019. The captured example targets Brooklyn, New York, five pages, and a file named zillow_listings.json.
The example output in protected module code0021 shows the intended record shape: URL, address, price, beds, baths, size, status, listing agent, a price-history array and a Zestimate. One captured record lists 1338 Park Place, Crown Heights, at $275K with 4 beds, 2 baths and 1,815 square feet, plus a Zestimate of $274,700. Use that shape as a validation target: if your output lacks the history array or estimate keys, the detail step did not run.
Final checklist before a larger run:
- Replace the API key placeholder with {{OWNER_SCRAPING_API_KEY}}
- Run one page first and inspect the JSON
- Verify N/A counts per field
- Confirm detail enrichment is enabled or disabled intentionally
- Re-inspect selectors if any required field is empty
Frequently asked questions
Why collect Zillow listing data?
Listing pages contain prices, locations, property features and trend signals. Collected consistently, they support market analysis, housing-trend tracking, competitor evaluation and building a real-estate application.
Why use a managed scraping API?
A managed scraping API can render JavaScript and reduce exposure to anti-bot controls, so the project can focus on parsing and analysis rather than request infrastructure.
{
"URL": "https://www.zillow.com/homedetails/1338-Park-Pl-Brooklyn-NY-11213/68310361_zpid/",
"Address": "1338 Park Place, Crown Heights, NY 11213",
"Price": "$275K",
"Beds": "4",
"Baths": "2",
"Size (sqft)": "1,815",
"Status": "Auction",
"Listing By": "ISLAND ADVANTAGE REALTY LLC",
"Price History": [
{
"Date": "9/10/2010",
"Event": "Sold",
"Price": "$175,000-41.8%"
},
{
"Date": "12/3/2009",
"Event": "Sold",
"Price": "$300,500-53.8%"
},
{
"Date": "8/28/2008",
"Event": "Sold",
"Price": "$650,000+364.3%"
},
{
"Date": "1/20/2005",
"Event": "Sold",
"Price": "$140,000"
}
],
"Zestimate": "$274,700"
},
{
"URL": "https://www.zillow.com/homedetails/173-E-31st-St-Brooklyn-NY-11226/30660450_zpid/",
"Address": "173 E 31st St, Brooklyn, NY 11226",
"Price": "$250,000",
"Beds": "2",
"Baths": "2",
"Size (sqft)": "1,976",
"Status": "family home for sale",
"Listing By": "LISTING BY: ISLAND ADVANTAGE REALTY",
"Price History": [
{
"Date": "1/24/2024",
"Event": "Listed for sale",
"Price": "$250,000-54.5%"
},
{
"Date": "12/11/2015",
"Event": "Listing removed",
"Price": "$1,850"
},
{
"Date": "10/21/2015",
"Event": "Listed for rent",
"Price": "$1,850"
},
{
"Date": "5/1/2008",
"Event": "Listing removed",
"Price": "$549,000"
},
{
"Date": "4/11/2008",
"Event": "Listed for sale",
"Price": "$549,000+5.7%"
}
],
"Zestimate": "$252,500"
},
... More listing data