Skip to content

Scraping Realtor.com Listings with Python and Selenium

Realtor.com holds a large share of US property listings, and its search results pages are the practical entry point for price tracking, market research and lead building. The catch is that the site actively resists automated access, so a plain HTTP request will not get you far. This guide walks through a Python and Selenium workflow that loads a city search page, reads each result card, follows the Next link across pages, and switches to a hosted scraping API when CAPTCHAs start interrupting the run.

What the scraper produces

The target is the search results page, not individual property detail pages. Each result card carries a price plus optional bed, bath, square footage, lot size and address values. Because listings are inconsistent, some cards will be missing one or more of those fields, so the extractor must check for presence before reading text.

The worked example uses Cincinnati, OH, but the URL pattern is predictable: swap the city and state segments to target another market. Hardcoding the search URL avoids the auto-suggest dropdown on the homepage, which tends to intercept clicks and break automation.

Prerequisites and project setup

Use a recent Python 3 release (the original walkthrough used 3.10.0). Create a project folder and an app.py file, then install two libraries: Selenium for browser control and undetected-chromedriver, which patches the automation flags that Realtor.com's basic bot detection looks for. A standard Selenium Chrome session is likely to be blocked quickly; the patched driver buys time but is not a permanent solution.

Project scaffolding and the install command are supplied as exact modules (code0001 and code0002) and are reproduced in full alongside this page.

Loading a Realtor.com search results page

Launch the patched Chrome driver and navigate directly to a search URL. The example targets Cincinnati with the newest-listings filter. To change market, replace the city and state in the path; a Redmond, WA variant is shown in the supplied code module.

The driver launch and both URL examples are exact modules (code0003 and code0004). Run the script and confirm a Chrome window opens on the results page before adding extraction logic.

Field map: which selector returns which value

Each result card is an li element with data-testid set to result-card. Inside the card, individual values sit on elements carrying a data-label attribute. The table below maps each field to its element type and attribute value so you can verify the DOM before writing selectors.

FieldElementAttribute
pricespandata-label='pc-price'
bedslidata-label='pc-meta-beds'
bathslidata-label='pc-meta-baths'
sqftlidata-label='pc-meta-sqft'
plot_sizelidata-label='pc-meta-sqftlot'
addressdivdata-label='pc-address'

Only price is treated as always present in the reference code; the other five are wrapped in existence checks. If Realtor.com changes its markup, this table is the first place to re-inspect.

Collecting all result cards

Use Selenium's find_elements with an XPath that selects every li whose data-testid is result-card. The leading double slash scans the whole document; the attribute predicate filters out other list items on the page. CSS selectors would also work here, but the walkthrough sticks with XPath.

The selector code is supplied as exact module code0006.

Extracting price, beds, baths, square footage and address

Loop over the collected cards and read each field relative to the current card. The XPath must start with a dot (.//) so Selenium searches inside the card rather than returning the first matching element on the whole page. Chain .text to strip the HTML and keep the visible value.

For the optional fields, call the plural find_elements first. An empty list means the field is absent, so the script skips it instead of raising an error. This is the difference between a missing square-footage value and a broken scraper: absence is expected data, not a failure.

The per-field extraction snippets and the combined loop are supplied as exact modules code0007, code0008 and code0009.

Following pagination to the next page

Search results are split across pages, with controls near the bottom. Rather than clicking numbered links, read the href from the anchor whose aria-label is 'Go to next page' and pass that URL to driver.get. Direct navigation is more reliable than simulating a click because it avoids background XHR requests that can stall the browser.

Loop while the href contains 'http'; when it no longer does, you have reached the last page. The href extraction and the full paginated script are supplied as exact modules code0010 and code0011.

When CAPTCHAs appear: switching to a hosted scraping API

undetected-chromedriver reduces the chance of hitting a CAPTCHA screen but cannot eliminate it. Once blocking becomes frequent, the practical options are solving challenges manually (with no guarantee of frequency) or moving the fetch to a service that rotates proxies and clears challenges for you.

A hosted scraping API accepts the target URL plus extraction rules written as CSS selectors. The rules mirror the field map above: a list selector for result cards, nested selectors for each field, and an @href output for the next-page anchor. JavaScript runs before extraction, so single-page applications are handled. The client setup and rule payload are supplied as private exact modules (code0012 and code0013) and are retained for owner review rather than published verbatim.

The response shape is shown in exact module code0014: a next_page path plus a properties list with the same field names used in the Selenium version.

Worked example: reading one Cincinnati result

Take the first captured Cincinnati card. Its price reads $99,900, beds 2bed, baths 1.5bath, square footage 1,296sqft, lot 3,528sqft lot, and the address is 6636 St, Cincinnati, OH 45216. Every field is present, so the conditional checks all pass and the dictionary is fully populated.

Now compare a Green Township new-build card from the same run: price From$472,995, beds 3bed, baths 2bath, square footage 1,865sqft, and an empty plot_size. The lot field is absent, so the existence check skips it and the record is stored with an empty string rather than crashing the loop. That contrast is the reason the conditional pattern matters more than any single selector.

Local Selenium vs hosted scraping API

SituationLocal Selenium + undetected-chromedriverHosted scraping API
Small, one-off runWorkable; you control the browserExtra setup for little gain
Frequent CAPTCHA screensManual solving, no frequency guaranteeProxy rotation and challenge clearing handled for you
Large-scale extractionIP bans and proxy pools become your problemOffloads rendering and blocking to the provider
JavaScript-heavy pagesHandled by the real browserHandled by the provider's renderer
Cost modelYour own infrastructurePer-request billing (confirm current terms with the provider)

For projects that keep growing, the original walkthrough also points to Scrapy as a fuller framework with pause/resume, filtering and multiple output formats, and notes that it can be combined with a hosted API.

Legality and responsible scraping

Prices, square footage and days on market are factual, publicly visible data points, and collecting them at scale is legal in most jurisdictions. The line to watch is personal data: agent names, phone numbers and license details are personally identifiable information, and commercial collection is regulated by laws such as GDPR.

Respect robots.txt, keep request rates low, and avoid actions that could overload Realtor.com's servers, which its terms of service prohibit. Treat this section as general guidance, not legal advice; confirm your own obligations before running a production job.

Realtor.com result card field map
FieldElementAttribute
pricespandata-label='pc-price'
bedslidata-label='pc-meta-beds'
bathslidata-label='pc-meta-baths'
sqftlidata-label='pc-meta-sqft'
plot_sizelidata-label='pc-meta-sqftlot'
addressdivdata-label='pc-address'

Selectors reflect the markup observed at the time of the original walkthrough; Realtor.com may change its DOM.

Local browser vs hosted scraping API
SituationLocal SeleniumHosted API
Small one-off runWorkableExtra setup
Frequent CAPTCHAsManual solving, no guaranteeHandled by provider
Large-scale extractionProxy pools and IP bans on youOffloaded to provider
JavaScript-heavy pagesReal browser rendersProvider renders
BillingYour infrastructurePer-request (confirm current terms)

Billing terms are provider-specific and not verified here; check current pricing before committing.

Frequently asked questions

How do I get past CAPTCHAs when scraping real estate sites?

Rotating residential proxies plus a scraping API that renders JavaScript and clears challenges automatically is the usual combination. A patched browser driver reduces the frequency but does not remove the problem.

Can BeautifulSoup scrape Realtor.com on its own?

No. BeautifulSoup parses HTML well, but Realtor.com depends on JavaScript and bot defenses. You need Selenium, Playwright or a scraping API to load the page before parsing.

Which language is best for real estate data scraping?

Python is the common choice because of its library ecosystem: Selenium, Scrapy and BeautifulSoup all have mature Python support.

How should I handle pagination on Realtor.com?

Find the Next Page element, read its href, and loop requests until the Next link is no longer present or no longer contains a usable URL.

Does Realtor.com offer an official API?

There is no public API for free data extraction. Developers typically build custom scrapers or use third-party real estate APIs.

Why are some fields empty in my output?

Listings are inconsistent. A card may lack square footage, lot size or another value. Use the plural find_elements check and skip the field when the list is empty; an empty value is expected data, not a scraper failure.