Skip to content

Scraping Idealista Property Listings with Selenium

Idealista publishes millions of sale and rental listings across Spain, Portugal, and Italy, which makes it a natural target for market research. The site also deploys anti-bot defenses, so a naive request loop will not survive long. This walkthrough builds a small Selenium crawler for the Spanish offering that walks the province → municipality → listing hierarchy, extracts five fields per property, follows pagination, and pauses for a human when a captcha appears. Each code block below is shown in full, with a short explanation of what it does and how the pieces fit together.

Prerequisites and project setup

Use a recent Python 3 (the original walkthrough used 3.10.0). Create a dedicated directory and an app.py file, then install the two libraries the crawler depends on: Selenium for browser control and undetected-chromedriver to patch ChromeDriver so the browser does not advertise itself as automated. The install command and directory scaffold appear in code0001 and code0002; run them in that order before writing any extraction code.

Command example

pip install selenium undetected-chromedriver 

Open the Idealista homepage

The crawler starts by launching a patched Chrome instance with use_subprocess=True and navigating to the Spanish homepage. That single call is enough to confirm the browser is not being flagged before you invest in extraction code. The snippet is shown in code0003. If the page loads normally, proceed; if you see a captcha, jump to the captcha section before continuing.

Decide the crawl scope and output shape

Before touching the DOM, fix the shape of the output. The walkthrough targets every sale listing in every municipality in every province, producing a nested dictionary keyed by province, then municipality, then a list of property records. Each record carries five fields: title, price, detail (rooms and area), a truncated description blurb from the listing card, and the property URL. The full description lives on the individual property page and is out of scope here. The target JSON shape is shown in code0004. A sample record illustrates the shape in practice: the A Baña entry carries the title "Casa o chalet independiente en o cruceiro, s/n, A Baña", a URL ending in /inmueble/93371668/, a price of "140.000€", a detail string of "2 hab. 60 m²", and a truncated description blurb. Use that record as a smoke-test fixture: if your first municipality run produces a dict with those five keys populated, the selectors are still matching.

Collect province URLs from the homepage

Scroll to the bottom of the homepage and you will find the province list. Each province link lives inside a div with the class locations-list. The extract_provinces function loads the homepage, selects that container by class name, then gathers every anchor tag inside it with a relative XPath (.//a) so links elsewhere on the page are excluded. Each province name becomes a key holding its URL and an empty municipalities dict. The function is shown in code0005. A single XPath expression could replace the two-step lookup, but the two-step version is easier to read and debug.

Collect municipality URLs from each province page

Loop over the province dictionary and visit each province page. The municipality list sits inside a ul with the id location_list, so a single XPath expression (//ul[@id="location_list"]//a) returns every municipality anchor in one call. Store each municipality name as a key with its URL and an empty properties list. The function is shown in code0007, and the main loop that calls it is in code0006. Inspect the province page in DevTools first; if Idealista changes the container id, this is the line to update.

Extract per-listing fields from a municipality page

Each property card is an article element whose class list contains item. Ads also use article tags, but only real listings carry the item class, so the XPath //article[contains(@class, 'item')] filters them out. Note that this is different from //article[@class='item'], which would only match articles whose class attribute is exactly item. The extract_properties function is built up step by step in code0009 through code0013; the final version pulls five fields per card. The field map below shows which selector feeds which output key.

Follow pagination to the last page

Municipalities with thousands of listings are split across pages. The next-page control is an li with the class next, and it only appears when a further page exists. The extract_properties function checks for that element after scraping the current page; if present, it reads the href from the anchor inside the li and calls itself recursively, appending the results. The final version is shown in code0014. Because the check is presence-based, the recursion stops naturally on the last page without a separate page counter.

Handle the DataDome captcha

Idealista's bot defense is DataDome, and it can return a captcha even to a patched ChromeDriver. In the original author's testing, two behaviors stood out: the captcha appeared on the very first visit, and once solved within a session it did not reappear for that same session. Treat those as starting observations rather than guarantees — bot defenses change, and your IP, region, and request cadence may produce different results. The simplest reliable approach is to pause the script with an input() prompt right after the homepage loads, let a human solve the captcha in the visible browser window, then press a key in the terminal to resume. The modified extract_provinces function is shown in code0015. This is a manual step by design; it trades automation for reliability on a site that actively resists scripted access.

Assemble the complete crawler

The full script combines the four functions and a main block that walks provinces, then municipalities, then properties, and finally prints the nested results dictionary. The complete file is shown in code0016. Run it once end-to-end on a small province to confirm the selectors still match before scaling to all of Spain.

Add rotating proxies with Selenium Wire

If you run the crawler repeatedly, Idealista will eventually block it. A User-Agent string alone will not help; rotating proxies are the practical answer. Selenium does not natively support authenticated proxies, so install selenium-wire (code0017) and swap the Chrome initialization for the Selenium Wire variant. The proxy URL embeds an API key and a parameter string that controls rendering and proxy mode; the exact configuration is shown in code0018. The rest of the crawler code is unchanged once the driver is swapped. This walkthrough uses {{OWNER_PROXY_SERVICE_NAME}} for this role; substitute your own proxy provider and sign up at {{OWNER_PROXY_SIGNUP_URL}}. Replace {{OWNER_API_KEY_PLACEHOLDER}} with your credential and confirm {{OWNER_PROXY_PARAMETER_STRING}} against your provider's documentation before running.

Command example

pip install selenium-wire

Where to take it next

The crawler above is deliberately minimal. Before using it on a real project, add error handling around each navigation, and persist the results dictionary to disk at intervals so a crash does not lose the whole crawl. The same structure can be pointed at Idealista's Italian and Portuguese sites by changing the homepage URL and confirming the selectors still match. Treat the selectors as fragile: Idealista can change class names and container ids at any time, so re-inspect the DOM whenever a run returns empty results.

Idealista listing field map
Output keySelector usedWhere it comes from
titlea.item-link (text)Listing card
urla.item-link (href)Listing card
pricediv[contains(@class, 'price-row')] (text)Listing card
detaildiv.item-detail-char (text)Listing card
descriptiondiv[contains(@class, 'item-description')] (text)Listing card (truncated blurb)

Selectors reflect the DOM structure used in this walkthrough; Idealista may change them.

Captcha handling: manual prompt vs proxy service
SituationRecommended approachWhy
One-off crawl, captcha appears on first visitinput() prompt in extract_provinces (code0015)In the original author's testing, the captcha cleared for the rest of the session once solved
Repeated runs, getting blockedSelenium Wire + rotating proxy (code0018)User-Agent alone is insufficient; proxies rotate per request
Need unattended runsProxy service with captcha solvingManual prompt requires a human in the loop

Behavior described reflects the original author's testing; treat it as a starting point, not a guarantee.

Frequently asked questions

Why use undetected-chromedriver instead of plain Selenium?

Plain Selenium leaves fingerprints that bot-detection services recognize. undetected-chromedriver patches ChromeDriver so the browser does not advertise itself as automated, which is the minimum needed to load Idealista's homepage without an immediate block. It is not sufficient on its own for repeated runs; rotating proxies are still required at scale.

Why does the property XPath use contains(@class, 'item') instead of @class='item'?

The contains() form matches any article whose class attribute includes the item class, even when other classes are present. The equality form only matches articles whose class attribute is exactly item, which would miss most real listings. Ads also use article tags, so the item class is what separates listings from ads.

Does the crawler capture the full property description?

No. The description field holds the truncated blurb shown on the listing card. The full description lives on the individual property page, which the walkthrough does not visit. If you need the full text, add a second request per property URL and extract the description from that page.

How does the crawler know when to stop paginating?

It checks whether an li with the class next exists on the current page. That element only appears when a further page is available, so the recursion stops naturally on the last page without needing a page counter or a known total.

Can the same code scrape Idealista Italy or Portugal?

The structure is the same, but the homepage URL and possibly the selectors differ. Point the crawler at the Italian or Portuguese homepage and re-inspect the DOM before running. Treat the selectors as fragile across locales.