Skip to content

How to Scrape TripAdvisor Hotel Data with Python

TripAdvisor hotel listing pages pack names, prices, ratings, review counts, rankings and short descriptions into a single scrollable view. That makes them a useful target if you want to compare stays, track price movement, or study review sentiment. The catch is that the page is heavily JavaScript-driven and defends itself against automated traffic, so a plain HTTP request often returns markup without the numbers you actually want.

This guide walks through a working Python setup that uses a scraping API to render the page, click the load-more button, and pull structured hotel records into a CSV. It also covers how to reach later pages and how to tell the difference between a rendering problem and a block.

What you need before you start

You need Python 3.6 or newer and two packages: the scraping client library and pandas for turning the response into a table. Install both from a terminal:

  • web scraping API — the Python client that talks to the scraping API and handles headless browsers, proxy rotation and JavaScript rendering.
  • pandas — provides DataFrames so extracted records can be cleaned and exported.

You also need an API key. Register an account to obtain one, then copy the key from your account dashboard. Keep it out of version control; the examples below show it as a placeholder. Any free-credit allowance or trial duration shown at sign-up is set by the account provider and may change, so check the current plan details before relying on it. The exact onboarding flow and any promotional credit offer are pending owner confirmation before this page goes live.

A quick note on scope: the target page is a hotel listing for a specific city. Pick the city you care about and copy its listing URL from the browser before writing any code. The example uses Rome.

Make your first request

Start with a minimal script that fetches the listing page and prints the raw response. That confirms your key works and the API can reach the target before you add extraction rules.

The client is created with your API key, then a GET call is made against the listing URL. Two parameters matter at this stage:

  • premium_proxy routes the request through a pool of legitimate proxy servers, which reduces the chance of an IP-based block.
  • js_scenario tells the headless browser what to do on the page. The simplest version waits 2000 milliseconds so the page can finish loading.

If the status code is not 200, print it so you can diagnose the failure. Otherwise print the response text. Expect messy HTML at this point — that is normal, and the next step turns it into structured records.

Turn HTML into hotel records with extract_rules

The extract_rules parameter is a dictionary that tells the API which CSS selectors to run against the page. It has three levels:

  1. A named container (for example Hotel card) with a selector that bounds each record and a type of list so every matching card becomes one row.
  2. An output dictionary whose keys become column names and whose values are CSS selectors scoped inside the card.
  3. Individual selectors for each field you want.

To find the right selectors, open the listing page in a browser, press F12 (or right-click and choose Inspect), and locate the element that wraps a single hotel card. Copy its selector. If the copied selector only matches one card, trim the extra parts so it targets the shared class instead.

A practical field map for the hotel card looks like this:

  • Hotel name — heading inside the card link.
  • Price from — the price container inside the card button.
  • Rating — the bubble rating value span.
  • Review amount — the bubble review count span.
  • Ranking — the position span inside the card header.
  • Description — the short text span under the card.

When the dictionary is complete, rerun the script. The output should now be a list of dictionaries, one per hotel, instead of raw HTML.

Click the load-more button to get past six results

A first extraction pass often returns only six hotels because the listing hides the rest behind a load-more button. To capture the full set, find that button's CSS selector in the browser inspector and add a click instruction to js_scenario after the wait.

The instruction list becomes: wait 2000 ms, then click the button selector. Rerunning the script with that change typically raises the count to around 30 cards, which matches the first page of results.

Once the records look right, convert the response JSON into a pandas DataFrame and write it to a CSV file. That gives you a reusable table you can filter, sort or join with other data.

Handle TripAdvisor's anti-bot defenses

TripAdvisor actively deploys anti-bot measures to prevent automated data collection. Frequent requests from a single IP, unusual traffic patterns, or incomplete rendering can quickly trigger blocks. The API is built to handle the most persistent of these challenges, and it exposes several tools you can combine as needed — none of them are forced on by default:

  • Premium residential proxies distribute requests across a large pool of IPs, with options to pick a desired geolocation.
  • If the default configuration does not let you scrape the data you need, you can add spoofed headers and user-agents so each request looks like it comes from a real browser rather than a script. This is an optional fallback you enable only when needed, not a default behavior.
  • Headless browser rendering executes JavaScript before extraction, so dynamic elements such as "See all hotels" buttons are present when selectors run.

If you still see failures, check the js_scenario instructions first — a missing wait or click is the most common cause of empty results. Then confirm premium_proxy is enabled.

Reach later pages with the oa30 offset

The first page of a city listing has no page indicator in the URL. From the second page onward, an oa segment appears in the middle of the path, and the number after it increases by 30 for each additional page. That gives a predictable pattern you can generate programmatically.

A small helper function can build a list of URLs: keep the base URL as page one, then for each extra page replace the city slug segment with an oa offset of 30, 60, 90 and so on. A user-input prompt lets you choose how many pages to target each run.

To fetch several pages at once, wrap the scraper call in a thread pool executor with a fixed number of workers. Concurrency adds complexity, so start with a small worker count and confirm the output before scaling up.

Diagnose empty or partial results

When a run returns nothing useful, work through this checklist in order:

  1. Did the page finish rendering? Add or increase the wait in js_scenario.
  2. Did the load-more button get clicked? Confirm the button selector still matches the current markup.
  3. Was a CAPTCHA shown? Empty responses often indicate a challenge page rather than a selector problem.
  4. Is the request going through a premium proxy? IP-based blocks produce empty or truncated responses.
  5. Do the field selectors still match? TripAdvisor changes class names, so a selector that worked last month may need updating.

Distinguishing these cases matters: a rendering failure usually returns partial HTML, while a block or CAPTCHA typically returns an empty body or a challenge page. Checking the raw response text before parsing saves time.

Frequently asked questions

Is scraping TripAdvisor legal?

Scraping publicly available travel data is generally treated as legal in many jurisdictions, but TripAdvisor's terms of service still apply. Review those terms and any local rules before you collect data, and keep your requests limited to public pages. This is general information, not legal advice.

Does the API handle TripAdvisor's dynamic content?

Yes. The HTML API supports JavaScript rendering, so content injected after the initial page load — prices, reviews and similar fields — is present when extraction rules run. That is why a wait instruction in js_scenario is part of the basic setup.

How do I extract only reviews or only prices?

Edit the extract_rules dictionary so the output keys and selectors target only the review text or price containers you need. Removing the other fields keeps the response smaller and focused on the content you plan to use.

Why do I get empty responses?

Empty responses usually mean JavaScript has not finished rendering or a CAPTCHA was triggered. Check the js_scenario instructions for a missing wait or click, and make sure premium_proxy is set to True so requests are not blocked by IP.