Scrape Google Flights with Python: Extract Prices, Times and Airlines
Google Flights presents search results as dynamic cards that load after the initial page request. This guide shows how to request a search URL through an HTML API, let a headless browser wait and scroll, then use CSS selectors to pull price, journey time and airline from each card. The result is a structured list you can export to CSV for comparison or analysis.
Quick answer: what the workflow does
The approach uses a Python client for an HTML API that handles proxies, JavaScript rendering and anti-bot protection. You supply a Google Flights search URL, an extract_rules dictionary of CSS selectors, and a js_scenario that waits and scrolls. The API returns JSON containing a list of flight cards. A short loop collects those cards, and pandas writes them to a CSV file. No Selenium or BeautifulSoup is required for this path.
Set up the Python environment
Use Python 3.6 or newer. Install the client package and pandas with pip. The client package replaces a separate requests library for this workflow because it manages rendering and anti-bot handling. Create a project file such as Google_flights.py and initialize the client with your API key. An IDE with syntax highlighting and a debugger can help catch mistakes before you run the script.
Define extract rules for flight cards
The extract_rules dictionary tells the parser which HTML blocks represent a flight and which child elements hold the values you want. In the captured example, the outer selector targets list items with class pIav2d. The type is list because a search page contains multiple cards. The output dictionary maps three fields: price, time and airline. Each field uses its own CSS selector. To confirm selectors, open a search URL in a browser, right-click a flight card and choose inspect element. Google can change page structure, so if extraction stops returning cards, inspect the page again and update the selectors.
Control JavaScript rendering and scrolling
The HTML API uses a headless browser, and rendering is enabled by default through the render_js parameter. The js_scenario dictionary adds instructions that run before extraction. In the example, the browser waits 2000 milliseconds, scrolls to the bottom of the page, then waits another 2000 milliseconds. The waits give dynamic content time to appear, and the scroll can trigger additional results. Adjust these instructions if a page needs different interaction.
Make the GET request and collect results
The script creates an empty list for all flights, then loops over the URL list. For each URL it calls the client's get method with three parameters: custom_google set to True, the extract_rules dictionary, and the js_scenario dictionary. A retries value of 2 is also passed. The response JSON contains a flights key. The script extends the all_flights list with that value and prints the URL, HTTP status code and number of extracted cards. A status code of 200 indicates a successful request; 400 indicates a poorly configured request.
Export the extracted data to CSV
After the loop, pass all_flights to pandas.DataFrame and call to_csv with the filename flights_data.csv and index=False. This produces a table with one row per flight card and columns for the fields defined in extract_rules. You can add more output fields by extending the output dictionary with additional CSS selectors. The captured example uses a single search URL, but the same function accepts a list of URLs for different routes or dates.
Why use an HTML API for this task
Manual scraping or browser automation with Selenium requires more setup and maintenance. An HTML API handles rotating proxies, CAPTCHA bypassing and JavaScript rendering behind the scenes. With one Python package you can extract airline names, prices and travel times from live search result pages. The same API can also support related workflows such as news collection when you need article-level coverage of airlines, airports or travel trends.
Decision checklist before you run the scraper
Use this checklist to avoid common setup mistakes:
- Confirm Python 3.6 or newer is active in the environment where you installed the packages.
- Verify the API key is present in the client initialization line.
- Open the target search URL in a browser and confirm flight cards are visible after the page loads.
- Inspect one flight card and check that the outer selector and each output selector still match the current markup.
- Confirm the js_scenario waits long enough for results to render; increase the wait values if cards are missing.
- Run with a single URL first and check the printed status code and flight count before scaling to a list.
- If the status code is 400, review parameter names and values; if the status is 200 but the count is zero, revisit the CSS selectors.
Worked example: from one card to one CSV row
Suppose a search result contains a flight card matching li.pIav2d. Inside that card, the price selector div.BVAVmf > div.YMlIz returns a price string, the time selector div.gvkrdb.AdWm1c returns a journey time, and the airline selector div.Ir0Voe > div.sSHqwe > span:not([class]) returns an airline name. The parser produces one dictionary with keys price, time and airline. After the loop, pandas turns the list of such dictionaries into rows. The CSV therefore has one row per card and three columns in this example. If a selector returns nothing for a particular card, that field may be empty in the row; inspect the card to see whether the markup differs.
Failure distinction: request configuration vs selector drift
A non-200 status code usually points to a request configuration problem, such as an invalid parameter or missing API key. A 200 status with zero extracted flights usually points to a selector problem: the page loaded, but the CSS selectors no longer match the current markup. The printed status code and flight count help separate these cases. If the count drops after a period of working runs, inspect the page and update the selectors before changing request parameters.
Field and control map
The main configuration objects have the following roles:
- extract_rules.flights.selector: the CSS selector for each flight card container.
- extract_rules.flights.type: set to list so multiple cards are returned.
- extract_rules.flights.output.price: CSS selector for the price element.
- extract_rules.flights.output.time: CSS selector for the journey time element.
- extract_rules.flights.output.airline: CSS selector for the airline name element.
- js_scenario.instructions: an ordered list of browser actions; each action is a wait or evaluate step.
- custom_google: a request parameter set to True for this Google scraping setup.
- retries: the number of retry attempts passed to the client call.
Frequently asked questions
Can this workflow return real-time flight prices?
The API fetches live data from Google Flights pages, including prices, availability and other content rendered with JavaScript. The values reflect what the page returns at request time.
How are blocks and CAPTCHAs handled?
Automatic residential proxy rotation is enabled by default for GET API calls. You do not need to manage cookies, user agents or solve CAPTCHAs manually in this workflow.
Can I extract departure times and airline names?
Yes. The extract_rules dictionary lets you define CSS selectors for the elements you want, including times, airlines and other fields present in the page markup.
How is usage priced?
{{OWNER_TRIAL_TERMS}} After any trial, {{OWNER_PRICING_BASIS}}. Check the current pricing page for details.