Scrape Google Search Results with Python
Google SERPs carry organic links, local businesses, related queries, questions, knowledge panels, ads, and map listings. Pulling that data by hand does not scale, and a naive script quickly runs into consent screens, bot checks, rate limits, and markup that shifts without notice. This guide walks through three practical routes in Python: a managed scraping API that returns structured JSON, a no-code request builder, and a do-it-yourself requests + BeautifulSoup parser. It also shows how to turn the output into a domain position tracker and how to display map results.
Why teams pull SERP data
Common reasons to automate SERP collection:
- SEO tooling that monitors keyword positions across large keyword sets.
- Site owners checking where their pages rank for a fixed keyword list.
- Competitor analysis: comparing your positions against another site for the same queries.
- Demand and trend research, for example combining SERP snapshots with time-series interest data.
- Content planning: seeing which topics and questions already surface for a query.
- Lead generation, news monitoring, price checks, and field research.
All of these share one requirement: the data must arrive repeatedly and in a shape a program can consume. That is where the choice of method matters.
What makes Google SERP scraping hard
Four obstacles show up again and again:
- Bot checks. CAPTCHA-style challenges are easy for a human and hard for a script. Repeated failures can get an IP blocked for hours.
- Consent screens. In a fresh browser session Google may show a cookie consent page before any results. A script has no mouse to click Accept.
- Rate limits. Sending too many requests too fast invites temporary blocks. Waiting, rotating IPs, or throttling requests are the usual mitigations.
- Noisy, unstable HTML. The response is full of scripts, headers, footers, and extra markup. Google also uses opaque element IDs that change between visits, so selector rules built on those IDs break quickly.
Managed API or build it yourself?
A custom script gives you full control and full understanding of every step, and you can extend it however you like. The trade-off is that you also own every failure: consent handling, proxy rotation, rate limiting, and parser maintenance when Google changes markup. You will likely still need to buy servers or proxies to work around rate limits.
A managed scraping API shifts those problems to the provider. You send one request and receive structured JSON instead of raw HTML, with the option to download the full HTML when you need it. The cost is a subscription, but it removes most of the operational work.
| Approach | What you control | What you maintain | Best for |
|---|---|---|---|
| Managed scraping API | Query, params, output shape | Nothing on the fetch side | Production SERP data, structured JSON |
| DIY requests + BeautifulSoup | Every line of code | Consent, proxies, selectors, retries | Learning, small experiments |
| Headless browser (Playwright, Puppeteer, Selenium) | Browser behavior | Heavier runtime, still detectable | Cookie/JS-dependent pages, small runs |
A headless browser can execute JavaScript, keep cookies, and follow redirects, which helps when Google expects browser-like behavior. It is heavier and slower than a plain HTTP request, and Google can still flag automation through request frequency, IP reputation, and session patterns. For production volume, it remains fragile.
Quickstart: structured SERP JSON from Python
The managed route needs an API key and a small Python project. The captured workflow uses uv to scaffold the project, installs requests and python-dotenv, and stores the key in a .env file that is added to .gitignore.
Setup commands and the boilerplate script are provided as exact code modules (code0001, code0002, code0003, code0004, code0005). The key points:
- The endpoint is the Google API route; the search parameter carries the query.
- The API key is read from an environment variable, never hard-coded.
- The default response language is English; a language parameter overrides it.
- The response body is JSON, and response.content holds the full payload.
A concrete example from the captured run uses the query "pizza new-york". The same pattern works for any keyword you want to monitor.
Field map: what the JSON response contains
The response is a JSON object. These are the fields worth knowing before you write analysis code:
| Field | What it holds | Key sub-fields |
|---|---|---|
| meta_data | Query context | search URL, number of results, number of organic results |
| organic_results | Standard search results | url, description, position, domain |
| local_results | Location-based listings | title, review (0–5), review_count, position |
| top_ads | Top advertisements | description, title, url, domain |
| related_queries | Similar searches | title, position, sometimes url |
| questions | Popular related questions | text, position, sometimes answer |
| knowledge_graph | General entity info | title, type, images, website, description |
| map_results | Location listings | position, title, address, rating, price |
Two notes on reading this map:
- review is a rating on a 0–5 scale (for example 4.5), and review_count is the number of reviews behind it.
- price in map_results is a symbolic tier such as $ for cheap and $$$ for expensive, not a numeric amount.
The sample response lives in the official documentation (link0069). The display helpers in code0006 through code0010 show how to iterate each field safely, including fallbacks when a value is missing.
Displaying organic, local, related, and question results
The captured scripts split display into small functions, one per result type. Each function reads a list from the JSON and prints position, title, and the relevant detail. The page parameter is available for pagination when you need page 2, 3, and beyond.
- Organic results print position, title, URL, and description.
- Local results print position and title, then a rating line that adapts when review or review_count is missing.
- Related queries print position and title, or a dash when position is absent.
- Questions print position and text, then the answer when Google returns one, otherwise a clear 'not shown' line.
The exact functions are preserved in code0006, code0007, code0008, code0009, and the wiring in code0010.
Example: tracking domain positions over time
A useful application is a position tracker. The captured example stores historical positions in a CSV file where the header row lists the domains and each following row holds their positions. For the "pizza new-york" query, the sample file holds three domains with previous positions 2, 3, and 4.
The workflow:
- Read the CSV header to get the monitored domains, and read the rows as history.
- Fetch fresh results for the query.
- Normalize domains by stripping a leading www. so Google's domain values match the CSV.
- Find each domain's first organic position; 0 means not found in the current results.
- Compare the current position with the latest history row and label the change: up by N, down by N, same position, newly found, dropped out, or still not found.
- Append the new positions as a row, ensuring the file ends with a newline first.
The captured code keeps the highest-ranking position when the same domain appears more than once, because organic results are already ordered by position. The full implementation is in code0011 through code0022. Run it with the command in code0022.
Displaying Google Maps results
Map results use the same endpoint with search_type set to maps. The captured script fetches the payload, then prints each listing's position, title, address, opening hours, link, price, and rating with review count. A small formatter turns missing values into N/A and joins list values with commas.
The supported search types are classic (the default), maps, and news. The full script is in code0023 through code0029.
No-code option: the request builder
If you would rather not write Python, the request builder lets you configure a request in a form and run it. The configurable options are:
- Search: the query text.
- Country code: the country the request appears to come from; defaults to USA.
- Number of results: how many results to return; defaults to 100.
- Search type: classic by default; maps and news are also supported.
- Language: display language; defaults to English.
- Device: desktop by default; mobile is also available.
- Page: defaults to 1; use 2 for the second page, and so on.
After clicking Try it, the results appear in the builder for you to use. The exact form controls are preserved in the form-controls-contract module.
DIY route: requests and BeautifulSoup
The DIY route sends a request directly to Google's search URL and parses the returned HTML. It is a good way to see the mechanics, but it is not a stable production setup.
The captured script:
- Sends a GET request with q, hl (language), and gl (country) parameters, a user agent header, and a CONSENT cookie.
- Parses the HTML with BeautifulSoup.
- Selects h3 elements inside the #search block as candidate result titles.
- Walks up to the nearest parent anchor with an href, cleans Google redirect links of the form /url?q=..., and skips duplicates.
- Assigns positions in the order results appear and prints them.
The CONSENT cookie value in the captured code is an example only. Google may use different consent values by region, language, account state, or session. In practice you may need to open Google in a browser, accept the banner, and copy the actual CONSENT value from developer tools. Even then the trick is not fully reliable: Google can still return a consent page, a fallback page, or a different format.
The exact code is in code0030 through code0044.
A failure distinction worth knowing
When the DIY parser returns nothing, the cause is usually not the parser. A common fallback page reads: "If you're having trouble accessing Google Search, please click here". That means Google did not send the SERP HTML you expected. Possible reasons:
- A cookie consent page was returned instead of results.
- A bot-detection page was returned.
- The HTML differs by region, language, device, or session.
- The markup changed without warning.
- A selector that worked yesterday returns nothing today.
- A request that worked once on your laptop fails after repetition.
Changing headers, cookies, user agents, or locale parameters sometimes helps and sometimes does nothing. The underlying issue is that you are parsing a consumer-facing page, not an API with a stable contract.
Why not just use Google's official API?
Using the official Google Search API looks like the clean answer, but the official path has narrowed. Google's Custom Search JSON API no longer accepts new sign-ups, and current users are expected to migrate to a replacement by January 1, 2027. For new projects, Google's stated direction is Vertex AI Search, which covers site-restricted search across up to 50 domains; teams that need full-web search are asked to reach out to Google directly.
That leaves a gap for anyone who needs broad coverage of Google Search results: the official product may not be open, flexible, or practical enough for the job. The DIY approach teaches the mechanics — send a request, parse the HTML, pull titles and links — but it is not dependable in production. A scraping API that absorbs browser rendering, JavaScript, retries, localization, and blocks is usually the steadier path when the data has to arrive reliably.
Using the Python client for rendered requests
The captured example also shows a Python client that sends requests through the scraping service with browser rendering enabled. The client is installed as a package and configured with the API key from the environment.
Key parameters in the captured request:
- custom_google: true tells the service the target is Google and needs special handling.
- render_js: true renders the page in a browser instead of returning raw HTML.
- wait_browser: load waits for the browser load event, which is more reliable than a fixed delay.
- block_resources: false keeps images and CSS enabled, since blocking them can break the rendered page.
- premium_proxy: true is often needed because Google returns CAPTCHAs or fallback pages on standard proxies. If premium proxies still fail, stealth_proxy=true may be required, at higher cost.
The script saves the rendered HTML, parses results with BeautifulSoup, skips internal Google links, and saves a full-page screenshot. Note that screenshot=true returns image bytes in the response body; the script writes them to a file itself. The exact code is in code0046 through code0061.
Compliance note
Google treats automated queries to Search, including scraping results for rank checking, as machine-generated traffic. Before building anything serious around SERP data, make sure your use case, tooling, and data handling comply with the relevant terms and laws.
| Approach | What you control | What you maintain | Best for |
|---|---|---|---|
| Managed scraping API | Query, params, output shape | Nothing on the fetch side | Production SERP data, structured JSON |
| DIY requests + BeautifulSoup | Every line of code | Consent, proxies, selectors, retries | Learning, small experiments |
| Headless browser | Browser behavior | Heavier runtime, still detectable | Cookie/JS-dependent pages, small runs |
Comparison reflects the captured tutorial's framing; headless browser reliability depends on Google's detection and is not guaranteed.
| Field | What it holds | Key sub-fields |
|---|---|---|
| meta_data | Query context | search URL, number of results, number of organic results |
| organic_results | Standard search results | url, description, position, domain |
| local_results | Location-based listings | title, review (0–5), review_count, position |
| top_ads | Top advertisements | description, title, url, domain |
| related_queries | Similar searches | title, position, sometimes url |
| questions | Popular related questions | text, position, sometimes answer |
| knowledge_graph | General entity info | title, type, images, website, description |
| map_results | Location listings | position, title, address, rating, price |
review is a 0–5 rating; price is a symbolic tier ($ to $$$), not a numeric amount. Field presence can vary by query and result type.
Frequently asked questions
Can I use Google's official Custom Search JSON API instead?
Not for most new projects. Google has stopped onboarding new Custom Search JSON API customers, and anyone already using it is expected to transition to a different product by January 1, 2027. As things stand, Google directs newcomers to Vertex AI Search, which handles site-restricted search over a maximum of 50 domains; anyone who needs to query the entire web is told to get in touch with Google. So if your goal is wide coverage of Google Search results, the official offering is unlikely to be a workable fit.
Does a CONSENT cookie reliably bypass Google's consent page?
No. The value shown in the tutorial is an example only. Google may use different consent values by region, language, account state, or session. You may need to accept the banner in a browser and copy the actual CONSENT value from developer tools. Even then, Google can still return a consent page, a fallback page, or a different format.
Will a headless browser make DIY scraping reliable?
It can help with cookie banners, JavaScript, and browser-dependent behavior, but it is not a guarantee. A headless browser is heavier, slower, and more expensive than a plain HTTP request, and Google can still detect automation through request frequency, IP reputation, sessions, and datacenter traffic. For production volume it remains fragile.
My parser returns zero results. Is BeautifulSoup broken?
Usually not. A common cause is that Google returned a fallback page instead of the SERP HTML, for example a consent page, a bot-detection page, or a page that varies by region, language, device, or session. Check the raw response first before changing selectors.
What does a position of 0 mean in the tracker?
A position of 0 means the domain was not found in the current organic results. The tracker labels this as 'not found', 'newly found', 'dropped out', or 'still not found' depending on the previous history row.