Skip to content

Scrape Google Play App Data and Reviews with Python

Google Play app pages are not static HTML. Descriptions and reviews often sit behind buttons and pop-ups, so a plain request can return a page that looks complete but is missing the fields you need. This guide shows a Python workflow that renders the page, clicks the right controls, parses the result with BeautifulSoup, and writes the collected fields to a CSV file. The example uses a rendering client that handles JavaScript execution and proxy rotation, so the script focuses on the extraction logic rather than browser plumbing. It is written for readers who already have basic Python familiarity and want a repeatable extraction task rather than a one-off copy-paste.

Quick answer: what the workflow does

The first decision is whether you need reviews at all. If you only want the app metadata, a single rendered request that opens the description panel is enough. If you also want reviews, plan for two requests: the description panel and the reviews panel are separate overlays, and dismissing one can drop the other, so each request opens only its own panel. Both requests render JavaScript, wait for the page to settle, and hand the finished HTML to BeautifulSoup. The collected fields go into a dictionary and are written to app_data.csv with pandas. The exact request parameters and selector strings are preserved in the accompanying protected code modules; this page explains the task and the decisions around them.

Set up the environment

Install Python and create an isolated environment so package versions do not collide with other projects. On Windows, confirm that Python is added to PATH during installation. The supplied setup uses a virtual environment named venv. The exact creation and activation commands are preserved in the accompanying protected modules. The libraries used in the full example are the scraping client, BeautifulSoup, and pandas. The source lists requests, beautifulsoup4, lxml, pandas, and web scraping API as possible installs, but the minimal working set for the supplied script is the client plus pandas and BeautifulSoup.

Request the app page with JavaScript rendering

The app page URL takes an app_id such as com.ludo.king. The request must render JavaScript and wait for the page to reach a stable network state. The example request passes custom_google, wait_browser set to networkidle2, premium_proxy, render_js, country_code set to us, and retries set to 2 to the rendering client. A js_scenario dictionary tells the renderer to wait for a button matching a CSS selector and click it, then wait a short interval. The exact parameter names and selector strings are preserved in the protected code modules. Treat those selectors as page-specific: if Google Play changes its markup, the selector may stop matching even when the request itself succeeds.

Parse the rendered HTML with BeautifulSoup

After the response arrives, load response.text into BeautifulSoup with the lxml parser. A small helper, extract_text, accepts a CSS selector, selects the first matching element, and returns its stripped text or None. That helper is used for individual fields. A second helper, extract_reviews_dict, selects the review containers and builds a dictionary keyed review_1, review_2, and so on. The exact selectors and helper definitions are preserved in the protected code modules. When a selector returns None, check whether the pop-up actually opened before changing the selector.

Fields to extract and where they come from

The supplied example collects the following fields. The field map below separates fields that come from the description request from fields that come from the reviews request. Exact selector strings are preserved in the protected code modules.

Avoiding blocks and incomplete data

Google Play can rate-limit or block repeated requests. Three controls help: realistic User-Agent headers, short irregular delays between requests, and proxy rotation. The example request sets premium_proxy to true so the rendering client routes through its rotating proxy pool. A manually chosen User-Agent can be added through the request headers. Delays mimic human browsing and reduce strain on the recipient server. The exact header and parameter strings are preserved in the protected code modules.

Wrap, handle errors, and save

The full example wraps the workflow in google_play_store_app_data(app_id), with nested extract_text and extract_reviews_dict helpers. A status check returns a failure message when the description response is not 200. The data dictionary is converted to a pandas DataFrame and written to app_data.csv with index=False. The exact function names, status check, and CSV call are preserved in the protected code modules. To process many apps, loop over a list of app IDs and call the function for each one, adding a short delay between calls. Because the rendering client handles proxy rotation and JavaScript rendering per request, the same function can be reused across a long list of app IDs without extra proxy management.

Worked example with the supplied app ID

The supplied example calls the function with app_id='com.ludo.king'. The expected output is a dictionary containing name, rating, description, downloads, content_rating, support_email, updated_on, tags, whats_new, developer, android_os_requirement, in_app_purchase_range, released_on, and a reviews dictionary. The CSV file app_data.csv is written in the working directory. If the description request fails, the function returns 'Failed to retrieve the page.' and does not write a row. If the request succeeds but a selector returns None, the field is empty rather than the whole call failing, so check the output for missing values before treating a run as complete.

Pre-run checklist

Use this checklist before running the scraper. Confirm the app ID is correct and the page loads in a browser. Confirm the rendering client's API key is set. Confirm render_js and the JavaScript scenario are enabled. Confirm the description and review selectors still match the current page. Confirm the status check is in place. Confirm the output path is writable. If reviews are missing, check whether the reviews pop-up opened before changing the review selector.

Field map for the supplied Google Play extraction
FieldSource requestNotes
nameDescription requestApp title
ratingDescription requestDisplayed rating
descriptionDescription requestRequires the description pop-up to be opened
downloadsDescription requestDisplayed install count
content_ratingDescription requestContent rating label
support_emailDescription requestSupport contact shown on the page
updated_onDescription requestLast update label
tagsDescription requestCategory or tag text
whats_newDescription requestRelease notes block
developerDescription requestDeveloper name
android_os_requirementDescription requestMinimum Android version
in_app_purchase_rangeDescription requestIn-app purchase price range
released_onDescription requestRelease date label
reviewsReviews requestDictionary of visible reviews keyed review_1, review_2, ...

Field availability depends on the app page and the current markup. A selector that returns None leaves that field empty.

Failure distinction: HTTP error vs. missing selector vs. incomplete dynamic content
SymptomLikely causeWhat to check
Function returns 'Failed to retrieve the page.'Description request status was not 200API key, request parameters, network, and whether the page is reachable
One or more fields are NoneSelector did not match the rendered HTMLWhether the pop-up opened and whether the selector still matches the current markup
Reviews dictionary is emptyReviews pop-up did not open or review containers were not renderedThe reviews JavaScript scenario and the review container selector
Request succeeds but data looks stalePage content changed or region-specific content was servedcountry_code parameter and the current page state

These are diagnostic categories, not guarantees about any specific run.

Frequently asked questions

Is it legal to scrape data from Google Play?

The supplied guidance says scraping public data from Google Play is legal, although it may violate Google's terms of service. This is a general statement, not legal advice. Review the current terms of service and applicable law before scraping.

Can I scrape user reviews that load dynamically on the page?

Yes, if the tool you use can render JavaScript. The supplied approach simulates a click on the reviews control and waits before parsing, so content that is not in the initial HTML can be reached. The exact JavaScript scenario and review selector are preserved in the protected code modules.

Why does my scraper get blocked or return incomplete data?

Common causes are missing or unrealistic headers, too many rapid requests, and no IP rotation. Incomplete data can also come from JavaScript rendering not being enabled or from a selector that no longer matches the page. Check the request parameters and the JavaScript scenario before changing the parsing logic.

How do I scrape multiple app pages in one run?

Loop over a list of app IDs and call the extraction function for each one. Add a short delay between calls to reduce the risk of rate limiting. The supplied example writes a single row to app_data.csv; for multiple apps, append each result to a list or write one row per app.

Do I always need two requests?

No. If you only need fields from the description page, one request with the description scenario is enough. The second request is needed when you also want reviews, because closing one pop-up can remove the other. The supplied example uses two clients and two requests for that reason.