Scrape Google Finance with Python: HTML to Clean JSON and CSV
This guide builds a small Python workflow for collecting a Google Finance quote page, then turning the rendered page into structured fields you can inspect, save, and compare over time. It is aimed at developers and analysts who need price, range, market-cap, volume, and valuation fields without manually copying each page.
Why Python fits this task
Python is a practical starting point because its syntax is readable and its ecosystem covers HTTP requests, HTML parsing, and tabular analysis. For this task you need two capabilities: a client that can render JavaScript and manage proxies, and a data library for turning extracted records into columns. The workflow below first fetches raw HTML to confirm access, then switches to structured extraction so the output is easier to validate.
Set up the environment and client
Confirm Python 3.6 or later is installed, then install the API client package configured as {{OWNER_API_PACKAGE}} together with pandas. Create or sign in to your account, copy your API key from the dashboard, and place it in the client variable shown in protected module code0002. Keep the key out of public repositories. Create a script file such as google_finance_scraper.py, import the client, pandas, and urlencode, and initialise the client with {{OWNER_API_KEY}}.
Build the quote URL and make a first request
Define a function in protected module code0003, prompt for a finance term such as a ticker and exchange code, and append it to the Google Finance quote base URL as shown in code0004. Add a JavaScript scenario in code0005 that waits 3000 milliseconds so dynamically loaded values can appear before extraction. In the first request, shown in code0006, enable the JavaScript scenario, custom Google handling, and premium proxy routing. Print the status code and response text using code0007; a 200 status indicates the page was reached, but raw HTML is only a diagnostic step.
Extract fields as JSON, then save as CSV
Open the target quote page in a browser, use Developer Tools to inspect the container and individual fields, and test each selector before adding it to code. The main container selector is introduced in code0008. The complete rule set in code0009 extracts Price, Difference today, Previous close, Day Range, Year Range, Market Cap, Avg Volume, and P/E Ratio. Because class names can change, prefer stable attributes such as jsname where available and re-test selectors after any layout change. Switch the result handling to JSON with code0010, convert the Finance_result list to a pandas DataFrame with code0011, print it for inspection, and append a CSV export step using result.csv so repeated runs can be compared in a spreadsheet or analysis tool.
Reliability checklist and failure diagnosis
A useful decision check is to separate three failure types. If the status code is not 200, treat it as an access or network problem and check proxy, throttling, and CAPTCHA handling. If the status is 200 but fields are empty, treat it as a selector or timing problem and re-inspect the rendered page. If JSON returns but values look wrong, treat it as a mapping problem and verify each output column against the visible page. For reliability, wait for JavaScript to finish, rotate proxies, space requests with exponential backoff, keep payloads minimal, and vary headers such as User-Agent and Accept-Language when blocks persist. The complete assembled example is preserved as protected module code0012.
| Request parameter | Purpose in this workflow | Module |
|---|---|---|
| js_scenario | Waits for rendered content before extraction | code0005 |
| custom_google | Enables specialised handling for Google pages | code0006 |
| premium_proxy | Routes requests through higher-quality proxy addresses | code0006 |
| extract_rules | Defines CSS selectors and JSON output fields | code0009 |
Parameter names and values are preserved from the protected modules; verify current API documentation before implementation.
Frequently asked questions
Is scraping Google Finance data allowed?
Publicly visible finance pages are described as generally scrapeable, while large-scale or commercial use may conflict with the platform's terms and can lead to restricted access. Check current terms for your intended use before running at scale.
How often can I request Google Finance pages?
There is no fixed safe frequency in the material. Rapid, repeated requests can trigger limits or blocks, so space requests, add backoff, rotate proxies, and throttle frequency rather than polling continuously.
Can I receive JSON instead of HTML?
Yes. Selector-based extraction rules can return structured JSON containing only the fields you define, instead of requiring you to parse the full HTML response.
What should I check when a request fails?
Timeouts, CAPTCHAs, and layout changes are common causes. Check the HTTP status code first, then inspect whether selectors still match. Retries, proxy rotation, and CAPTCHA handling in the API layer can reduce, but not eliminate, these failures.