Scrape a Live Stock Price in Python
Stock pages rarely hand you a plain HTML quote. Prices are usually injected by JavaScript after the page loads, and the surrounding markup is designed to resist automated collection. This guide shows a practical path: send the target URL through a scraping API that renders JavaScript, then pull the price out of the returned HTML with an XPath that targets stable data attributes rather than fragile CSS class names. The result is a small Python script you can schedule to collect quotes for a portfolio.
What you will build
You will end up with a Python function that takes a ticker symbol, asks a scraping API to load the corresponding finance page with JavaScript enabled, and extracts the current price from the response. A second step writes each observation to a CSV row with a timestamp, company name, and symbol. The same pattern extends to multiple tickers, a SQLite history table, threshold alerts, and a scheduler that only runs during US market hours.
Prerequisites
You need Python 3.9 or newer, the requests library, and lxml for parsing. A scraping API account supplies the API key that authorizes requests. You should also be comfortable opening browser developer tools and reading HTML structure, because choosing the right selector is the part most likely to break later. The install commands are shown in the supplied code modules.
Why route finance pages through a scraping API
Finance sites load quotes dynamically and deploy bot detection that can block a plain requests call. A scraping API handles JavaScript execution, proxy rotation, browser fingerprinting, and CAPTCHA challenges on your behalf. That matters when you need a price that reflects the rendered page rather than a placeholder. The trade-off is that you depend on the API's rendering behavior and credit consumption, so the request parameters below are worth understanding before you scale up.
Step 1: Choose a target and a stable selector
Yahoo Finance is a reasonable starting point because its quote pages expose price data in a fin-streamer element with data-symbol and data-field attributes. Those attributes describe what the element means, so they tend to survive visual redesigns better than generated class names. The supplied HTML modules show the contrast: one example uses a long class string, the other uses data attributes. Prefer the data-attribute form.
Step 2: Make the request
The request is a GET to the API endpoint with the target URL encoded as a parameter. The table below maps the parameters you will use. Keep the API key server-side; do not embed it in client code.
Step 3: Parse and save the result
Parse the returned HTML with lxml, run the XPath for the price, and fall back to the value or data-value attribute if the text node is empty after a JavaScript update. Then write a row with a UTC timestamp, the company name from the page heading, the symbol, and the price. Appending to a CSV keeps a lightweight history; a pandas DataFrame gives you an in-memory table for downstream work. The supplied code modules contain the exact parsing and CSV-writing steps.
Hardening the script for real use
Three habits make the difference between a demo and a script you can leave running. First, wrap the fetch in a retry loop with exponential backoff so transient network errors do not end the run. Second, set country_code when the target serves different content by region, and enable premium_proxy on sites with aggressive bot detection. Third, extract only the fields you need instead of storing whole pages; this reduces memory use and makes each XPath's purpose obvious.
Extending the script
Once a single quote works, the natural next steps are a loop over a portfolio list, a SQLite table for historical prices, threshold-based alerts, and a scheduler. The supplied code modules include a multi-symbol function, a SQLite schema, an alert helper, and a schedule loop that checks US market hours in the America/New_York timezone before running. Treat these as building blocks rather than a finished trading system.
| Parameter | Purpose | When to use |
|---|---|---|
| api_key | Authorizes the request; keep it secret and server-side. | Every request. |
| url | The encoded target page URL, for example a Yahoo Finance quote page. | Every request. |
| render_js | Tells the API to execute JavaScript before returning HTML. | Pages whose price is injected after load. |
| country_code | Selects the proxy location. | When the target serves region-specific content or restricts access. |
| premium_proxy | Uses higher-reputation proxies. | Sites with sophisticated bot detection. |
Parameter names and behavior are taken from the supplied contract modules; availability and credit cost depend on the API plan.
| Approach | Example | Risk |
|---|---|---|
| Generated CSS class | span with a long class string | Class names change during redesigns. |
| Data attribute XPath | fin-streamer with data-symbol and data-field | Lower risk; tied to displayed data. |
| Text-node fallback | value or data-value attribute | Needed when JS updates leave the text node empty. |
Examples reflect the captured Yahoo Finance markup at the time of writing; verify selectors against the current page.
| Symptom | Likely cause | First check |
|---|---|---|
| Non-200 HTTP status | API or network error | Response body and status code. |
| Empty price list | Selector mismatch or page not rendered | Whether render_js was enabled and whether the XPath still matches. |
| Retries exhausted | Persistent block or outage | Proxy settings, country_code, and target availability. |
Diagnostic guidance, not a guarantee of resolution.
Frequently asked questions
Is scraping stock prices legal?
Publicly available stock prices can generally be collected, but you must respect the site's terms of service and robots.txt. Some sites forbid scraping entirely; others allow personal use but not commercial redistribution. Check the target's terms before you build a pipeline around it.
Do I always need JavaScript rendering?
For most modern finance sites, yes. Quotes are typically loaded after the initial HTML arrives, so a request without rendering returns placeholders or nothing. If a target serves prices in static HTML, you can skip rendering and save credits.
How do I avoid getting blocked?
Use the API's proxy rotation and browser fingerprinting, add rate limiting in your own code, avoid scraping during peak traffic, and enable premium proxies for sensitive targets. Retries with backoff also help you ride out temporary blocks without hammering the site.
Can I extract historical prices too?
Yes. Many finance sites publish historical data on a separate page. You will need to navigate there and adjust your selectors; some sites require clicking buttons or choosing date ranges, which the API's JavaScript execution can handle.
Why save to CSV instead of a database?
CSV is a quick way to inspect results and share them with spreadsheet users. Once you are collecting on a schedule, a database such as SQLite gives you indexed queries and safer concurrent writes. The supplied code modules show both.