Skip to content

Scrape Yahoo Search and Yahoo Finance with Python

Yahoo Search and Yahoo Finance both serve dynamic pages that a plain HTTP request often cannot read reliably. This guide shows a Python workflow that sends each target URL to a rendering and proxy API, applies CSS selectors, and writes the extracted rows to a CSV file. You can use the same script for search-result pages and for a single stock quote summary, then adapt the selectors for other Yahoo sections.

What this workflow extracts

The script has two modes. Search mode collects the title, link and description of each organic result container on a Yahoo Search page, and can repeat that across several result pages. Finance mode collects the quote-summary fields for one ticker, including previous close, open, bid, ask, day's range, 52-week range, volume, average volume, market cap, beta, P/E ratio, EPS, earnings date, forward dividend and yield, ex-dividend date, and the one-year target estimate. Both modes return JSON from the API, which pandas turns into a table and saves as a CSV file.

Before you start: a short decision checklist

Use this checklist to avoid the most common setup mistakes:

  • Confirm Python 3.6 or newer is installed and that pip can reach the package index.
  • Decide whether the task needs search results, a finance quote summary, or both. The supplied script runs one mode per execution.
  • Have an API key ready and store it outside version control.
  • Open a Yahoo page in a browser and confirm the selectors still match the current markup before relying on a long run.
  • Decide how many search pages you need; each additional page adds another API call.
  • Check Yahoo's terms and your local rules before commercial or high-volume use.

Install the Python packages

Install the client library and pandas together. The client library sends requests to the rendering API, and pandas handles the tabular output. The exact install command is preserved in the accompanying code module.

Initialize the client and build Yahoo Search URLs

Create a client object with your API key, then build a search URL from the base address and an encoded search term. The supplied pagination logic starts at the first page and adds a page parameter in steps of ten for each additional page. The exact client initialization and URL-building code are preserved in the accompanying code modules.

Set the JavaScript wait and extraction rules

A short browser scenario waits three seconds so the page can finish rendering before extraction begins. The search extraction rule uses one container selector for each result and then reads the heading, link and description inside it. For finance pages, the rule targets the quote-summary list and reads each labeled value. The exact selectors and field names are preserved in the accompanying code modules.

Field and control map

The table below explains what each request parameter changes. It is a reading aid, not a replacement for the exact code modules.

Loop over pages and export to CSV

For each URL in the list, the script sends a GET request with the extraction rules, the browser scenario, the premium proxy flag and the resource-blocking flag. Each JSON response is converted to a DataFrame and appended to a list. After the loop, the frames are concatenated and written to result.csv. The finance mode follows the same pattern and writes result_finance.csv. The exact loop and export code are preserved in the accompanying code modules.

Choose a mode at runtime

The final script asks whether to run search mode or finance mode and calls the matching function. A finance run for the Apple ticker produces a terminal printout and a CSV row set with the same quote-summary fields. Treat the displayed values as a reference snapshot from the time of capture, not as current market data, and re-run the script when you need fresh figures.

When a run fails: two different problems

A blocked request usually shows up as a CAPTCHA page, an HTTP error, or an empty response body. A selector problem looks different: the request succeeds, but the extracted list is empty or missing fields. Check the response status first, then inspect the page markup and update the selector. Keeping those two failure modes separate saves time, because retrying with a different proxy will not fix a stale selector and editing selectors will not fix a block.

Request parameter reference
ParameterWhat it controlsTypical setting in this guide
extract_rulesCSS or XPath selectors for the fields to returnSearch or finance rule set
js_scenarioBrowser actions to run before extractionWait 3000 ms
premium_proxyUse residential-quality proxy networkTrue
block_resourcesSkip images and heavy assetsFalse

Parameter names and meanings are taken from the supplied tutorial. Values shown are the ones used in the example, not universal recommendations.

Finance quote-summary fields collected
Field groupFields
Price contextPrevious Close, Open, Bid, Ask
RangeDay's Range, 52 Week Range
VolumeVolume, Avg. Volume
ValuationMarket Cap (intraday), Beta (5Y Monthly), PE Ratio (TTM), EPS (TTM)
Events and incomeEarnings Date, Forward Dividend & Yield, Ex-Dividend Date
Analyst estimate1y Target Est

Field list reflects the selectors in the supplied example. Yahoo may change its markup or field set; verify against the live page before a production run.

Frequently asked questions

Can I scrape Yahoo Search with Python?

Yes. The supplied workflow sends each search URL through the rendering API with a container selector and returns title, link and description for each result. You need an API key and the client library installed.

Does Yahoo block scrapers?

Yahoo uses bot detection, IP rate limiting and CAPTCHA challenges. The rendering API in this guide handles proxy rotation and browser rendering, but you should still expect occasional failures and design retries or logging around them.

What Yahoo Finance data can I collect?

The example collects the quote-summary fields listed in the field table, including price context, ranges, volume, valuation ratios, earnings date, dividend information and a one-year target estimate. Other Yahoo Finance pages need their own selectors.

Is scraping Yahoo legal?

Scraping publicly available pages is generally discussed as permissible, but Yahoo's terms, local law and the purpose of the collection all matter. Review the terms and seek legal advice before commercial or large-scale use.

Why did my extraction return an empty list?

An empty list usually means the selector no longer matches the page markup. Open the page in a browser, inspect the result container, and update the selector. If the response itself is an error or CAPTCHA page, fix the request path first.