Skip to content

How to Scrape Baidu Search Results with Python

This walkthrough builds a Baidu scraper end to end: you enter a query, the script expands it into a paginated list of result URLs, each page is fetched through a rendering and proxy API, and the output lands in one CSV with a title, description and link per result. Three mechanics carry the whole pipeline — the wd query parameter, the pn page offset that steps by 10, and CSS selectors that you re-check whenever Baidu's class names change.

What you will build

The pipeline has four stages: build a Baidu search URL from a query term, expand it into a list of paginated URLs, fetch each page through an API that renders JavaScript and rotates proxies, and parse the returned HTML into a pandas DataFrame that is printed and saved as a CSV file. You need Python 3.6 or newer. The tutorial relies on two external libraries: pandas for tabular output, and the client SDK for the rendering/proxy API. The install command and the full working script are supplied as protected code modules and are not reproduced in prose here; the SDK package name is pending owner confirmation (OWNER_PACKAGE_NAME) and must not be published until resolved. Trial credit terms are likewise pending owner confirmation (OWNER_TRIAL_CREDITS) and are not asserted here.

Decision checklist before you start

Use this checklist to pick the right approach for your case:

  • Static HTML only? A plain HTTP request may work, but Baidu's result pages are JavaScript-heavy, so blank or partial HTML is a common outcome.
  • Need rendered results? Enable JavaScript rendering. In the API used here, headless rendering is on by default via a hidden render_js=true parameter on every Get API call.
  • Seeing 403s or CAPTCHAs? Route through premium or stealth proxies and set browser-like headers; rotating user-agents and IPs reduces the chance of tripping Baidu's filters.
  • Selectors returning nothing? Baidu uses dynamic class attributes, so CSS selectors need periodic re-checking against the live page.
  • Large-scale runs? Plan for proxy management; the API handles rotation for you per request.

Step 1 — Obtain an API key and initialize the client

Obtain an API key from your scraping API provider's dashboard and initialize the client with it. The client will not execute scraping logic without a valid key. Before writing the full script you can test a single request with curl, replacing the placeholder key. The exact curl and client-initialization snippets are provided as protected code modules; the API base origin (OWNER_BASE_URL) and key (OWNER_API_KEY) are owner-supplied values and must be filled in before publication.

Step 2 — Build the search URL and paginate with pn

Baidu accepts the query in the wd parameter. The first page URL is the base search URL with the encoded query; each additional page appends &pn= with an offset that grows by 10 per page. For a three-page iPhone run, the URL list is:

  • Page 1: https://www.baidu.com/s?wd=iphone
  • Page 2: https://www.baidu.com/s?wd=iphone&pn=10
  • Page 3: https://www.baidu.com/s?wd=iphone&pn=20

The script asks for a search term and a page count, encodes the query, seeds the URL list with page 1, and appends further URLs in a loop when more than one page is requested. The exact URL-building and loop snippets are supplied as protected code modules.

Python example

search_term = input("Search_term: ").strip()
Pagination = int(input("How many pages to scrape? "))

Python example

params = {"wd": search_term}

Step 3 — Send the request with rendering and proxy settings

The main function defines a JavaScript scenario that waits 4 seconds so the page can finish loading, then issues a GET request for the first URL in the list with premium_proxy enabled. The response text is printed along with the HTTP status code; a status of 200 indicates a successful connection. On Windows, the Command Prompt does not display Chinese characters by default — switch the console to UTF-8 with chcp 65001 and choose a Unicode-capable font such as NSimSun to read the output. The exact scenario and request snippets are provided as protected code modules.

Step 4 — Extract title, description and link

Extraction is defined with CSS-based rules. Find the container selector that wraps each result by inspecting the page in the browser's developer tools (F12 or right-click → Inspect); the search bar inside dev tools (Ctrl+F) helps confirm a selector's matches. The rules declare a list-type output with three fields per result: Title, Description and Link. The finished selector set is supplied as a protected code module. Because Baidu's class attributes change over time, re-inspect the page if extraction suddenly returns nothing for a query that previously worked.

Step 5 — Collect pages and export to CSV

An empty list collects one DataFrame per page. The loop iterates the URL list, sends each request with the extraction rules, the JavaScript scenario and the premium proxy, converts the JSON response into a DataFrame, and appends it. After the loop, the per-page DataFrames are concatenated into one, printed, and written to Baidu_extraction.csv with the index omitted. The final status code is printed as well. The exact loop and export snippets are provided as protected code modules.

Field and control map

Use this map to trace each input to its effect on the output:

InputWhere it is setEffect
search_termUser promptBecomes the wd query parameter
PaginationUser promptNumber of URLs in the list; page 1 is always included
pnLoop offsetPage 2 = 10, page 3 = 20, and so on
js_scenarioRequest paramsWaits 4 seconds before capturing HTML
premium_proxyRequest paramsRoutes the request through premium proxies
extract_rulesRequest paramsDefines the container selector and the three output fields
TitleOutput fieldCSS selector for the result title
DescriptionOutput fieldCSS selector for the result summary
LinkOutput fieldh3 > a@href — the result URL
Baidu_extraction.csvExportCombined rows from all pages

Distinguishing the three common failures

Not every empty result means the same thing, and the fix differs:

  • Blank or partial HTML with a 200 status. The page's JavaScript has not finished rendering. Increase the wait in the JavaScript scenario or confirm rendering is enabled.
  • 403 or a CAPTCHA page. Baidu is filtering the request. Switch to premium or stealth proxies, customize user headers to look like a normal browser, and rotate user-agents or IPs.
  • A 200 response with no extracted rows. The page loaded but the CSS selectors no longer match. Re-inspect the page and update the container and field selectors.

Treating these as one problem leads to wasted proxy spend or unnecessary selector churn.

Query language and result content

English search terms are accepted, but Baidu primarily indexes Chinese-language content, so most results and snippets will appear in Chinese. If you need broader or English-only data, plan to filter or translate the extracted text after scraping.

Frequently asked questions

Can a rendering and proxy API fetch Baidu without getting blocked?

The API used in this guide rotates IPs and simulates browser activity to reduce rate-limiting restrictions from the recipient server. That is the mechanism the tutorial relies on; it is not a guarantee that every request will succeed, so keep the failure-mode distinctions above in mind when a request returns a 403 or a CAPTCHA.

Do I need my own proxies to scrape Baidu?

For large-scale scraping, proxies are effectively required. With the API shown here you do not manage them yourself — proxy rotation and IP management are handled per request, so no external proxy deal is needed.

What is the fastest way to build a Baidu scraper in Python?

Use the Python SDK so proxy rotation and header management are handled for you, and start from the copy-ready example script. The alternative — raw requests plus your own rendering and proxy stack — is viable but requires you to solve blocking and JavaScript rendering yourself.

Does Baidu support English queries?

Yes, English terms are accepted. Baidu primarily indexes Chinese-language content, so most results and snippets will be in Chinese; filter or translate the extracted text if you need English-only data.

Why does my Baidu scraper return an empty CSV?

Check the failure mode first. A 200 response with no rows usually means the CSS selectors no longer match Baidu's current class attributes. A 403 or CAPTCHA means the request was filtered and needs proxy or header changes. Blank HTML means rendering did not complete before capture.