Skip to content

How to use XPath selectors in Python

Python offers two common routes for XPath-based extraction: a lightweight fetch-and-parse workflow and a browser-driven workflow.

OUTPUT FORMATSHTMLJSONMarkdown

This guide explains when each fits, walks through the same first-heading example in both, and gives a short checklist for choosing between them.

Two approaches at a glance

There are two practical patterns. The first fetches page HTML over HTTP and parses it locally: Requests retrieves the markup, BeautifulSoup parses it, and lxml provides XPath querying over the resulting DOM. The second drives a real browser with Selenium and asks the browser to locate an element by XPath. Choose the first when static HTML is enough; choose the second when the page needs browser rendering or interaction before extraction.

Worked example: reading the first h1

Both examples aim at the same target: the text of the first h1 element on the page. In the lightweight workflow, the XPath query returns a list of matching nodes, so the code takes the first entry and reads its text. In the browser workflow, the locator returns a single element directly, and its text property gives the same result. The expected output in each case is the heading text shown in the protected example modules.

Decision checklist

Before writing code, confirm three things: whether the target content is present in the initially fetched HTML, whether a browser session is required to render or interact with the page, and whether you need one element or a list of matches. If static HTML contains the target, the Requests-based workflow is simpler and faster to run. If rendering or interaction is required, use Selenium and locate the element inside the live page. If your XPath can match several nodes, plan to iterate over the returned list rather than assuming a single result.

Choosing between the two XPath workflows
WorkflowFetch stepParsing / locating stepResult handling
Requests + BeautifulSoup + lxmlRequests retrieves the page HTMLBeautifulSoup parses it; lxml queries it with XPathXPath returns a list; take the first match and read its text
SeleniumBrowser loads the pageSelenium locates the element by XPath in the rendered pageSingle element returned; read its text property

Comparison of the two workflows described in the source; code details are in protected modules code0001 and code0002.

Frequently asked questions

When should I use Selenium instead of Requests with lxml?

Use a lightweight fetch-and-parse stack when the page HTML is available without browser interaction: fetch with Requests, parse with BeautifulSoup, then query the resulting DOM with lxml. Use Selenium when you need a real browser session, for example to interact with the rendered page before locating an element with an XPath locator.

What does the h1 example return?

In the worked example, both approaches target the first h1 element with the XPath //h1 and read its text. The Requests approach returns a list from the XPath call and takes the first item, while the Selenium approach finds a single element and reads its text property.

Where can I learn more about class-based XPath selection?

Yes. A related question covers selecting elements by class with XPath, and a broader tutorial covers practical XPath use for web scraping.