Skip to content

How to extract data from a website using Selenium and Python

Selenium drives a real browser, so it can reach content that only appears after JavaScript runs.

OUTPUT FORMATSHTMLJSONMarkdown

The useful move is not to print the entire page source but to locate the one element you need and read its text. This guide shows the minimal pattern, explains each line, and flags the failure modes that produce empty output.

Start from the whole page, then narrow to one element

A common first step is to load the page, pause long enough for the dynamic content to appear, and print the full HTML. That confirms the browser reached the content, but it also returns markup you do not need. Once you know which element holds the data, switch to locating that element directly and reading its text property. The result is the visible text of that node rather than the surrounding HTML.

What each part of the extraction snippet does

The extraction snippet imports the WebDriver, the Chrome options class and the By locator helper. It builds headless Chrome options, starts the driver with the path to your ChromeDriver binary, navigates to the demo URL, sleeps for six seconds, locates the element with id 'content', prints its text, and quits the browser. The demo page is a placeholder URL supplied by the source; substitute your own target. The ChromeDriver path is also a placeholder you must fill in.

Worked example: reading the 'content' div

On the demo page, the relevant markup is a div with id 'content' whose text is 'This is content'. The full-page approach prints the entire document, including the doctype and html wrapper. The targeted approach prints only 'This is content'. That difference is the whole point of the technique: less output to parse, and a stable hook (the id) that survives layout changes better than positional selectors.

Decision checklist before you run it

Use this short checklist to avoid the most common empty-output cases:

  • Confirm the element exists in the rendered DOM, not only in the raw HTML response.
  • Prefer a stable locator such as an id or a data attribute over brittle CSS chains.
  • Decide whether a fixed sleep is acceptable or whether you need an explicit wait for the element to appear.
  • Check that the element is visible; hidden nodes may return an empty text value.
  • Keep the ChromeDriver path and browser version aligned.
  • Always call driver.quit() so the browser process does not linger.

Failure distinctions that matter

Three different problems can look like 'no data':

  • NoSuchElementException: the locator did not match anything in the current DOM. Usually a wrong id, wrong frame, or content that has not loaded yet.
  • Empty element.text: the element was found but has no visible text, often because it is hidden or its text lives in a child node you did not select.
  • StaleElementReferenceException: the element was found but the page changed afterwards. Re-locate the element after the update.

Telling these apart saves time, because each has a different fix.

Where to go next

If you want the broader Selenium workflow, including navigation and interaction, see the related Python Selenium scraping question. For a deeper treatment of Selenium with Python, the linked long-form article covers setup and patterns in more detail. The related Python web scraping questions below cover framework choice, library comparisons and learning paths.

Frequently asked questions

Why does element.text return an empty string?

The element was located but has no visible text. It may be hidden, or its text may sit in a child node. Inspect the rendered DOM and select the node that actually holds the text.

Is time.sleep the right way to wait for dynamic content?

A fixed sleep works for a demo but is fragile in production. An explicit wait for the element to appear is more reliable and usually faster.

Can I use a CSS selector instead of By.ID?

Yes. By.ID is one locator strategy; By.CSS_SELECTOR and By.XPATH are alternatives. Choose whichever gives the most stable hook for the page you are targeting.

What is the difference between page_source and element.text?

page_source returns the full serialized HTML of the page. element.text returns only the visible text of the located element, which is usually what you want for extraction.