How to web scrape with Python and Selenium
A plain HTTP request returns the HTML the server sends first.
If the text you want only appears after the browser runs JavaScript, that first response can contain an empty container instead of data. This guide shows the decision point, then walks through a small Selenium script that opens the page in a real browser, waits, and captures the rendered markup.
When Requests is enough, and when it is not
Python's Requests library is a good fit for static pages: the content you need is already inside the server's original HTML response. It is a poor fit when the page fills itself in with JavaScript after that response arrives. A useful test is to fetch the URL with Requests and inspect the target element. If the element is present but empty, the data is likely injected later by script, and a browser-driven tool such as Selenium becomes the practical option.
What Selenium adds
Selenium is a collection of open-source projects for browser automation, with bindings for all major languages, Python included. Its API drives a real browser such as Chrome, Firefox or Safari through the WebDriver protocol. Beyond fetching dynamically loaded data, that control lets a script act like a person: pressing buttons, completing forms, scrolling the page, running custom JavaScript, and similar interactions.
The example page and the empty-div symptom
The demonstration page used here inserts the text "This is content" five seconds after load. Requesting it with Requests yields markup whose content container is still blank, because the script that fills it has not run. That blank container is the signature of a rendering problem rather than a selector problem.
The script, step by step
The example starts a headless Chrome session, points it at the demo URL, pauses six seconds, prints the page source, and closes the browser. Read it as a control map rather than a fixed recipe:
- Options and headless: the browser runs without a visible window, which suits servers and scripts.
- executable_path: the location of the ChromeDriver binary on your machine must be supplied.
- get: navigates to the target URL, here the demo page that loads content after five seconds.
- sleep(6): a fixed pause longer than the five-second delay, so the injected text has time to appear.
- page_source: returns the current DOM as HTML, which is what you parse afterwards.
- quit(): shuts the session down and releases the browser process.
A fixed sleep is the simplest wait, but it is also the most fragile: too short and the content is missing, too long and every run is slower than necessary. Explicit waits that poll for a specific element are the sturdier alternative once the basic flow works.
What the rendered output looks like
After the wait, the same container now holds the injected text, so the printed source contains the data the initial response lacked. Compare the two outputs side by side: identical page structure, one difference — the content div is populated. That comparison is the clearest way to confirm the wait was long enough and the right element was targeted.
A short pre-flight checklist
Before scaling up, confirm each of these:
- The target data is absent from the raw HTTP response, not merely hidden by a selector you have not matched.
- A browser driver matching your installed browser version is available and its path is configured.
- The wait covers the slowest realistic load, or better, waits on the element itself.
- The session is closed on every path, including errors, so browser processes do not accumulate.
- You have checked the site's terms and robots guidance before automating it.
Going further
Once the basic pattern works, the natural next steps are explicit waits, handling multiple pages, and deciding whether a hosted scraping API fits better than maintaining browsers yourself.
| Situation | Requests | Selenium WebDriver |
|---|---|---|
| Content present in the server's first HTML response | Works | Works, but heavier |
| Content injected by JavaScript after load | Returns empty container | Returns populated DOM after waiting |
| Needs clicks, form fills, scrolling | Not supported | Supported |
| Needs custom JavaScript execution | Not supported | Supported |
Behaviour described reflects the demonstration page and general tool capabilities; real sites vary in timing and rendering.
| Line | Purpose | What to change for your site |
|---|---|---|
| Options() / headless = True | Run Chrome without a visible window | Keep for servers; disable when debugging visually |
| executable_path | Locate the ChromeDriver binary | Point to your installed driver |
| driver.get(url) | Navigate to the target page | Use your target URL |
| time.sleep(6) | Fixed pause longer than the 5-second demo delay | Replace with an explicit wait on the element |
| driver.page_source | Read the rendered HTML | Parse with your preferred HTML parser |
| driver.quit() | Close the browser session | Ensure it runs even on exceptions |
The six-second pause is tied to the demo page's five-second delay; it is not a universal timing rule.
Frequently asked questions
Why does Requests return an empty div on this page?
Because the text is inserted by JavaScript after the initial response. Requests only sees the server's first HTML, where the container is still blank.
Do I have to run Chrome headless?
No. Headless mode is convenient for scripts and servers, but you can run with a visible window while debugging so you can watch what the page does.
Is a fixed sleep the best way to wait?
It is the simplest. It is also brittle: it either wastes time or fails when the page is slow. Prefer waiting until the specific element you need is present.
Which browsers can Selenium control?
Chrome, Firefox and Safari are named in the original material, and Selenium provides bindings for all major programming languages including Python.
What does page_source give me?
The current DOM serialized as HTML, after any JavaScript has run. That is what you parse to extract the data.