Skip to content

Web Scraping with Selenium and Python: Setup, Waits, Parsing, and Scale

Selenium controls a real browser from Python, so it can execute JavaScript, click, scroll, and sign in where a plain HTTP request sees only an unfinished shell. That power costs speed and memory, so this guide shows both how to use it well and how to tell when a lighter tool is enough. Worked examples use a public book-catalogue sandbox and a practice login site, with protected modules code0001 to code0039 carrying the exact runnable snippets.

When a real browser is worth it

Selenium speaks to Chrome, Firefox, or Edge through WebDriver, so anything a person can do in a browser — navigate, type, click, scroll, wait, read rendered output — is reachable from Python. That makes it suited to JavaScript-heavy pages, while a full browser process is wasteful for static markup. Three checks decide the tool: if the data is already in the HTML returned to requests, parse it directly; if you need interaction but only a few values, Selenium alone suffices; if JavaScript renders many values, let Selenium render and hand the result to a parser. Playwright is a credible alternative for new projects wanting speed and automatic waiting, while Selenium retains the widest browser and language coverage and Grid support.

Setup, headless mode, and first checks

Create a virtual environment and install Selenium (module code0001); current releases need Python 3.10 or newer, plus an installed desktop browser. Since Selenium 4.6, Selenium Manager finds the browser, downloads the matching driver, and caches it, so webdriver.Chrome() needs no manual driver step (module code0002). For production headless runs, use the newer headless flag with a desktop viewport of 1920 by 1080, because the default 800 by 600 viewport triggers mobile layouts and misleading screenshots; in containers also add the no-sandbox and shared-memory flags shown in that module. Firefox follows the same pattern with its own options and driver resolution (module code0003). Verify by printing the title and window size (module code0005): a correct run reports the sandbox title and a 1920 by 1080 window, with the driver version reported by Selenium Manager and no manual download (module code0004). Open pages with driver.get(), remembering it waits for the load event rather than later JavaScript rendering, and always call quit() to avoid leaked browser processes (module code0006). Capture a viewport screenshot to confirm layout before trusting selectors (module code0008); a correct capture is 1920 pixels wide, though its height reflects the content viewport rather than the full window (module code0009). One boundary to plan around: save_screenshot() records just the on-screen viewport, so content below the fold is left out. Selenium's Chrome API offers no built-in whole-page capture, and reaching beyond the viewport means driving the DevTools protocol yourself; Firefox users have driver.save_full_page_screenshot() available when a complete-page image is required. Common failures are an old driver shadowing Selenium Manager, missing no-sandbox as root in Docker or CI, and small shared memory in containers.

Finding elements reliably

Older find_element_by_* helpers were removed in Selenium 4.3.0 and now raise AttributeError (modules code0010 and code0011); use find_element with a By strategy instead. The singular form returns the first match or raises NoSuchElementException, while the plural form returns all matches or an empty list (module code0012), so prefer the plural form when absence is possible and check its length. Prefer a stable ID, then CSS selectors for tag, class, and hierarchy, and reserve XPath for text, parent, or sibling relationships CSS cannot express (module code0013). A class-name locator accepts only one class; compound classes need CSS dot notation, otherwise an InvalidSelectorException results. Test selectors in DevTools with querySelectorAll before coding them. Against the sandbox, the h1 reads All products, twenty product cards are found, the first title is A Light in the Attic, and the first price is £51.77 (module code0014).

Reading, clicking, attributes, and forms

Each lookup returns a live WebElement handle, not a static copy, so it reflects later DOM updates but can become stale after a reload or DOM change, in which case locate it again. Read visible text with the text property, noting hidden elements return empty text while textContent gives raw text (module code0015). Click with click(), waiting first for clickability on dynamic pages to avoid interception errors (module code0016). Use get_attribute() for href, src, or full titles hidden behind truncated labels (module code0017). For forms, type with send_keys into each field, click submit, then wait for the result before reading it (module code0018); omitting that wait after a redirect is a frequent cause of NoSuchElementException. Confirm login success from a post-login element, wait for JavaScript-loaded fields, and only automate logins where site terms permit. The worked run navigates from the listing to the book detail page and lands on the secure area with its success message (module code0019).

Waiting, scrolling, and async JavaScript

Fixed sleeps waste time when pages are fast and fail when they are slow. WebDriverWait polls until a stated condition holds or a timeout raises TimeoutException, returning as soon as the element is ready (module code0020); explicit waits are preferable to blanket implicit waits. Match the condition to the next action: presence for existence, visibility before reading user-facing text, clickability before clicking, text presence for in-place updates, and presence of all elements for result sets. For infinite scroll, run a scroll script to the bottom, pause briefly for the request, and stop when scrollHeight stops growing (modules code0021 and code0022); the sandbox example grows from ten quotes to one hundred over ten scrolls (module code0024). Use execute_async_script() only when a value exists after in-page asynchronous work, passing the injected callback as the last argument (module code0023).

Pairing Selenium with BeautifulSoup

Repeated browser round trips make bulk extraction slow, so snapshot the rendered DOM with page_source and parse it in memory (module code0025). Iterate with find_all() to build records, using dot navigation for known structure, mapping class-encoded ratings to numbers, and testing element existence for boolean flags (module code0026); select() accepts the same CSS selectors used in DevTools. Quit the browser right after the snapshot to free the process before parsing (module code0027). The sandbox pipeline yields twenty records, the first being A Light in the Attic at £51.77, in stock, rated three (module code0028). Check first whether requests plus a parser can already see the data; only escalate to a browser when rendering or interaction is required.

Exporting, cleaning, and paginated tables

Write records with the csv module for spreadsheet use, keeping the empty-newline setting and UTF-8 encoding so Windows line breaks and characters such as £ survive (module code0029); the module handles quoting of commas and quotes. Use JSON when downstream code needs preserved types, with readable indentation and non-ASCII output kept literal (module code0030). Clean with pandas by stripping currency symbols and casting prices to float, dropping duplicates from overlapping pages, then summarising with mean and value counts (module code0031). For paginated tables, loop over page URLs, extract cells per row, and accumulate records (module code0032); the three-page example collects seventy-five rows, and pandas reports a mean price of £38.05 across the twenty books (module code0033). If a table is static, the same loop works without a browser, so choose per site.

Staying undetected: honeypots, CAPTCHAs, and speed

Basic defences are manageable, but serious anti-bot systems fingerprint below script control, so stealth tools raise the bar without guaranteeing passage. The source names four tools with distinct caveats: undetected-chromedriver, once the go-to, is increasingly caught by Cloudflare and DataDome now that they fingerprint for it directly; nodriver is the same author's official successor, an async-first rewrite that does not depend on Selenium at all; SeleniumBase includes a UC Mode wrapping these evasions in a maintained testing framework; and selenium-stealth patches the most obvious headless tells but is less actively maintained. Filter links and form fields through is_displayed() to skip hidden honeypots (module code0034), set a realistic User-Agent, add randomised delays, and reuse sessions. For CAPTCHAs, first remove triggering signals, then consider solver services, with manual solving only for one-off runs. Blocking images through a Chrome preference speeds text-only scraping while leaving data intact (module code0035); disabling JavaScript is only safe on confirmed static pages. The worked checks keep the visible link, drop the honeypot, and still parse all twenty records with images blocked (module code0036).

Scaling with Grid and proxies

Grid distributes sessions from one script across worker nodes for parallelism, cross-browser coverage, and load spreading; it earns its complexity only at high volume. A standalone Docker container provides a hub and Chrome node on port 4444 with enlarged shared memory (module code0037), and the client connects with webdriver.Remote using the same options and calls as a local run (module code0038). Route traffic through proxies for geo access and IP spreading, but note the proxy-server flag cannot carry credentials because Chrome shows a native auth dialog; a credentials extension is the robust workaround. The archived selenium-wire library is not recommended for new code; CDP-based tooling or the extension route is preferable. Rotate proxies per session on blocks, and consider a managed API when Grid, pool, and evasion maintenance outweighs the value of the data ({{OWNER_MANAGED_API_OFFER}}).

Selenium, Playwright, and BeautifulSoup comparison
AxisSeleniumPlaywrightBeautifulSoup
Renders JavaScriptYes (real browser)Yes (real browser)No (parser only)
SpeedSlowest (browser overhead)Faster than SeleniumFastest (no browser)
Auto-waitingNo (explicit WebDriverWait)Yes (built in)Not applicable
Async supportSync (async via wrappers)Native async and syncNot applicable
Browser and language supportWidest (all major browsers, 5+ languages, Grid)Chromium, Firefox, WebKitNot applicable
Ecosystem and ageLargest, oldest, most Q&ANewer, growing fastHuge for parsing
Situation: Data already in static HTMLRequests plus BeautifulSoupRequests plus BeautifulSoupRequests plus BeautifulSoup
Situation: Interaction needed, few valuesSelenium aloneSelenium aloneSelenium alone
Situation: JavaScript renders many valuesSelenium plus BeautifulSoupSelenium plus BeautifulSoupSelenium plus BeautifulSoup

Qualifications from the comparison in the source: Selenium is slowest due to browser overhead, Playwright is faster with built-in auto-waiting, and BeautifulSoup is fastest without a browser.

Frequently asked questions

Is Selenium still the best choice for web scraping?

Choose Selenium when broad browser and language coverage, existing Selenium code, or Grid distribution matters. For a new project where speed and built-in waiting are priorities, Playwright may fit better. If the needed markup is already present in static HTML, requests plus BeautifulSoup avoids browser overhead. The deciding question is whether a browser is actually required.

How can I avoid detection when using Selenium for web scraping?

Use a realistic User-Agent, vary timing between actions, reuse sessions, and check visibility with is_displayed() before interacting so hidden honeypots are skipped. First reduce bot signals that trigger CAPTCHAs, such as datacenter IPs; third-party solving services are an option when a challenge must be passed. These steps help against basic filters, not serious anti-bot fingerprinting.

How do I handle infinite scrolling websites with Selenium?

Run a scroll command through execute_script() to reach the bottom, allow new items to load, then compare scrollHeight before and after. Repeat until the height stops changing. That stopping point signals the end without assuming a fixed number of pages.

Is headless mode detectable by websites?

Yes, it can be. The newer headless mode resembles a visible browser more closely than the legacy mode, yet anti-bot systems may still spot differences in rendering, timing, or browser properties. It helps with basic checks; stronger protection may require stealth tooling or a managed service.

Can I run Selenium scrapers in the cloud?

Yes. It can run on a VM, in a container, in CI, or in a serverless setup with a headless browser layer. Use headless mode where no display exists, and in containers add the no-sandbox and shared-memory workarounds described in the setup section. Docker-based Grid is the usual route for spreading runs across machines.

How can I handle cookie consent popups or modals automatically?

Find the accept control by its visible wording and click it, matching text case-insensitively with XPath translate() rather than matches(), which browsers do not support in XPath 1.0. Because consent banners are not always present, guard the click with try/except.

Do I still need to download ChromeDriver for Selenium?

No. Since Selenium 4.6, Selenium Manager is included and fetches and caches the matching driver for an installed Chrome, Firefox, or Edge browser. Manual driver downloads, PATH edits, and webdriver-manager are reserved for unusual cases such as a custom Chromium build or a blocked download.