Scrape a Public Twitter Profile with Python and Selenium
Twitter/X renders profile pages with JavaScript, so a plain HTTP request often returns markup without the fields you want. This guide walks through a Selenium-based workflow: launch Chrome, wait for the page and its tweet list to finish rendering, then pull profile and per-tweet data using stable data-testid attributes and XPath. The code blocks below are preserved verbatim from the referenced tutorial so you can adapt them directly.
When Selenium makes sense for Twitter data
The official Twitter API is rate-limited and requires an approval step before you can pull data. Scraping a public profile page with a browser automation tool avoids the API approval queue and rate limits, and lets you read the rendered profile page. The tutorial uses a standard Chrome window, not headless mode. Use Selenium when you need a small, targeted set of public profile fields; use the API when you need high-volume or authenticated access.
Prerequisites and project setup
The tutorial targets Python 3.10 but should work on most modern Python versions. Create a project directory and an app.py file, then install the two libraries the workflow depends on: selenium for browser control and webdriver-manager for automatic driver binary resolution. The exact shell commands and pip line are preserved in the protected code modules referenced below.
Launching Chrome and loading a profile
The first script imports webdriver, Service, and ChromeDriverManager, then constructs a Chrome driver whose binary is downloaded automatically. Passing ChromeDriverManager().install() avoids the manual executable_path approach, which requires re-downloading the driver binary whenever the browser updates. The final line navigates to a public profile URL. Expect a short delay on first run while the driver binary downloads.
Fields this workflow extracts
The tutorial targets eight profile-level fields and five per-tweet fields. Use the checklist below as a scope contract before you extend the script.
Waiting for JavaScript-rendered content
Two waits are layered. First, poll document.readyState via execute_script until it returns "complete", sleeping a random 3–5 seconds between checks. That alone is insufficient because tweets arrive via a later XHR request. Second, use WebDriverWait with expected_conditions.presence_of_element_located on the CSS selector [data-testid="tweet"], capped at 10 seconds. If the tweet element never appears, a WebDriverException is raised and the script prints a hint to disable headless mode for debugging.
Field-to-selector map
The table below is a reader-facing summary of the selectors used in the tutorial. It is not a regenerated schema; the exact code for each selector lives in the protected code modules referenced at the end of each row's section.
Extracting profile name and handle
Twitter generates class names randomly, so class-based selectors are unreliable. The div with data-testid="UserName" is unique on the page and contains the display name and handle separated by a newline. Read its .text, split on \n, and unpack into name and username variables.
Extracting bio, location, website, and join date
The bio lives in div[data-testid="UserDescription"]. Location, website, and join date each have their own data-testid attributes (UserLocation, UserUrl, UserJoinDate) and can be read with the same .text pattern. Because each attribute is unique on the page, no child traversal is needed.
Extracting follower and following counts
No data-testid exists for these counts, so XPath is used. The pattern finds the span containing the literal text "Following" or "Followers", walks up to the ancestor anchor tag, then descends to the first child span and reads its text. This works because each label appears only once on the profile page.
Extracting tweets and their stats
Use find_elements (plural) with [data-testid="tweet"] to collect every tweet on the page; find_element would return only the first. For each tweet, read the author block from div[data-testid="User-Names"], split its text into name, handle, a discarded separator, and timestamp. Tweet body text comes from div[data-testid="tweetText"]. Retweet, like, and reply counts come from div[data-testid="retweet"], div[data-testid="like"], and div[data-testid="reply"] respectively.
Telling retweets from original tweets
Compare the tweet author handle with the profile page handle. If they differ, the item is a retweet rather than an original post by the profile owner. The tutorial treats both as tweets for extraction purposes, but this comparison gives you a clean filter if you only want original content.
Failure distinction: NoSuchElementException vs WebDriverException
Two different failures mean two different things. A NoSuchElementException from find_element means the selector did not match at the moment you queried the DOM — usually because the page had not finished rendering. A WebDriverException raised by WebDriverWait means the explicit wait timed out after 10 seconds waiting for [data-testid="tweet"] to appear. The first is a selector/timing bug in your code; the second indicates the tweet element did not appear within the timeout. Disable headless mode to see what the browser is actually showing.
Complete script
The complete script combines the driver setup, both wait layers, all profile-field extractions, and the per-tweet loop into a single file. It is preserved verbatim in the protected code module so you can copy it directly and adapt selectors as the page markup evolves.
Extending the workflow
Two natural extensions are automated scrolling to load more tweets and authenticated sessions to reach content behind login. Both are described in the tutorial as follow-on work rather than implemented in the base script. If you would rather not build these yourself, an open-source Selenium-based Twitter scraper project on GitHub already handles auto-scrolling and media extraction; the tutorial links to it as a reference implementation.
| Field | Selector type | Selector value | Notes |
|---|---|---|---|
| Display name + handle | CSS | div[data-testid="UserName"] | Split .text on newline |
| Bio | CSS | div[data-testid="UserDescription"] | Direct .text |
| Location | CSS | span[data-testid="UserLocation"] | Direct .text |
| Website | CSS | a[data-testid="UserUrl"] | Direct .text |
| Join date | CSS | span[data-testid="UserJoinDate"] | Direct .text |
| Following count | XPath | //span[contains(text(), 'Following')]/ancestor::a/span | Label appears once on page |
| Followers count | XPath | //span[contains(text(), 'Followers')]/ancestor::a/span | Label appears once on page |
| Tweet author + timestamp | CSS | div[data-testid="User-Names"] | Split .text into 4 parts |
| Tweet text | CSS | div[data-testid="tweetText"] | Direct .text |
| Retweet count | CSS | div[data-testid="retweet"] | Direct .text |
| Like count | CSS | div[data-testid="like"] | Direct .text |
| Reply count | CSS | div[data-testid="reply"] | Direct .text |
Selector values reflect the page markup at the time of the original tutorial; verify against current markup before relying on them.
Frequently asked questions
Why do I get NoSuchElementException when I try to read a profile field?
The selector ran before the JavaScript-rendered content appeared. Add the document.readyState polling loop and the WebDriverWait on [data-testid="tweet"] before any find_element calls.
Why does the script time out waiting for tweets even though the profile loads?
WebDriverWait raises WebDriverException after 10 seconds if [data-testid="tweet"] never appears. Re-run with headless disabled to see the actual page state.
How do I get every tweet instead of just the first one?
Use find_elements with [data-testid="tweet"] rather than find_element. The plural form returns a list of all matching elements, which you then iterate.
How can I tell whether a tweet is a retweet?
Compare the tweet author handle (from div[data-testid="User-Names"]) with the profile page handle. If they differ, the item is a retweet.
Do I need to download the Chrome driver manually?
No. Passing ChromeDriverManager().install() to Service lets webdriver-manager download the matching binary and return its path automatically, so you do not need to hardcode executable_path.