Scrape TikTok Profile Stats and Video Data with Python
This guide shows two practical Python workflows: extracting public profile metrics and collecting per-video engagement data, with optional downloads. It is written for analysts, developers, social media managers, marketers, creators, and researchers who need repeatable data rather than manual screenshots.
Why collect TikTok data
TikTok data can reveal trend movement, audience response, and content performance. Analysts can track engagement, managers can compare competitors, marketers can judge campaign response, creators can review their own posts, and researchers can study interaction patterns. The useful question is not simply whether data exists, but which fields support a decision: follower and like totals for profile health, or views, comments, shares, bookmarks, tags, music, and posting date for individual videos.
Ethical checklist before coding
Start with TikTok's robots.txt and terms of service. The captured example treated general areas such as /foryou, /discover, and /about differently from restricted areas such as /inapp and /auth, so always re-check the current file rather than relying on that snapshot. Keep requests limited to what the task needs, avoid personally identifiable information, and comply with applicable privacy rules such as GDPR or CCPA where relevant. Use a profile whose owner has agreed to the exercise, as in the @fpl_insights example.
Set up the profile scraper
Use Python 3 in an isolated virtual environment so project dependencies do not interfere with other work. The profile workflow uses asyncio for asynchronous operations, BeautifulSoup for HTML parsing, and Nodriver for browser interaction. The supplied setup modules are code0001 for the virtual environment, code0002 for installing asyncio, nodriver, and beautifulsoup4, and code0003 for a short browser test that opens TikTok and waits before closing. A visible homepage load confirms the setup before profile work begins.
Map profile fields, then extract them
Inspect a profile in browser Developer Tools and record the stable data-e2e markers before writing extraction logic. Username uses user-title, display name uses user-subtitle, follower and following totals use followers-count and following-count, likes use likes-count, and the bio uses user-bio. These markers are a field map, not a guarantee: if extraction returns empty values, re-inspect the page because platform markup can change. The profile extraction modules are code0005 through code0011, covering imports, browser startup, a 10-second load wait, HTML capture, BeautifulSoup parsing, field extraction, cleanup, and the complete assembled script.
Save profile results in the right format
Choose storage according to the next task. JSON keeps the profile record readable and structured. CSV is convenient for spreadsheet review, especially when comparing several profiles. SQLite is better when you expect repeated queries or a growing collection. The supplied storage modules are code0012 for JSON, code0013 for CSV, and code0014 for SQLite. A practical decision rule is: JSON for interchange, CSV for quick analysis, SQLite for queryable history.
Collect video metrics and optional downloads
For each video, the target fields are URL, description, music title, posted date, tags, views, likes, comments, shares, and bookmarks. Views are collected from the profile grid, while the remaining fields come from the individual video page. The selector map includes video-views for views, like-count for likes, comment-count for comments, share-count for shares, undefined-count for bookmarks, search-common-link for tags, and specific class selectors for description and music. Posted date is taken from the last span inside the browser-nickname element. The video modules are code0015 through code0033, including Playwright and yt-dlp installation, imports, logging, human-like pauses, CAPTCHA pausing, extraction, optional downloading, progress saving, CSV export, and SQLite storage.
Handle dynamic pages, CAPTCHAs, and long runs
Playwright is used for the video workflow because TikTok pages load dynamically and need waiting, scrolling, and element interaction. Add randomized pauses and scrolling to reduce abrupt request patterns. If a CAPTCHA dialog appears, the supplied approach pauses for manual solving and resumes afterward, with a five-minute wait in the module. Keep DOWNLOAD_VIDEOS off unless downloads are actually needed; when enabled, yt-dlp saves files and the record gains a local filename. Save progress every ten videos so a long run can resume instead of starting over.
Final verification checklist
A reliable run has four checks: the browser test loads TikTok, profile selectors return non-empty values, video records include views plus engagement fields, and the chosen output file contains the expected columns or table. If a field is missing, distinguish a changed selector from a genuinely absent value before changing code. Keep the ethical checks in the workflow, limit collection to the stated purpose, and re-check platform rules and markup before reuse.
| Profile field | Selector marker |
|---|---|
| Username | user-title |
| Display name | user-subtitle |
| Followers | followers-count |
| Following | following-count |
| Likes | likes-count |
| Bio | user-bio |
Re-inspect if extraction returns empty values; TikTok markup can change.
| Video field | Selector marker |
|---|---|
| Views | video-views |
| Likes | like-count |
| Comments | comment-count |
| Shares | share-count |
| Bookmarks | undefined-count |
| Tags | search-common-link |
Views come from the profile grid; other fields come from the video page.
| Format | Best use |
|---|---|
| JSON | Structured interchange and tool compatibility |
| CSV | Spreadsheet review and quick comparison |
| SQLite | Repeated querying and growing datasets |
Choose one primary format for the immediate task; additional exports are optional.
Frequently asked questions
What should I check before scraping TikTok?
Check TikTok's robots.txt and terms first, collect only necessary public data, avoid excessive requests, and do not collect personally identifiable information. The example profile was used with its owner's permission, and video downloading requires permission and respect for copyright.
Which tools are used for profiles versus videos?
Profile extraction uses Nodriver with BeautifulSoup and asyncio. Video extraction uses Playwright for dynamic pages, with yt-dlp only when downloading is enabled. If selectors return empty results, inspect the page again because TikTok may have changed its HTML attributes or classes.
How should scraped data be stored?
JSON preserves structure, CSV suits spreadsheet review, and SQLite supports querying. For long video runs, save progress periodically so an interruption does not lose completed records.