Web Scraping Questions
This hub collects short answers to the questions developers hit most often when building a scraper.
Each topic below links to a deeper page that covers the tool or concept in detail. Use the decision table to find the right starting point, then follow the topic link for the full explanation.
How to use this hub
Start with the symptom or goal you are facing, not the tool name. If your requests are being blocked, the proxy and anti-blocking topics are the fastest route. If the page content only appears after JavaScript runs, look at the headless-browser topics. If you already have HTML and need to pull fields out of it, the parsing and selector topics are the right entry point. Each topic page expands on the short answer here.
Which topic fits your task?
Use this table to match your situation to the most relevant topic page.
Static HTML versus rendered pages
A plain HTTP client such as Python requests or Guzzle returns the HTML the server sends. If the data you need is already in that HTML, a parser like BeautifulSoup or Cheerio is enough. If the data is inserted by JavaScript after load, you need a browser-driven tool such as Playwright, Puppeteer, or Selenium to render the page first. The topic pages for each of these tools explain the setup and common pitfalls.
Selectors and parsing
Once you have HTML, you locate the fields you want with CSS selectors or XPath. CSS selectors are usually shorter and easier to read for class and attribute matching. XPath can walk up and across the tree and is useful when the structure is irregular. The CSS selector and XPath topic pages cover syntax, common expressions, and how to test them.
Blocking and proxies
Sites may rate-limit, fingerprint, or challenge automated requests. Proxies spread requests across addresses, and rotating proxies change the address between requests. The proxy and anti-blocking topic pages explain the options and the trade-offs. A managed scraping API can also handle headless browsers and proxy rotation on your behalf, which is one way to reduce the maintenance burden.
Languages and tools
The same scraping concepts appear across languages. The topic list includes guides for Python, PHP, Java, Ruby, Node.js, R, C#, C++, Elixir, Perl, Rust, and Go, plus tool-specific pages for cURL, JSON, HTTP, and data parsing. Pick the language you already use and follow its topic page.
| Situation | Recommended topic | Why |
|---|---|---|
| Requests are being blocked or rate-limited | Anti-blocking and proxies | Covers detection and address rotation |
| Page content appears only after JavaScript runs | Playwright, Puppeteer, or Selenium | These tools render the page in a browser |
| You have HTML and need specific fields | CSS selectors, XPath, BeautifulSoup, or Cheerio | Parsing and selection tools extract values from markup |
| You need to convert a cURL command | cURL topic | Explains cURL usage and conversion |
| You need to work with JSON responses | JSON topic | Covers parsing and handling JSON |
| You are scraping with a specific language | Language topic (Python, PHP, Java, etc.) | Each page covers setup and examples for that language |
| You are using PHP's DOM crawler or Guzzle | DOM crawler or Guzzle topic | Tool-specific parsing and HTTP guidance |
| You need to understand crawling versus scraping | Web crawling topic | Explains the distinction and when to use each |
The topic list reflects the categories present on this hub; individual topic pages may cover additional details.
Frequently asked questions
What is this page for?
It is a starting point for common web scraping questions. Each topic links to a deeper page that covers the tool or concept in more detail.
Should I start with requests or a headless browser?
If the data is present in the HTML returned by the server, a plain HTTP client plus a parser is usually simpler. If the data is added by JavaScript after load, use a browser-driven tool such as Playwright, Puppeteer, or Selenium.
CSS selectors or XPath?
CSS selectors are concise for class and attribute matching. XPath is more flexible when you need to traverse upward or match by text or position. The dedicated topic pages show examples of each.
Do I need proxies?
Proxies help when a site limits or blocks repeated requests from one address. The proxy and anti-blocking topic pages explain the options and trade-offs.