Skip to content

Web Scraping Questions

This hub collects short answers to the questions developers hit most often when building a scraper.

OUTPUT FORMATSHTMLJSONMarkdown

Each topic below links to a deeper page that covers the tool or concept in detail. Use the decision table to find the right starting point, then follow the topic link for the full explanation.

How to use this hub

Start with the symptom or goal you are facing, not the tool name. If your requests are being blocked, the proxy and anti-blocking topics are the fastest route. If the page content only appears after JavaScript runs, look at the headless-browser topics. If you already have HTML and need to pull fields out of it, the parsing and selector topics are the right entry point. Each topic page expands on the short answer here.

Which topic fits your task?

Use this table to match your situation to the most relevant topic page.

Static HTML versus rendered pages

A plain HTTP client such as Python requests or Guzzle returns the HTML the server sends. If the data you need is already in that HTML, a parser like BeautifulSoup or Cheerio is enough. If the data is inserted by JavaScript after load, you need a browser-driven tool such as Playwright, Puppeteer, or Selenium to render the page first. The topic pages for each of these tools explain the setup and common pitfalls.

Selectors and parsing

Once you have HTML, you locate the fields you want with CSS selectors or XPath. CSS selectors are usually shorter and easier to read for class and attribute matching. XPath can walk up and across the tree and is useful when the structure is irregular. The CSS selector and XPath topic pages cover syntax, common expressions, and how to test them.

Blocking and proxies

Sites may rate-limit, fingerprint, or challenge automated requests. Proxies spread requests across addresses, and rotating proxies change the address between requests. The proxy and anti-blocking topic pages explain the options and the trade-offs. A managed scraping API can also handle headless browsers and proxy rotation on your behalf, which is one way to reduce the maintenance burden.

Languages and tools

The same scraping concepts appear across languages. The topic list includes guides for Python, PHP, Java, Ruby, Node.js, R, C#, C++, Elixir, Perl, Rust, and Go, plus tool-specific pages for cURL, JSON, HTTP, and data parsing. Pick the language you already use and follow its topic page.

Topic decision table
SituationRecommended topicWhy
Requests are being blocked or rate-limitedAnti-blocking and proxiesCovers detection and address rotation
Page content appears only after JavaScript runsPlaywright, Puppeteer, or SeleniumThese tools render the page in a browser
You have HTML and need specific fieldsCSS selectors, XPath, BeautifulSoup, or CheerioParsing and selection tools extract values from markup
You need to convert a cURL commandcURL topicExplains cURL usage and conversion
You need to work with JSON responsesJSON topicCovers parsing and handling JSON
You are scraping with a specific languageLanguage topic (Python, PHP, Java, etc.)Each page covers setup and examples for that language
You are using PHP's DOM crawler or GuzzleDOM crawler or Guzzle topicTool-specific parsing and HTTP guidance
You need to understand crawling versus scrapingWeb crawling topicExplains the distinction and when to use each

The topic list reflects the categories present on this hub; individual topic pages may cover additional details.

Frequently asked questions

What is this page for?

It is a starting point for common web scraping questions. Each topic links to a deeper page that covers the tool or concept in more detail.

Should I start with requests or a headless browser?

If the data is present in the HTML returned by the server, a plain HTTP client plus a parser is usually simpler. If the data is added by JavaScript after load, use a browser-driven tool such as Playwright, Puppeteer, or Selenium.

CSS selectors or XPath?

CSS selectors are concise for class and attribute matching. XPath is more flexible when you need to traverse upward or match by text or position. The dedicated topic pages show examples of each.

Do I need proxies?

Proxies help when a site limits or blocks repeated requests from one address. The proxy and anti-blocking topic pages explain the options and trade-offs.