Scraping Craigslist Search Results: A Practical Python Walkthrough
Craigslist search pages are a useful source of listing titles, prices, links and dates, but a plain request often runs into IP rate limits, CAPTCHA screens and user-agent checks. This walkthrough shows a Python workflow that fetches a search page, parses the result cards, and optionally returns structured JSON. It also covers how to add filters and pagination, and where the failure points usually appear.
Quick answer
A Craigslist search page can be fetched with an HTTP client that supports proxies and JavaScript rendering. The response HTML contains result cards with a title, price, link and date. You can parse those cards with BeautifulSoup, or define extraction rules and receive structured JSON directly. The example below targets a laptop search in New York City and limits output to the first three listings for readability.
Step 1: Install the client and set your API key
Install the client library and BeautifulSoup, then initialize the client with your API key. The key authenticates each request. Keep it out of source control and load it from an environment variable or secret store in real projects.
Step 2: Request a Craigslist search page
The target URL follows the pattern https://{location}.craigslist.org/search/{category}?query={term}. For a laptop search in New York City, the URL is https://newyork.craigslist.org/search/sss?query=laptop. A successful response returns HTML content; a non-200 status means the request did not complete and should be handled before parsing.
Step 3: Parse result cards with BeautifulSoup
Parse the HTML, select the result cards, and read the title, link and price from each card. The selectors used here target the typical Craigslist search-results structure. If the page layout changes or you target a different page type, adjust the selectors before relying on the output.
Step 4: Return clean JSON with extraction rules
Instead of parsing HTML manually, define extraction rules for the listing container and each field. The response then contains a structured list of listings with title, price, link and date. This is useful when the output feeds a CSV, a database or a downstream analysis step.
Step 5: Add filters and pagination
Craigslist filters are expressed through URL parameters. The helper below accepts a query, a location subdomain, a category code, a minimum price, a maximum price and a page number. Pagination uses the s parameter, with each page showing 120 results, so page n starts at n × 120. Space out requests to reduce the chance of CAPTCHA challenges or IP bans.
Why direct scraping usually fails
A bare script that hammers Craigslist tends to stall for a few predictable reasons. The site watches how many requests arrive from one address and will cut off an IP that floods it. Suspicious traffic can be met with a CAPTCHA that a simple scraper cannot clear. Repeated or obviously automated browser identifiers make a client easier to flag, and cookie-based session tracking adds another signal. When those checks trip, access from that address can be suspended for a while. Working around this reliably means spreading requests across many addresses, varying client identifiers, and rendering pages that depend on JavaScript. Those are operational capabilities, not something a plain requests loop provides. Whether a given scraping service supplies them, and under what terms, is a question for that provider's own documentation rather than an assumption.
Decision checklist: HTML parsing, extraction rules or a full API call
Use manual HTML parsing when you need custom logic per card, such as conditional fields or post-processing. Use extraction rules when you want a predictable JSON shape and less parsing code. Use a full API call with JavaScript rendering when the page depends on client-side rendering or when direct requests are consistently blocked. In all three cases, confirm that the selectors still match the current page before running a large batch.
Field and parameter map
The workflow uses the following inputs and outputs. Inputs: query (search term), location (Craigslist subdomain), category (category code such as sss for all), min_price, max_price and page (0-based). Outputs per listing: title, price, link and date. The pagination offset is page × 120. A non-200 response and a JSON decode error are different failures: the first means the request did not succeed, the second means the response body was not valid JSON.
Worked example: laptops in New York City
For the URL https://newyork.craigslist.org/search/sss?query=laptop, the parser selects result cards and prints the first three titles, prices and links. The same URL can be sent with extraction rules to return a JSON list containing title, price, link and date. If the request returns a non-200 status, stop and inspect the status before parsing. If the status is 200 but JSON parsing fails, inspect the raw response body.
Legal and rate considerations
This section is general information, not legal advice. Accessing public data is not automatically unlawful, but Craigslist's terms prohibit unauthorized scraping. The source guidance suggests that personal use is less likely to create legal exposure than commercial collection or gathering contact details; treat that as a starting point for your own review, not a guarantee. For large projects, read the site's terms and consult legal counsel. On pacing, the source recommends roughly one to two requests per second for small projects and wider spacing for larger datasets. Whether a particular scraping service throttles requests for you is a provider-specific claim to verify rather than assume.
Frequently asked questions
Can I legally scrape Craigslist?
Web scraping sits in a legal gray area. Accessing public data is not inherently illegal, but Craigslist's terms prohibit unauthorized scraping. The source guidance suggests personal use is less likely to cause legal consequences than commercial use or collecting contact details; that is general information, not legal advice. For large-scale projects, consult legal counsel and review the site's terms first.
How often can I scrape Craigslist?
The source recommends adding reasonable delays between requests. For small projects, one to two requests per second is generally safe. For larger datasets, spread requests over time to mimic human behavior. Whether a scraping service handles rate limiting automatically is a provider-specific claim to confirm in its documentation.
Does Craigslist have a public API?
No. Craigslist does not offer a public API for developers. It previously had an experimental API but discontinued it. Web scraping remains the primary way to access Craigslist data, and a scraping service can act as an alternative to direct API access.
How do I avoid being blocked when scraping Craigslist?
Use residential proxies with automatic IP rotation, add reasonable delays between requests, and avoid collecting personal information such as phone numbers. If those measures are too complex to maintain, a scraping service may handle the technical side; confirm the specific capabilities with the provider.