Scrape eBay with Python: Search Results to CSV
eBay search pages are heavier than they look: the markup is nested, the site watches request patterns, and some data only appears after client-side rendering. This guide walks through a Python pipeline that collects listing titles, prices and URLs from eBay search results into a CSV, then explains which request settings carry the load and where the approach needs adjusting.
Quick answer
A Python scraper for eBay search results needs three things: a request path that survives eBay's bot checks, a parser that targets the listing card markup, and pagination so you can move past page one. In the reference setup, the request is routed through a managed scraping client with premium_proxy enabled, device set to mobile, and render_js left off because search-result listings already appear in the initial HTML. The client rotates proxies for you, so the script only has to build the search URL, parse each card, and write rows out.
Setup: account and API key
Create an account with the scraping provider and copy the API key from the dashboard. The dashboard also shows trial expiry, available credits and concurrent connections, which are useful to check before running a multi-page job. The key authenticates every request, so keep it out of source control and load it from an environment variable in real projects.
The scraping pipeline, step by step
The reference script loops over page numbers, builds an eBay search URL with the query in _nkw and the page number in _pgn (page one omits it), sends the request through the client, and parses the response with BeautifulSoup. Each listing card is selected with li.s-card; cards carrying the srp-river-answer class are skipped because they are answer blocks rather than products. From each card the script reads the title from div.s-card__title span.su-styled-text, the price from span.s-card__price, and the link from a.s-card__link:not(.image-treatment). A card is kept only if it has a title and that title is not Shop on eBay. Results are written to ebay_products.csv with the columns title, price and url.
Request parameters that decide success
Four settings do most of the work in this setup:
premium_proxyset toTrueroutes through higher-quality residential proxies that are harder for eBay to detect and block.country_codepicks the country the request exits from, for exampleus.render_jsset toFalseskips JavaScript rendering. Search results are already in the initial HTML, so leaving it off saves credits and time. Turn it on only for pages that build their data client-side.deviceset tomobilerequests the mobile page, which is lighter and blocked less often than the desktop version.
Extracting titles, prices and shipping text
Once the HTML is in hand, BeautifulSoup selectors pull the fields you need. Titles come from div.s-card__title span.su-styled-text, prices from span.s-card__price, and shipping or delivery text from div.s-card__attribute-row rows whose text contains delivery or shipping. Selectors can be adjusted to whatever fields matter for your use case; CSS selectors and other navigation methods both work.
What you can collect from eBay
Beyond title and price, eBay pages expose several categories of data:
- Product information: title and description, current price and Buy It Now price, condition (new, used, refurbished), available quantity, item location and shipping options.
- Seller details: username and feedback score, positive feedback percentage, the seller's other items, and whether the seller is a business or an individual.
- Listing metrics: number of watchers, number of items sold, listing duration and end time, return policy details.
- Competitive data: similar items and their prices, best-offer availability, and auction bid history.
Why eBay resists simple scrapers
Plain HTTP requests against eBay tend to return incomplete data or nothing at all. The site identifies automated traffic by request patterns, headers and IP address, and it changes its HTML structure often enough to break scrapers that depend on fixed selectors. Several specific obstacles show up repeatedly:
- JavaScript-rendered content: product details, prices and images may load after the first response.
- Infinite scroll pagination: some pages load more results as you scroll rather than through numbered pages, which needs browser automation.
- Bot detection: IP-based rate limiting and behavioural analysis flag repetitive traffic.
- CAPTCHA challenges: frequent attempts trigger verification that halts collection.
- Shifting markup: CSS selectors and XPath expressions go stale as the page structure changes.
Avoiding blocks: a practical checklist
The common failure causes are default headers, no IP rotation, predictable request timing, and unnecessary JavaScript rendering. Work through these checks before scaling up:
- Send realistic headers. Python's
requestslibrary advertises itself with apython-requestsUser-Agent, which anti-bot systems flag immediately. Use a browser-like User-Agent plus headers such asAccept-LanguageandAccept-Encoding, and rotate User-Agent strings rather than reusing one. - Rotate IPs and pace requests. Dozens of requests per second from one IP is one of the fastest routes to a block. Randomise delays and rotate proxies; on a CAPTCHA or 429, back off exponentially instead of retrying immediately.
- Render JavaScript only where needed. Rendering is slow and costly. Search result pages usually carry their data in the initial response, while product detail pages often build pricing, shipping and seller information client-side. Test each page type with rendering on and off before defaulting to on.
- Consider a managed API. Proxy infrastructure, realistic headers and CAPTCHA handling all need ongoing maintenance as eBay's defences change. A managed service absorbs proxy rotation, fingerprinting, rendering and retries, so the script only handles parsing.
Legal considerations and best practice
eBay's Terms of Service restrict automated access to the platform. Scraping publicly available data for personal or research use is generally treated as legal or a grey area depending on jurisdiction, so the terms and local law both matter. To stay on safer ground: respect robots.txt directives, restrict collection to publicly accessible pages, avoid collecting personal data, and keep request volume low enough not to affect eBay's infrastructure. Commercial use deserves legal advice before you start.
| Parameter | Value used here | What it controls | Failure it prevents |
|---|---|---|---|
| premium_proxy | True | Routes through high-quality residential proxies | IP-based blocking and detection |
| country_code | us | Country the request exits from | Geo-mismatched results or routing |
| render_js | False | Whether a headless browser renders the page | Wasted credits and slow runs on pages that don't need rendering |
| device | mobile | Desktop vs mobile page version | Heavier desktop pages that are blocked more often |
Values reflect the reference setup for eBay search result pages; product detail pages may need render_js enabled.
| Format | Best for | Tradeoff |
|---|---|---|
| CSV | Spreadsheet import and simple flat records | Awkward for very large volumes |
| Parquet | Large volumes of data | Less convenient for quick manual inspection |
| JSON | Nested or structured records | Less convenient for spreadsheet tools |
CSV is the default in the reference script; the other formats are alternatives for different data shapes.
Frequently asked questions
Can I scrape eBay for commercial use?
Scraping is technically possible, but check eBay's Terms of Service before using it commercially, since many e-commerce sites prohibit automated collection for commercial purposes. The official eBay API is an alternative for commercial applications, and any scraping you do should comply with eBay's terms and rate limits.
Do I need proxies to scrape eBay?
Proxies are effectively required. eBay applies IP-based rate limiting and blocking, which stops single-IP scrapers quickly. A managed proxy feature rotates high-quality proxies automatically and keeps session consistency while avoiding blocks.
Does eBay use JavaScript to load content?
Not for search results. Listings appear in the initial HTML, so search pages can be scraped without JavaScript rendering. Product detail pages often load pricing, shipping and seller information client-side, so enable render_js for those to get the full page a browser would show.
Can I scrape seller ratings and feedback?
Yes. Navigate to the seller's profile page and extract the relevant elements with BeautifulSoup selectors. The approach mirrors product-listing scraping, but the CSS selectors target seller information instead of listing cards.
Can you scrape eBay sold and completed listings?
Yes. Add LH_Complete=1&LH_Sold=1 to the search URL to filter results down to completed and sold items. The same selectors continue to work, and the prices shown are final sale values.
How do you scrape eBay item descriptions that load in an iframe?
Extract the iframe's source URL first, then make a separate request to that URL. For eBay, that second request returns the HTML of the actual item description.
What is the best way to export scraped eBay data?
CSV is the most common choice and Python's csv module handles it natively; spreadsheets import it easily. For very large datasets, Parquet is usually a better fit. If the records are nested or structured, JSON is more convenient.
How do I tell an empty result from a block?
Check the response status before parsing. A 200 response with no li.s-card elements usually means the markup changed or the query genuinely returned nothing. A non-200 status, a CAPTCHA page, or a 429 points to blocking or rate limiting, which needs backoff and proxy rotation rather than selector changes.