Scrape Indeed Job Listings with Python and BeautifulSoup
A plain request to Indeed often ends at a CAPTCHA or block, so a reliable workflow separates three jobs: fetching rendered pages through an API, parsing job cards with BeautifulSoup, and cleaning the result before export. This guide explains that sequence, the fields you can realistically get from search results, and the checks to run before scaling it.
Inspect the page before writing the scraper
Start with a search-results page rather than an individual posting. One results request can return many cards, while a detail page returns depth for a single job. Open the results page in a browser, open developer tools, and inspect a card. The card wrapper and the inner elements for title, company, and location tell you exactly which selectors to target. Indeed paginates with a start parameter: the first page uses start=0, the next uses start=10, and so on. Your loop should advance that value instead of re-fetching the same page.
Set up Python and your API key
Install Python with pip available, add the HTTP and parsing libraries, obtain an API key, and create a Python file that holds the key and imports. The setup module for imports and the key placeholder is code0003. Keep the key out of shared files and replace the placeholder only in your local environment.
Fetch pages with pagination
For each page, build the Indeed search URL from the query, location, and start offset, then request it through the API with the key and target URL. The pagination and request logic is code0004. JavaScript rendering matters for this site; when using the API workflow described here it is enabled by default, so you do not need to manage a separate browser. Check the status code, parse only successful responses, and log skipped pages instead of failing the whole run.
Parse cards and extract core fields
Pass each HTML response to BeautifulSoup and select every job-card wrapper. Inside each card, read the title, company, and location elements and build one dictionary per posting. That extraction logic is code0005. Treat selectors as fragile: Indeed can change its markup, so if a field returns nothing, re-inspect the live card structure before changing the rest of the pipeline.
Clean the data before export
Raw scraped records often have missing values, extra whitespace, or inconsistent keys. Normalize each record to the same three fields, trim text, and substitute a clear placeholder such as N/A when a value is absent. The cleanup function is code0006. This step is what makes later CSV and JSON output predictable for spreadsheets, databases, or analysis tools.
Export to JSON or CSV
Print cleaned records as indented JSON for a quick visual check, using code0007, or save them to CSV for tools such as Excel or Google Sheets, using code0008. Guard against an empty list before writing a header. A useful decision check is: if you only need a quick inspection, use JSON; if you need to sort, filter, or share the dataset, use CSV.
Scale responsibly: delays, region, and errors
Add a short delay between page requests to respect rate limits, even when proxy rotation is handled for you; the delay example is code0009. Set country_code to match the region whose listings you want, as shown in code0010, because Indeed results vary by location. Wrap requests in error handling and add retry logic for failures, as shown in code0011. For a larger run, combine pagination, cleanup, and CSV export in one workflow, as shown in code0012: ten pages at roughly ten results per page gives about one hundred postings.
What you can extract, and what needs a second stage
Search-result pages reliably provide job title, company name, location, and job URL. Those four fields support aggregation, tracking, and market analysis. Salary, full description, and requirements are not on the card; collect them in a second stage by requesting each job URL separately. That two-stage design gives you breadth first and depth only where it is worth the extra requests.
Why collect Indeed listings?
Market research uses repeated snapshots to track salary trends, in-demand skills, and hiring shifts. Job aggregation feeds a dedicated board or a single interface for niche roles. Lead generation looks for companies posting many roles as a signal of growth. Automation replaces daily manual checking with alerts for new postings matching a role or location. Choose the use case first, because it determines whether card-level fields are enough or detail pages are required.
Check robots.txt and site policy first
A robots.txt file is a widely respected guideline for automated tools, not a legal contract. Indeed disallows several job-related paths and patterns, including resume directories, application start pages, and certain company pages, while some search-engine bots receive broader access. Before running any scraper, check the current robots.txt, respect rate limits and server load, and avoid disallowed paths. Proxy rotation and rate management help you request more responsibly, but they do not replace following site policies.
| Stage | Input | Output | Check |
|---|---|---|---|
| Fetch | Query, location, start offset | Rendered HTML per page | Status code is successful |
| Parse | HTML response | Title, company, location per card | Selectors still match live markup |
| Clean | Raw dictionaries | Consistent records with placeholders for missing values | Every record has the same keys |
| Export | Cleaned records | JSON for inspection or CSV for analysis | File is non-empty and readable |
Card-level fields only; salary and description require individual job pages.
Frequently asked questions
Does Indeed allow web scraping?
Indeed's terms do not explicitly permit scraping. Respect its robots.txt directives, avoid overloading servers, and consider data-protection rules such as GDPR or CCPA if personal data could be involved. Focus on publicly available listing data, avoid unnecessary personal data, and do not reproduce full copyrighted descriptions at scale without a clear legal basis. Violations can lead to IP bans, blocked requests, account restrictions, or legal risk.
How does Indeed gather jobs?
Indeed crawls employer career pages and job boards, receives direct employer posts, and receives listings through applicant tracking systems via XML feeds or API integrations. Search results draw from that aggregated index rather than visiting each employer site at search time.
How can I reduce the chance of being blocked?
Space requests, vary timing, limit concurrency, and use a service that handles proxies and browser-like requests. Add delays between pages, use error handling with retries, and choose a country_code matching the region you want to target.
How can I scrape jobs by keyword or location?
Change the query and location values used to build the search URL. For example, a machine-learning search in Chicago is expressed by setting those two parameters rather than changing the parsing logic.
What is the best way to extract salary or description?
Use a two-stage approach: first collect job URLs from search results, then request each individual job page and parse it for salary, description, requirements, and other detail fields. This keeps the initial pass lightweight and reserves extra requests for postings you actually need in detail.