Skip to content

Scrape Job Postings: Build a Structured Job Board Pipeline

Job boards are one of the most useful public data sources for anyone tracking hiring demand, and one of the most frustrating to scrape reliably. Search pages paginate, cards load late, and the same role can appear with a salary in one place and a range in another. This guide walks through a repeatable pipeline: choose the right board type, define the fields you want, run an AI-assisted extraction request, then clean and store the output so it is actually usable.

Start with the board type, not the code

Before writing a request, decide which kind of board you are targeting. The three families below behave differently and need different handling. The classification is a planning aid, not a claim that every site in a category behaves identically.

Set collection goals before you scale

Decide how often you need fresh data and how deep each record must go. A daily refresh for competitive tracking needs different scheduling than a weekly snapshot for trend analysis. Depth matters too: basic title/company/location records are cheap; full descriptions, salary ranges, and application deadlines are heavier and more prone to missing values. Posting dates and salary fields are usually the least consistent across boards, so plan a normalization step for them from the start.

Set up a clean Python environment

Use a virtual environment so package versions do not leak between projects. The commands below create and activate one.

Next, install the Python SDK for the extraction service. The package name is owner-confirmed and published as {{OWNER_SDK_PACKAGE_NAME}}; the SDK handles authentication, request formatting, and response parsing so your script can focus on the data you want. The client class used in later sections comes from that package, so install it before running any request code.

Define extraction rules as a schema

Instead of writing CSS selectors that break on every redesign, describe the fields you want and their types. The rules object below is the contract your request will carry: job_title, company, location, salary_range, required_skills, and posting_date. Treat it as the single source of truth for downstream cleaning and storage.

Send the extraction request

The request below targets a SimplyHired search URL and passes the rules plus a natural-language query. render_js is set to true because many boards load listings with JavaScript; keeping it on is a safe default even when a specific page renders server-side.

Loop over keywords and locations

Most real projects need more than one search. The function below loops over every keyword and location pair, builds the SimplyHired search URL, sends the same extraction rules with a keyword-specific query, parses the JSON body on a 200 response, and prints the status code on failure so you can see which combination broke. The usage example after it shows the shape of a multi-keyword, multi-location run. When you scale this up, add a short delay between requests and handle non-200 responses explicitly rather than silently dropping them.

What a clean record looks like

Before trusting the output, inspect it. The first snippet prints the first three collected listings as indented JSON so you can eyeball the shape. The sample output after it shows the target record: a title, company, location with work mode, salary range, a skills array, and an ISO-style posting date. Use it as a fixture when testing your cleaner.

Clean and normalize before storing

Raw extraction is only half the job. The cleaner below standardizes text fields, collapses remote locations, and rewrites salary strings into a consistent range format. Keep the original value alongside the normalized one so you can audit changes later.

Store in a format that matches how you will query

CSV is fine for quick spreadsheet review; SQLite is better when you need joins, filters, or dashboards. The storage class below supports both. Pick based on the questions you expect to ask, not on habit.

Optional: discover listings through search results first

An alternative to crawling each board directly is to use a search API to find candidate job URLs, then run the AI extraction on each URL. The discovery function below sends a query like "<job title> jobs <location>" to the search endpoint at {{OWNER_GOOGLE_SERP_ENDPOINT}}, requests structured data only (add_html off), and keeps organic results whose URLs point at known job-board domains such as linkedin.com/jobs, indeed.com, or glassdoor.com.

Once you have URLs, the extraction loop below runs the same rules object against each one and adds a wait parameter so lazy-loaded content has time to appear. The orchestrator then ties discovery and extraction together, and the final usage example calls it for a single title and location.

Handle login-gated boards with a scenario

Some boards require a session before listings appear. A JavaScript scenario can drive the login: wait for the email input, fill the email and password fields, click the submit button, then wait for the listings container to appear. Pair the scenario with custom headers and, where appropriate, premium proxies. Never hard-code credentials in the script; load them from environment variables or a secrets store.

Pre-flight checklist

Run through this before your first production request:

  • Confirm the board type (traditional, company career page, or niche) and note whether listings load from the URL or require JavaScript.
  • Decide update frequency and required depth; write both down.
  • Define the extraction rules object and freeze it as a schema version.
  • Set render_js and, if needed, a wait value.
  • Add retry and delay logic for non-200 responses.
  • Choose CSV or SQLite based on the queries you expect.
  • Store raw and normalized values side by side.
  • Keep credentials out of source control.

Telling failures apart

Three failure modes look similar but need different fixes. A block usually returns a challenge page or an unexpected status; check the response body and status before changing your rules. Missing content often means the page needed more time or JavaScript; increase wait or confirm render_js. Schema drift shows up as fields that are present but wrong or empty across many records; compare a few raw responses against your rules object before editing the cleaner.

Job board types and what they demand
Job Board TypeJob Posting CharacteristicsTechnical Challenges
Traditional Job Boards (Indeed, foundit, SimplyHired)• Search-based navigation • Paginated results • Structured job cards• Rate limiting on search pages • Dynamic loading of job postings • Session-based access controls
Company Career Pages (Workday, Greenhouse, Lever)• Custom implementations with ATS platforms • Inconsistent data structures • JavaScript-heavy frameworks• Framework-specific handling • Render JavaScript content • Handle inconsistent layouts
Niche Industry Boards (Wellfound, Stack Overflow, Dribbble)• Industry-specific fields • Structured data formats • Less aggressive anti-bot measures• Limited API documentation • Unique field mappings • Specialized authentication requirements

This classification is a planning aid. Individual sites within a category may behave differently; verify the target board before relying on any category assumption.

Frequently asked questions

Do I need CSS selectors to scrape job postings?

Not necessarily. With an AI extraction API you describe the fields and their types, and the service maps them to the page. Selectors are still useful when you need to target a specific element or debug a mismatch.

Why keep render_js on for a page that seems to load server-side?

Because many job boards load listings with JavaScript, and a page that works today may change. Keeping render_js on is a low-cost default that reduces surprises when you move to a new board.

How should I handle salary fields that come in different formats?

Extract the raw string first, then normalize it in a separate step. The cleaner example in this guide rewrites a matched range into a consistent format and leaves unmatched values untouched so you can review them.

Should I store job data in CSV or SQLite?

CSV is convenient for quick spreadsheet review. SQLite is better when you need to filter, join, or build dashboards. Choose based on the queries you expect to run, and keep the raw record alongside the normalized one.

What should I check first when a scrape returns nothing?

Check the response status and body before changing your rules. A challenge page or unexpected status points to a block; an empty listings container often means the page needed more time or JavaScript; fields that are present but wrong across many records point to schema drift.

{
  "job_title": "Senior Python Developer",
  "company": "TechCorp Inc.",
  "location": "San Francisco, CA (Hybrid)",
  "salary_range": "$150,000 - $200,000 per year",
  "required_skills": [
    "python",
    "django",
    "aws",
    "docker",
    "kubernetes"
  ],
  "posting_date": "2026-01-18"
}