Skip to content

Build a Job Aggregator: Source, Dedupe, Refresh and Serve

A job aggregator is not a pile of scrapers. It is a pipeline: source postings from the right layer, normalize every source to one record, deduplicate the same role across boards, keep the feed fresh, then store and serve it. Scraping one job page is the easy part; the work is everything that turns a pile of pages into one clean, current feed. This guide walks the full architecture end to end with Python for the pipeline: sourcing from applicant tracking systems and job boards, collecting and extracting postings, normalizing fields, handling duplicates and expiry, storing the data, scheduling runs, serving results, scaling, and scraping responsibly.

What a job aggregator actually is

A job aggregator collects postings from many sources — job boards, company career pages, and applicant tracking systems — normalizes them to one shape, removes duplicates, keeps them fresh, and serves them in one place. The scraping is one step. The aggregator is the pipeline around it.

That distinction is where most builds go wrong. You can scrape a single job page in an afternoon, and then the hard parts arrive all at once: every source has a different layout, the same role shows up on five boards, listings go stale or turn into ghost jobs, and you still have to decide how job seekers and employers will actually consume the data. Those four problems — many layouts, duplicates, freshness, and serving — are the aggregator, and they are what this guide focuses on.

Extraction itself is a separate decision. If you have not yet chosen how to pull fields from a page, compare the available job scraping tools first; from here on the assumption is that extraction works and the question is how to wrap it in a pipeline.

Choose your sources: the ATS layer, not 2,000 sites

You do not need one scraper per company. Most companies post through a handful of applicant tracking systems (ATS) like Greenhouse, Lever, Workday, Ashby, and BambooHR. Target that layer and you can gather job listings from job boards and company career sites with a few shaped collectors instead of thousands.

Three source types each earn their place:

  • Job boards such as Indeed and Glassdoor give breadth: a single source covers many employers, but you pay for it with heavier anti-bot defenses. Two starting points are a walkthrough for scraping a large board like Indeed and a prebuilt Indeed API.
  • Company career pages give freshness and cut out the middleman. A company page is the original source of its openings, so a role usually appears there before it reaches the boards.
  • ATS endpoints or ATS-shaped scrapers give structure. Several systems — Greenhouse, Lever, and Ashby among them — publish a documented per-company jobs endpoint you can read directly with no HTML parsing. Greenhouse's job board API, for instance, returns a company's open roles as JSON.

The exceptions matter. Workday and BambooHR have no clean public feed, so in practice they need scraping, and Workday carries the heaviest anti-bot defenses of the group. Fully bespoke career sites also need per-site handling. This is where AI extraction pays off: it reads fields from plain English rather than from selectors that break on every redesign.

The real payoff of targeting the ATS layer is maintenance. A scraper's cost is not the first version; it is every repair after a site changes, and that cost tracks the number of distinct layouts you parse, not the number of postings you collect. A hundred companies on Greenhouse share one shape, so one collector serves all of them and survives any single company redesigning its careers page.

A workable order: begin with ATS endpoints and the largest boards, and add a bespoke source only when a role you care about cannot be reached any other way. Coverage can grow while the maintenance surface stays small.

Collect the postings

Collection works like this: hand the source URL to a web scraping API that renders the JavaScript and returns the page, then ask AI extraction for the fields you want in plain English. Because the request describes fields rather than DOM paths, a redesign does not automatically break it the way per-site selectors do.

The collect step below runs against a public GitLab posting hosted on Greenhouse, with the source hardcoded for this one example. The extraction request and its JSON output are supplied as protected modules; their exact contents accompany this page. The request sends the URL, enables JavaScript rendering, and passes an AI query asking for title, company, location, salary, remote, and posted_at. The response is a dict that also records the URL and source.

One caveat from the first run: posted_at came back null because that Greenhouse page does not show a date. AI extraction returns what the page actually contains, not what you wish it did, which is a good reason to fill date gaps later in the pipeline. This is the extraction step, and it is well-trodden ground, so it is not re-taught here. For the full field-by-field walkthrough, see how to scrape job postings with AI extraction.

Normalize every source to one schema

Every source hands you a slightly different shape. Salaries are written a dozen ways, “remote” might be a boolean, the string "Yes", or baked into the location, and dates arrive as "3 days ago", an ISO date, or nothing at all. Normalization maps any raw posting into one common job record so the later steps can treat every job the same way:

id, title, company, location, remote, salary_min, salary_max, currency, url, source, posted_at, fetched_at

The messy parts are salary and the remote flag, so they get their own helpers. parse_salary() is intentionally naive: it grabs every number it finds, so an hourly or monthly rate ends up looking annual, and a stray year in the text can land in salary_min. Once your sources mix pay periods, capture the period as its own field. normalize() ties the helpers together and produces the common record.

When normalize() ran over the raw postings, the inconsistent inputs came out uniform. The script prints one JSON object per line; here are the two records being watched, and both held up: "$108k–$130k a year" became 108000/130000, and "Remote (US)" still read as remote even though the flag lived in the location string. That single shape is what lets you compare and dedupe across sources. If you need tighter control over the raw fields before they reach normalize(), data extraction rules let you map selectors to named JSON.

Deduplicate the same job across sources

This is the step that turns a scraper into an aggregator. The same role appears on a job board, an ATS, and the company’s own careers page, so naive title-plus-company matching is not enough: locations read "Remote, US" on one and "United States" on another, and titles pick up qualifiers like "(Remote)". Do it in layers instead. Canonicalize the URL, then build a structured match key from a normalized employer, title, and location, and fall back to text similarity for near-duplicates worded differently.

canonicalize_url() strips tracking noise, yet it never follows redirects. Resolve redirects while fetching, then canonicalize the URL you land on. The dedupe() function walks the records, decides which existing job each one belongs to, and merges duplicates. When records collapse, it keeps the earliest posted_at, records every source URL on the survivor, and backfills a salary if the first copy lacked one.

This is the step that took the most time, and the first attempt got it wrong in an instructive way. The initial match key collapsed the Greenhouse and Indeed copies of the GitLab role but left the third one, from the company careers page, sitting on its own, because its title read "AI Engineer (Remote)" and its location said "United States" instead of "Remote, US". That is exactly the near-duplicate an aggregator has to catch, so parenthetical stripping and US-location folding were added; only then did all three collapse into one.

For roles at the same employer, the fuzzy fallback compares normalized titles with difflib.SequenceMatcher, which catches wording differences the structured key still misses. If you need to match semantically different phrasings, embeddings are the heavier upgrade. Running the deduper over six raw records, with the GitLab role appearing three times, yielded four unique jobs. All three GitLab copies merged into a single job that records all three sources and retains the earliest posting date.

Handle freshness and expired jobs

Freshness is the product. A job board that shows roles filled two months ago is worse than useless, so on every run you do two things: stamp last_seen on any job you saw again, and expire the ones you did not. Companies pull listings, and they also post ghost roles they never intend to fill, so a job that lingers unseen past a threshold should drop off.

With the store in place, both operations shift into the database layer as upsert_jobs() and expire_in_db(), performing the same work in SQL; the in-memory versions show the logic most plainly.

A stronger signal is a death-check that treats a 404 or 410 as gone. Keeping fetched_at and last_seen on every record is what enables automatic expiry of stale postings; expire_stale() assumes last_seen is always present, so set it as soon as a job is first inserted. To confirm expiry fires, two passes ran ten days apart while Nimbus Cloud removed its listing in between, and the stale job dropped off by itself.

Store it

A simple SQLite layer is enough to start, and it scales further than most people expect. The schema is the common record, with a UNIQUE constraint on the dedup key so re-running never inserts the same job twice, and indexes on the fields you actually query: location, posted_at, and primary_source. A deduped job can carry several sources, so sources keeps the full list as JSON while primary_source records the one the job was first seen on; that single value is the one you filter and index on, and it does not change on later runs.

connect() applies the schema and assigns sqlite3.Row as the connection's row factory. That single line lets every query here pass its rows directly to dict(); without it, SQLite hands back plain tuples and dict(row) fails.

Writes use an idempotent upsert keyed on dedup_key, so a matching job is updated in place (its last_seen, source list, and status) rather than duplicated. Running the same batch twice on purpose confirmed a second run would not silently double the rows; the count remained four.

At volume, a few thousand new postings a day is normal, so you index for serving, not just for storage. When SQLite stops keeping up, the same schema moves to Postgres unchanged.

Schedule and orchestrate

This final step turns the one-off script into a running system. Every source is fetched through a retry wrapper so a single flaky board cannot kill the run, sources are staggered so they are not all hit at once, and each run appends a line to a run log. run_once() is the job you schedule. Two runs, ten days apart, show the store maintaining itself as Nimbus Cloud’s listing disappears. Point cron at it every few hours. If you would rather not maintain cron yourself, the CLI can run the same job on a schedule; the guide to scheduling recurring scrapes from the command line covers the setup.

Serve the data

Serving is often the last thing a builder designs, yet it is the least demanding stage once the data is clean. Normalization and deduplication have already resolved the hard inconsistencies, so a read path only needs to restrict results to the indexed columns, sort by posted_at, and hand back the matching rows. The query shape below is the one used for the worked example: it returns active jobs, applies optional filters for location, remote status, a posted-since date, and primary source, and orders the newest postings first. The exact search_jobs() implementation is supplied as a protected module and accompanies this page unchanged. When that query ran for remote roles posted since the start of the month, it returned exactly the two expected records, newest first. Search quality is a competitive advantage here, and AI can later support job recommendations or candidate matching. What makes an aggregator feel better than the source boards is advanced search and strong search results quality, not just more listings. The common shapes are a read API or JSON feed over your store, with filters for job title, job category, job type, or salary range; an RSS or email digest keyed to a user’s filters (freshness as a notification); and a simple front end on top. The same query powers a digest. Instead of making users pull, you push: run a saved search per subscriber, diff it against what they saw last time, and email or RSS the new matches. That is a convenient way for job seekers to keep up without checking back, and since every job already carries a posted_at and a last_seen, you get it almost for free. A front end is then just a thin layer over the same search_jobs() call, with filters mapped to query parameters. Two properties separate a good aggregator from the boards it draws on: repeated listings are collapsed, and roles that have gone quiet are retired rather than left to linger. Readers notice both, and neither is easy to imitate at scale. An aggregator that gets them right is more useful than one that simply covers more sources with older data.

Keep it running at scale

As you add sources and refresh often, collection is where cost and blocks live. Match the proxy tier to each source instead of paying for the heaviest option everywhere. Turn on render_js for JavaScript-heavy boards, add premium_proxy (residential IPs) for datacenter-blocked boards, and reserve Stealth for the hardest anti-bot sites, with polite per-source rate limits. These choices drive both reliability and cost as you add sources. The credit cost scales with the tier, so this is also your cost control. As of August 2026, a Classic request costs 1 credit without JavaScript rendering and 5 with it, Premium is 10 to 25, and Stealth is 75, with AI extraction adding 5 on top. Only successful requests are billed, so an HTTP 500 costs nothing and retries are safe. Prefer an ATS endpoint or a career page over a heavily defended board when you can, and the cost of a scheduled multi-source run stays predictable. For the full playbook, see web scraping without getting blocked, and confirm current rates on the pricing page.

Do it responsibly

Aggregating public job postings is a common and generally accepted use, but do it well. Scrape only public postings, never login-gated content, which puts sites like LinkedIn off limits. Respect each site’s robots.txt and Terms, keep request rates polite, and link back to the original posting so you send applicants and credit to the source rather than replacing it. This is not legal advice; rules vary by site and country, so check the ones that apply to you.

A decision checklist before you write a scraper

Use this checklist to decide where each source belongs before you write code.

  • Is the employer on a documented ATS feed (Greenhouse, Lever, Ashby)? If yes, read the JSON endpoint first and skip HTML parsing.
  • Is the role on a large board with heavy anti-bot defenses? Treat it as a breadth source and budget for rendering and proxies.
  • Is the role only on a bespoke career site or Workday/BambooHR? Plan per-site handling and AI extraction.
  • Does the source show a posted date? If not, expect posted_at to be null and fill the gap from fetched_at or last_seen.
  • Does the source mix pay periods? Capture the pay period alongside salary_min and salary_max.
  • Is the content login-gated? If yes, leave it out.

A useful failure distinction: a duplicate is the same role seen on more than one source, while a stale listing is a role that no longer appears at its source. Deduplication fixes the first; last_seen and expiry fix the second. Confusing the two is why some aggregators show one job five times and another job that was filled last month.

Field map for the common job record

The normalized record is the contract every later step depends on. Here is what each field is for and where it comes from.

  • id: built from source and URL; identifies the raw posting before dedupe.
  • title, company, location: cleaned strings from AI extraction.
  • remote: boolean derived from the source flag or from the location string.
  • salary_min, salary_max, currency: parsed from the salary text; pay period is not captured by the simple parser.
  • url: the original posting URL.
  • source: the source name, such as greenhouse or indeed.
  • posted_at: the date shown on the page, or null when absent.
  • fetched_at: the time the record was collected.

After dedupe, the survivor also carries source_urls, sources, last_seen, and active. primary_source is the source where the job was first seen and is the value to filter and index on.

Worked example: one role, three sources

The GitLab AI Engineer role appeared on Greenhouse, Indeed, and the company careers page. The Greenhouse copy had salary "$108,400 - $129,600 USD" and location "Remote, US". The Indeed copy had company "GitLab Inc.", location "Remote (US)", and salary "$108k–$130k a year". The careers copy had title "AI Engineer (Remote)" and location "United States".

A naive title-plus-company match would keep the first two together and leave the third separate. Canonicalizing the URL removes tracking parameters. Normalizing the title strips the parenthetical qualifier and the word "remote". Normalizing the location folds US variants to "us" and drops the word "remote". Only after those steps did all three collapse into one record with the earliest posted_at and all three source URLs attached.

Frequently asked questions

How many scrapers do I need to build a job aggregator?

Fewer than you think. The number is bounded by the applicant tracking systems your targets use, not the number of companies: if most sit on Greenhouse, Lever, or Workday, you write a handful of ATS-shaped scrapers instead of one per company. Bespoke career sites still need per-site handling, which is where AI extraction helps.

How do I deduplicate the same job across different sites?

Work in layers. Start by canonicalizing the URL, then match on a normalized employer name plus title plus location, and finally fall back to text similarity or embeddings for near-duplicates that are worded differently. Retain the earliest posting date and keep every source URL on the surviving record.

How often should a job aggregator refresh its data?

Often enough to feel fresh without hammering sources: every few hours per source, staggered, is a common choice. On each run, remove postings that no longer appear at the source so expired and ghost jobs do not pile up. Freshness, not raw coverage, is what makes an aggregator useful.

Can I get job data without scraping, through an API?

Sometimes. Several applicant tracking systems, like Greenhouse, Lever, and Ashby, publish a structured jobs endpoint per company, and some job boards have official or third-party APIs. Use those where they exist, and scrape public pages for the sources that do not (Workday, for example, has no clean public feed). A good aggregator mixes both.

Can I build a job aggregator without coding?

Partly. No-code tools and RSS glue can collect from a small, fixed set of sources, which is enough for a narrow niche board. But cross-source deduplication, freshness logic, and serving usually need some code or a developer once you grow past a handful of sources. See no-code web scraping for how far the no-code route goes.

Is scraping job postings legal?

Scraping public job postings is a common and generally accepted use, but this is not legal advice. Respect each site’s robots.txt and Terms, never scrape login-gated content, keep request rates polite, and link back to the original posting. Rules vary by site and country.

{
  "title": "AI Engineer",
  "company": "GitLab",
  "location": "Remote, US",
  "salary": "$108,400 - $129,600 USD",
  "remote": true,
  "posted_at": null,
  "url": "https://job-boards.greenhouse.io/gitlab/jobs/8565469002",
  "source": "greenhouse"
}
{"id": "greenhouse:e045ac763a", "title": "AI Engineer", "company": "GitLab", "location": "Remote, US", "remote": true, "salary_min": 108400, "salary_max": 129600, "currency": "USD", "url": "https://job-boards.greenhouse.io/gitlab/jobs/8565469002", "source": "greenhouse", "posted_at": null, "fetched_at": "2026-07-24T09:00:00+00:00"}
{"id": "indeed:1af2e37380", "title": "AI Engineer", "company": "GitLab Inc.", "location": "Remote (US)", "remote": true, "salary_min": 108000, "salary_max": 130000, "currency": "USD", "url": "https://www.indeed.com/viewjob?jk=abc123&from=serp&vjs=3", "source": "indeed", "posted_at": "2026-07-21", "fetched_at": "2026-07-24T09:00:00+00:00"}