Build a News Crawler with a Scraping API in Python and Flask
This guide builds a small news crawler in two stages: first a terminal program that collects headlines from three news sections, then a Flask page that refreshes those headlines on a schedule. The key idea is to separate fetching from presentation — one function retrieves headlines for any URL and rule set, and the rest of the app only has to store and display them.
Prerequisites and environment setup
Before writing code, gather four things: an account and API key for the scraping service, Python, Flask for the web page, and APScheduler for timed refreshes. The reference environment used Python 3.11.0 and APScheduler 3.10.1; other versions may differ in small ways. Create a project folder, isolate dependencies in a virtual environment, and install the client library plus Flask and APScheduler. The exact commands are provided in protected modules code0001 through code0006 and code0009, which accompany this page.
Choosing targets and per-site extract rules
The example targets three sections: a CNN business page, an NBC tech and media page, and Yahoo Sports. Because each site marks up headlines differently, one generic selector will not work everywhere. Instead, each target gets its own extract rules: a CSS selector that finds headline elements, with output requested as a list of text. That per-site mapping is the main thing to adapt when adding a new source — inspect the page, find the headline element, and write a matching rule. The three URLs and their rules are defined in protected modules code0007 and code0013.
Terminal version: one reusable fetch function
The terminal version keeps the design deliberately small. A single helper function accepts a URL, its extract rules and a display name, sends the request through the client, reads the headlines from the JSON response, and prints them under the source name. Calling that helper once per source produces a grouped terminal listing. This separation matters later: the same fetch logic can be reused unchanged when the output moves from terminal to web page. The full implementation is in protected module code0007, and running it is covered by code0008.
Flask version: scheduled refresh and display
The web version adds three pieces. First, headlines are stored in separate lists per source, with a timestamp recording the most recent crawl. Second, a startup hook runs the crawl once on the first request, so the page is never empty for the first visitor; after that, a background scheduler refreshes the lists on an interval — every minute in the example. Third, a single route renders a template that loops over each source list and shows the last crawl time. If no crawl has happened yet, the page says so explicitly rather than showing a blank timestamp. Modules code0010 through code0021 hold the exact implementation, including the template and the run command.
Adaptation checklist
Use this checklist when adapting the example. Confirm each selector still matches current page markup, since headline classes change frequently. Decide a refresh interval based on how often the sections update rather than copying the one-minute default blindly. Keep the fetch function independent of Flask so it can be tested from the terminal first. Store the API key outside version control. Finally, treat an empty headline list as a signal to re-check the selector before debugging the scheduler.
| Source | Target section | What the rule captures |
|---|---|---|
| CNN | Business section | Headline text as a list |
| NBC | Tech and media section | Headline text as a list |
| Yahoo | Sports section | Headline text as a list |
Selectors are tied to captured page markup and may need updating as sites change.
| Stage | Output | Refresh behaviour |
|---|---|---|
| Terminal version | Headlines printed per source | Manual, on each run |
| Flask version | Headlines rendered on a web page with last crawl time | Once on first request, then on a schedule |
Interval shown is the example value, not a recommendation for all sites.
Frequently asked questions
What do I need before starting?
You need an account and API key for the scraping service, plus Python, Flask and APScheduler. The tutorial environment used Python 3.11.0 and APScheduler 3.10.1, and different versions may behave differently.
Why does each news site need different extract rules?
Each site needs its own extract rules because headline markup differs. A rule pairs a CSS selector with list output as text, so adapting the pattern to another site means inspecting its markup and choosing the matching selector.
How does the Flask version differ from the terminal version?
The first version prints headlines in the terminal. The Flask version stores headlines per source, runs the crawl once on the first request so visitors see content immediately, then refreshes on a schedule and shows the last crawl time on the page.
How often should the crawler run?
The example schedules a crawl every minute. That interval is a starting point you can change to suit how often the target pages actually update and how much API usage you want to spend.