Skip to content

Build a News Crawler with a Scraping API in Python and Flask

This guide builds a small news crawler in two stages: first a terminal program that collects headlines from three news sections, then a Flask page that refreshes those headlines on a schedule. The key idea is to separate fetching from presentation — one function retrieves headlines for any URL and rule set, and the rest of the app only has to store and display them.

Prerequisites and environment setup

Before writing code, gather four things: an account and API key for the scraping service, Python, Flask for the web page, and APScheduler for timed refreshes. The reference environment used Python 3.11.0 and APScheduler 3.10.1; other versions may differ in small ways. Create a project folder, isolate dependencies in a virtual environment, and install the client library plus Flask and APScheduler. The exact commands are provided in protected modules code0001 through code0006 and code0009, which accompany this page.

Choosing targets and per-site extract rules

The example targets three sections: a CNN business page, an NBC tech and media page, and Yahoo Sports. Because each site marks up headlines differently, one generic selector will not work everywhere. Instead, each target gets its own extract rules: a CSS selector that finds headline elements, with output requested as a list of text. That per-site mapping is the main thing to adapt when adding a new source — inspect the page, find the headline element, and write a matching rule. The three URLs and their rules are defined in protected modules code0007 and code0013.

Terminal version: one reusable fetch function

The terminal version keeps the design deliberately small. A single helper function accepts a URL, its extract rules and a display name, sends the request through the client, reads the headlines from the JSON response, and prints them under the source name. Calling that helper once per source produces a grouped terminal listing. This separation matters later: the same fetch logic can be reused unchanged when the output moves from terminal to web page. The full implementation is in protected module code0007, and running it is covered by code0008.

Flask version: scheduled refresh and display

The web version adds three pieces. First, headlines are stored in separate lists per source, with a timestamp recording the most recent crawl. Second, a startup hook runs the crawl once on the first request, so the page is never empty for the first visitor; after that, a background scheduler refreshes the lists on an interval — every minute in the example. Third, a single route renders a template that loops over each source list and shows the last crawl time. If no crawl has happened yet, the page says so explicitly rather than showing a blank timestamp. Modules code0010 through code0021 hold the exact implementation, including the template and the run command.

Adaptation checklist

Use this checklist when adapting the example. Confirm each selector still matches current page markup, since headline classes change frequently. Decide a refresh interval based on how often the sections update rather than copying the one-minute default blindly. Keep the fetch function independent of Flask so it can be tested from the terminal first. Store the API key outside version control. Finally, treat an empty headline list as a signal to re-check the selector before debugging the scheduler.

Example targets and extraction output
SourceTarget sectionWhat the rule captures
CNNBusiness sectionHeadline text as a list
NBCTech and media sectionHeadline text as a list
YahooSports sectionHeadline text as a list

Selectors are tied to captured page markup and may need updating as sites change.

Terminal versus Flask versions
StageOutputRefresh behaviour
Terminal versionHeadlines printed per sourceManual, on each run
Flask versionHeadlines rendered on a web page with last crawl timeOnce on first request, then on a schedule

Interval shown is the example value, not a recommendation for all sites.

Frequently asked questions

What do I need before starting?

You need an account and API key for the scraping service, plus Python, Flask and APScheduler. The tutorial environment used Python 3.11.0 and APScheduler 3.10.1, and different versions may behave differently.

Why does each news site need different extract rules?

Each site needs its own extract rules because headline markup differs. A rule pairs a CSS selector with list output as text, so adapting the pattern to another site means inspecting its markup and choosing the matching selector.

How does the Flask version differ from the terminal version?

The first version prints headlines in the terminal. The Flask version stores headlines per source, runs the crawl once on the first request so visitors see content immediately, then refreshes on a schedule and shows the last crawl time on the page.

How often should the crawler run?

The example schedules a crawl every minute. That interval is a starting point you can change to suit how often the target pages actually update and how much API usage you want to spend.