MechanicalSoup Tutorial: Stateful Python Scraping, Forms and Pagination
MechanicalSoup is a Python library for driving a lightweight, stateful browser over HTTP. It keeps cookies and page state between requests, exposes the current document as a Beautiful Soup object, and provides helpers for selecting forms and following links. This guide walks through a minimal static scrape, then a multi-step sandbox workflow that navigates, searches, extracts a table to CSV and moves to a second results page.
What MechanicalSoup handles for you
MechanicalSoup is not a complete scraper on its own. It is a browser-automation layer that pairs with Beautiful Soup, which is bundled, for parsing and querying HTML. The practical value is the stateful browser: it retains state and tracks changes between requests, which simplifies scripts that fill forms, submit data, click through links and move between pages. Typical actions include filling out forms, submitting data, clicking buttons and navigating through pages. Because the current page is available as a Beautiful Soup object, the same selector methods used in a plain Beautiful Soup script apply here.
Setup: Python, a project directory and a virtual environment
You need Python installed locally and any IDE you prefer. Create a project directory, move into it, then create and activate a virtual environment so the install does not touch global packages. The exact commands are preserved in the setup code modules below. After activation, install MechanicalSoup with pip3. Once that finishes, you can write scripts against the library.
First scrape: title and reference links from a Wikipedia page
The first exercise targets a static Wikipedia article. Create wikipedia_scraper.py, import mechanicalsoup, define a main() function and call it at the bottom. Inside main(), create a StatefulBrowser and call open() with the article URL. The browser.page attribute then gives you the parsed document.
To find the title, inspect the page in the browser's developer tools and locate the span with class mw-page-title-main. Use select_one() with the CSS selector 'span.mw-page-title-main'. select_one() returns the first matching element; the string attribute extracts its text. Running the script should print the page title.
For references, select the ordered list with class references, then select every span with class reference-text. Working at the span level matters because the references list also contains same-page backlinks that you do not want. For each reference span, find all anchor elements and print each href value that exists. The captured output shows DOI and PDF links, with the middle of the list truncated.
Multi-step workflow on a sandbox site
The advanced exercise uses a public scraping sandbox with a hockey-teams page that combines a search form, a results table and pagination. Create dynamic_scraper.py with placeholder functions for extract_table_data, navigate, handle_form and handle_pagination, and call them from main(). The goal: search for teams whose names contain "new", save the first page of results to CSV, follow the pagination link to page two, and save that page too.
Submitting a search form
The search form has classes form and form-inline, and its input box is named q. Select the form with select_form('form.form-inline'), pass the value with browser.form.input({"q": "new"}), then call submit_selected() to submit without locating the submit button manually. The last line of handle_form() calls extract_table_data with the filename first-page-results.csv so the loaded results are captured immediately.
Extracting table rows to CSV
extract_table_data(file_name) selects the table element, opens a CSV writer, selects the th header cells, strips whitespace and removes commas from each header string, and writes the header row. It then selects rows with class team, collects each row's td cells, applies the same cleaning, and writes each row. The file is closed at the end. The comma removal is a deliberate safeguard for CSV output, not a change to the page.
Following a pagination link
MechanicalSoup has no dedicated pagination method. Select the ul with class pagination, select its li elements, choose the entry at index 1 for page two, select the anchor inside it and call follow_link(). Then call extract_table_data('second-page-results.csv'). The captured terminal output shows the browser arriving at a URL with page_num=2&q=new, and two CSV files are created in the project directory.
Decision checklist before you run a MechanicalSoup script
- Confirm the target page returns usable HTML without JavaScript execution; MechanicalSoup drives HTTP requests, not a rendering engine.
- Inspect the live DOM in developer tools and record the exact selectors you will use: title span, references list, form class, input name, table row class and pagination list class.
- Decide which links are same-page backlinks and exclude them at the span or container level rather than filtering after extraction.
- Check whether any table cell can contain a comma, and decide whether stripping commas is acceptable for your downstream use.
- Verify the pagination index against the actual list order before relying on index 1 for page two.
- Print browser.url at each navigation step while developing so a wrong selector is visible immediately.
Field and control map for the sandbox exercise
The following map records the selectors and values used in the captured workflow so you can adapt them deliberately rather than guessing.
- Navigation item: li with id nav-sandbox; anchor inside it.
- Listing container: div with class page; second item leads to the hockey-teams page.
- Search form: form with class form-inline (also class form).
- Search input name: q; value used in the example: "new".
- Table header cells: th.
- Table data rows: tr with class team; cells are td.
- Pagination container: ul with class pagination; page links are li elements; the anchor inside the chosen li is followed.
- Output files: first-page-results.csv and second-page-results.csv.
Common failure distinctions
A selector that returns None usually means the page structure changed or the selector targets content that only appears after JavaScript runs. A form submission that appears to do nothing often means the input name does not match the field in the live form. A pagination step that lands on the wrong page usually means the list order differs from the assumed index. A CSV with shifted columns usually means a cell contained a comma that was not removed. These are different problems and should be diagnosed separately: inspect the current browser.page before changing the script logic.
What the captured CSV output looks like
The captured second-page CSV begins with a header row: Team Name, Year, Wins, Losses, OT Losses, Win %, Goals For (GF), Goals Against (GA), + / -. The rows include New York Islanders, New York Rangers and New Jersey Devils across the 1998 to 2000 seasons. Some OT Losses cells are empty in the captured sample. Treat this as a reference for the expected column shape, not as a live data feed; the sandbox content and its values can change.
Where to take this next
You have now used MechanicalSoup for a static extraction and for a stateful workflow that navigates, submits a form, writes table rows to CSV and follows a pagination link. The same building blocks apply to other HTML-driven tasks: select the element, act on it through the browser object, and inspect browser.page when something does not match. If a target depends heavily on JavaScript rendering, rate limits, proxies or CAPTCHA handling, a plain MechanicalSoup script will not cover that layer on its own; plan for those constraints separately.
Frequently asked questions
Is MechanicalSoup a full web scraper?
No. It is a browser-automation library that pairs with Beautiful Soup for parsing. You use its stateful browser to open pages, follow links and submit forms, then query the current page with Beautiful Soup methods.
Can MechanicalSoup scrape pages that rely on JavaScript?
MechanicalSoup works over HTTP requests and does not render JavaScript. If the content you need only appears after client-side execution, you need a different approach for that layer.
How do I fill and submit a form without clicking a button?
Select the form with select_form(), set field values through browser.form.input(), and call submit_selected(). The input keys must match the field names in the live form.
Does MechanicalSoup have a built-in pagination method?
No. You select the pagination container, choose the relevant list item, select its anchor and call follow_link(). Verify the list order before relying on a fixed index.
Why does the example remove commas from table cells?
The example strips commas from header and cell strings before writing CSV rows. That is a simple safeguard against column shifts when a value contains a comma; it is not a general CSV escaping strategy.