Scrape Tables with BeautifulSoup
A practical way to turn an HTML table into rows of plain text in Python: fetch the page, parse it, find the table, then walk its rows and cells in order.
The row-by-row approach
Start with the table, not the whole document. Once the correct table element is located, every row search stays inside that table, which prevents unrelated markup elsewhere on the page from leaking into the result.
Inside the table, collect all tr elements. For each row, collect its td and th children together. Including th matters when the first row is a header: treating header and data cells the same way gives you one uniform list of rows instead of two separate passes.
Then normalise each cell: strip surrounding whitespace from its text and discard empty strings. Append the cleaned list as one row in the output data. Printing that list is enough to inspect the shape of the extraction before you save it anywhere.
Worked example and pre-flight checklist
The complete worked example is provided as protected module code0001. Read it in this order:
- Fetch the demo table page and parse the response content with the html.parser parser.
- Initialise an empty data list, then find the table element and its tbody element.
- Find all tr elements in the table and loop over them one row at a time.
- Within each row, find all td and th elements, strip each cell's text, and filter out empty entries.
- Append each cleaned row to data and print the final list.
Before running it, confirm three things: the page actually contains a table element, the rows you need are tr elements rather than a different structure, and empty cells in your data are safe to drop rather than needing to be kept as blank placeholders.
Pitfalls to avoid
A common mistake is searching the entire parsed document for cells instead of scoping the search to one row. That mixes cells from different rows and tables. The fix is to nest the searches: table first, then rows within the table, then cells within each row.
A second mistake is forgetting th cells, which silently drops header rows. Searching for both td and th in the same call avoids that.
A third issue is untrimmed text. Cell text often carries newlines and spaces from the HTML formatting, so stripping each value keeps the output usable for comparison, sorting, or export.
Related BeautifulSoup questions
If you need to go further, these related BeautifulSoup questions cover the neighbouring tasks: choosing between Scrapy and BeautifulSoup, using XPath selectors, finding all links, finding elements without specific attributes, finding an element by class, finding elements by attribute, finding elements by multiple tags, finding sibling nodes, and selecting values between two nodes. A fuller BeautifulSoup web scraping tutorial is also available for a step-by-step walkthrough.
Frequently asked questions
Should header cells be included when extracting a table?
Use the table element as the scope, collect every tr inside it, then collect td and th children within each row. That keeps header cells and body cells together in row order and avoids picking up cells from a different table on the same page.
Why clean and filter cell text?
The workflow trims whitespace from each cell's text and drops empty strings before appending the row. That produces cleaner lists, but it can also shift columns if a genuinely empty cell should be preserved as a placeholder — check whether your downstream data needs fixed column positions.
What is the role of tbody in the example?
The example locates a tbody element but then searches the whole table for rows. If a page has multiple tables or a table without a tbody, decide explicitly whether to scope rows to that tbody or to the table, rather than assuming both searches return the same set.