Skip to content

Scraping Financial Statements with Python: EDGAR Tables to pandas

Public companies file income statements, balance sheets, and cash flow statements with the U.S. Securities and Exchange Commission, and EDGAR holds decades of those documents. Pulling them by hand does not scale past a handful of companies. This guide shows a Python workflow that fetches a filing, extracts a specific table, converts it into a pandas DataFrame, repeats across a ticker list, and saves the result for analysis. The examples use a scraping API's extract_rules to return table data as a two-dimensional array, which removes most manual HTML parsing.

Quick start: one filing, one table, one DataFrame

The shortest path is: choose a filing URL, describe the table you want with a selector, request the page, and load the returned rows into pandas. The supplied quick-start code (code0001) does exactly that against Apple's 10-K filing, selecting any table that contains the text "Net sales" and returning it as table_array. The first row becomes the column headers and the remaining rows become data. If you see a DataFrame, the request and selector both worked. If not, check the API key first, then the selector. SEC HTML pages are static, so JavaScript rendering is usually unnecessary for EDGAR filings.

Why automate financial statement collection

Manual downloads introduce inconsistent formatting, missed updates, and human error, and they become impractical once you track more than a few companies. An automated script can run on a schedule, apply the same extraction rules every time, and keep a dataset current without repeated copying. The trade-off is that you must design selectors that survive layout changes. EDGAR filings appear as both HTML and XML depending on year and form type, and table structures shift over time, so selectors should target stable text or structure rather than brittle positional paths.

What you can extract

The core targets are the income statement (revenue, expenses, profit over a period), the balance sheet (assets, liabilities, equity at a point in time), and the cash flow statement (money moving in and out). Notes to the financial statements are also worth extracting because they carry accounting methods, debt obligations, and future commitments, often in structured HTML tables. Most filings follow consistent row labels such as "Total Revenue" or "Net Income" and column headers for periods, which makes reusable rules possible, though occasional variations require flexible parsing.

Set up the Python environment

Use Python 3.8 or newer and a Jupyter Notebook for interactive development. Install pandas for data manipulation and CSV/Excel export, matplotlib for charts, python-dateutil for reporting dates, and the scraping API client for requests. The supplied install commands (code0002) cover Windows, macOS, and Linux. Create an isolated virtual environment with the supplied activation commands (code0004) so project dependencies do not collide.

Extract one company's income statement

Identify the filing URL, usually a 10-K annual report or 10-Q quarterly filing. The request sends extract_rules describing the target table, and the API returns structured data. The supplied single-company example (code0005) targets Apple's 10-K and pulls the table containing "Net sales". The extract_rules object has four useful keys: selector_type chooses CSS or XPath, selector locates the table, type: 'table' tells the API to parse the HTML table, and output: 'table_array' returns rows as arrays. CSS selectors are usually sufficient and more readable; XPath adds precision for complex documents. The first returned row holds column headers, so pandas construction uses table[1:] as data and table[0] as columns.

Control the period and decide on JavaScript

Date range control comes from choosing the filing URL, not from query parameters. Each SEC filing URL embeds the period, so selecting the right 10-K or 10-Q is the date filter. For pages whose content loads after the initial render, set render_js to True so extraction runs after the page is fully loaded. For static EDGAR HTML, leaving it off is usually faster and sufficient.

Inspect and clean the DataFrame

Print the first rows with df.head() to confirm the extraction. Financial tables often carry stray whitespace and dash placeholders. Two common cleanups are stripping column names with df.columns = df.columns.str.strip() and replacing em dashes with zeros using df.replace('—', 0) before numeric calculations. Do this before computing ratios so string values do not break arithmetic.

Scale to multiple tickers

Batch scraping needs a ticker list mapped to filing URLs, a loop, and error handling. The supplied multi-company example (code0006) iterates a list of companies, applies one XPath rule that matches either "Net sales" or "Total net sales", skips tables with fewer than two rows, writes one CSV per ticker, and sleeps one second between requests. That pattern keeps a single failure from stopping the batch. Add progress prints such as the ticker being processed, and wrap each request in try/except so timeouts or missing filings are logged and the loop continues. For empty or missing tables, skip the company or record a placeholder rather than terminating the run.

Analyze and export the data

After extraction, convert numeric columns, compute ratios, and save results. The supplied analysis function (code0007) coerces Revenue, Net Income, and Total Assets to numeric, calculates Profit Margin and Revenue Growth, and returns the DataFrame. Export with df.to_csv('financial_analysis.csv', index=False) or df.to_excel('financial_analysis.xlsx', index=False); Excel preserves formatting and is convenient for stakeholders who work in spreadsheets. For trend work, a four-quarter moving average such as df['Revenue_MA'] = df['Revenue'].rolling(window=4).mean() smooths seasonal swings. Matplotlib line charts can show revenue over time and compare metrics on separate axes. Financial statement data can also feed fundamental inputs for long-term strategies, but keep early models simple and focused on clear relationships rather than complex prediction.

Reliability at scale

Large projects need proxy rotation, browser automation for dynamic pages, and retry logic when individual requests fail. Those are the areas where a managed scraping API reduces maintenance, letting the Python code focus on parsing and analysis. Before committing to a large run, test the workflow on a small ticker list and confirm that selectors still match current filings.

extract_rules field map
FieldPurposeTypical value
selector_typeChoose selector languagecss or xpath
selectorLocate the target tabletable:contains("Net sales") or an XPath expression
typeTell the API to parse a tabletable
outputReturn formattable_array

Field names and values reflect the supplied examples; confirm current API behavior in the official documentation.

Failure checklist
SymptomLikely causeFirst check
No DataFrame returnedBad API key or selectorAPI key, then selector match
Empty table or fewer than two rowsFiling lacks the target tableSkip company or record placeholder
Request error or timeoutNetwork or rate limitRetry logic and delay between requests
Numeric calculations failDash placeholders or whitespaceStrip columns and replace dashes with zeros

Checklist derived from the supplied workflow; not a guarantee of every failure mode.

Frequently asked questions

What are the main benefits of using Python to scrape financial statements?

Python offers pandas for data manipulation, broad community support for financial analysis, and integration with scraping APIs. Its readability makes it approachable for beginners while still supporting advanced modeling and analysis.

Which Python libraries are essential for scraping financial data?

Core choices include pandas for data manipulation, requests or a scraping client for fetching pages, matplotlib for visualization, and python-dateutil for reporting dates. BeautifulSoup helps with HTML parsing when needed, and numpy supports numerical calculations.

How can I handle errors when scraping data for multiple companies?

Wrap each company's request in try/except, log enough detail to debug, and use continue so the remaining companies still run. Add retry logic for temporary network issues and validate that the table has data before saving.

What types of financial statements can I scrape with Python and a scraping API?

You can target income statements, balance sheets, cash flow statements, and notes to financial statements from SEC filings. The workflow covers 10-K annual reports, 10-Q quarterly filings, and 8-K current reports, as well as specialized forms such as proxy statements and insider trading reports.