How to Collect Data for Machine Learning: A Step-by-Step Guide
A model cannot learn what the data does not show. Before tuning an algorithm, decide what you are predicting, where the evidence lives, and how you will turn raw pages or records into a clean table. This guide walks through the four stages of that pipeline and gives you a checklist for each one.
Quick answer
Most machine learning datasets come from three places: internal company systems, public repositories, and the open web. Internal data is usually proprietary and gives the strongest competitive advantage. Public datasets are useful for benchmarking but are available to everyone. The open web is the largest source of unstructured text and images, and it is where web scraping becomes useful.
If you need high-volume or highly specific external data, a managed scraping API removes the proxy, CAPTCHA, and browser infrastructure that otherwise becomes a bottleneck. The request below shows the shape of a structured extraction call. The exact request code and environment setup are supplied in the accompanying protected modules; treat them as the contract, not as illustrative pseudocode.
Step 1: Define the objective and the data shape
Write down the prediction target before you write any collection code. The target determines whether you need rows and columns, raw text, or images with labels.
- Regression and tabular models: structured CSVs or SQL exports, such as house prices or stock movements.
- NLP and LLMs: large volumes of unstructured text, such as comments or news articles.
- Computer vision: image or video files, often with metadata or labels.
The structured versus unstructured distinction drives every later tool choice. A tabular model can be trained from a clean CSV; an LLM pipeline needs a text corpus and a tokenization plan.
Step 2: Choose the data sources
Use this decision checklist before collecting:
- Does the data already exist in a CRM, activity log, or transaction history? If so, start there; it is proprietary and usually the highest-value source.
- Is a public dataset sufficient for benchmarking or a baseline? Kaggle, Google Dataset Search, and AWS Public Datasets are common starting points.
- Does the task require niche or real-time text that no public repository contains? Then the open web is the source, and you need a scraping method.
A practical pipeline often combines all three: internal data for the target, public data for pretraining or comparison, and web data for domain-specific coverage.
Step 3: Pick a collection method
There are three practical methods, and they differ mainly in how much infrastructure you own.
Public APIs. Platforms such as Reddit or the New York Times return clean JSON, but they impose rate limits. The Reddit example in the supplied contract module fetches top posts from a subreddit and prints titles and scores. For production use, Reddit recommends authenticated OAuth access with a registered application and a unique User-Agent, which also gives more reliable rate limits and API stability. Reddit's API Terms restrict using user-generated content for machine learning or AI training, so review the platform terms before collecting at scale.
Custom scraping with Python. BeautifulSoup and Playwright can extract pages when no API exists. The cost is operational: you manage proxies, CAPTCHAs, and headless browsers for JavaScript-heavy sites, and that work grows quickly with dataset size.
Managed scraping API. A service handles proxy rotation, CAPTCHA solving, and JavaScript rendering, so your code focuses on extraction rules and the resulting JSON. The supplied request modules show both a quotes example and a product-listing example; the extraction rules turn page elements into a JSON object that can be loaded into a Pandas DataFrame.
Field and control map for a structured extraction request
The request contract uses a small set of inputs. Read this map alongside the supplied code modules rather than retyping field names from memory.
api_key: read from the environment; the module raises an error if it is missing.url: the target page to extract.render_js: set to"false"for static pages; enable only for dynamic sites because rendering is slower and more expensive.extract_rules: a JSON string describing selectors, list types, and output fields.timeout: a client-side guard against hanging requests.
The response is JSON. A top-level key such as quotes or products contains a list of extracted records. If the selector matches nothing, the list is empty rather than an error, so check record counts before treating a run as successful.
A useful failure distinction:
- Missing key: the script raises before any network call.
- HTTP error:
raise_for_status()reports a 4xx or 5xx response. - Empty extraction: the request succeeds but the selector returns no records; inspect the page markup and selector.
Step 4: Clean, normalize, and format
Extraction is only half the job. Raw output usually contains missing values, duplicates, inconsistent casing, and mixed types. A standard cleaning pass covers four operations:
- Missing values: drop empty rows or impute with a mean or median.
- Duplicates: remove repeated records so the model does not overfit on the same example.
- Normalization: lowercase text and scale numeric values, for example to the 0–1 range.
- Formatting: store the final dataset as CSV, JSONL, or Parquet.
The supplied Pandas module demonstrates the pattern on a small sentiment dataset: drop rows with missing text, strip and lowercase, remove empty strings, drop duplicate text, reset the index, and write a Parquet file with the pyarrow engine.
Legal and ethical checklist
Collection is a legal and ethical responsibility, and the requirements are tightening. The EU AI Act's provisions are being phased in through 2027, with data provenance and transparency obligations for teams operating in Europe. Use this checklist before scaling a pipeline:
- robots.txt and public access. Check the target site's
/robots.txtfirst. One often-cited precedent is the Ninth Circuit's hiQ v. LinkedIn decision, which held that harvesting publicly accessible pages did not amount to a Computer Fraud and Abuse Act (CFAA) violation. Read that narrowly: it is not a blanket permission for scraping. A site may still raise other legal claims, and reaching content through logins or paywalls stays legally risky. - Rate limiting. Cap concurrency and add sleep intervals so requests resemble human browsing. This protects the target server and reduces blocking.
- Data minimization and PII. Under GDPR and CCPA, collect only what the model needs. Strip names and IP addresses when they are not required, and anonymize or mask personally identifiable information immediately after collection.
- AI opt-outs. Respect machine-readable signals such as
noaithat site owners use to opt out of generative AI training.
These are engineering and compliance starting points, not legal advice. Confirm your obligations with counsel for your jurisdiction and use case.
Scaling the pipeline
Public datasets are good for practice, but production models usually depend on custom collection. The durable skill is the transition from raw pages to a structured, clean dataset: define the target, choose the source, extract into JSON, clean and normalize, then store in a format your training code can read. Keep the extraction rules and cleaning steps in version control so a dataset can be rebuilt when a site changes.
Frequently asked questions
Can I use web scraped data for machine learning?
Yes. Scraping is a common way to build custom datasets, especially for NLP and LLM training. Respect ethical guidelines, avoid PII, and review the target site's terms and robots.txt before collecting at scale.
How much data is needed for machine learning?
It depends on model complexity. A simple regression may need a few thousand rows, while deep learning and LLMs may need millions. Clean, accurately labeled data matters more than raw volume.
What is the best data format for machine learning?
Parquet is a strong choice for large tabular datasets because it is compressed and fast to read. JSONL is common for text because each record sits on its own line, which simplifies preprocessing. Structured JSON from an extraction API can be converted to either format.
Are public datasets enough for real-world ML projects?
They are useful for benchmarking, but they rarely provide a competitive edge because everyone can access them. Production teams usually add proprietary data collected through internal systems or web scraping.
How do I know whether an extraction request failed or simply found nothing?
Check three stages. A missing API key raises before the request is sent. An HTTP error is reported by raise_for_status(). If the request succeeds but the response list is empty, the selector did not match the page markup; inspect the page and adjust the extraction rules.