LangChain Web Scraper: Fetch, Parse and Chunk HTML for RAG
LangChain is strong at organising text for retrieval and generation, but it does not fetch or render web pages on its own. A practical pipeline therefore pairs it with a scraping layer that returns usable HTML, then converts that HTML into clean text and manageable chunks. This guide walks through that sequence in Python, using a scraping API client for the network side and LangChain's text splitter for the processing side.
What the pipeline actually does
The workflow has four distinct stages, and each one can fail independently. First, a request fetches the target page. Second, the response body is parsed so that markup is removed and visible text remains. Third, that text is divided into overlapping chunks. Fourth, the chunks are printed or passed to a downstream store such as a vector database.
Keeping the stages separate matters because a blank or blocked result is a fetch problem, while garbled text is usually a parsing problem, and oversized or fragmented chunks are a splitting problem. Diagnosing the stage first saves a lot of guesswork.
The scraping layer supplies what LangChain does not: JavaScript rendering, proxy rotation, geo-targeted routing and resource blocking. LangChain then handles the text-processing side that feeds embeddings and retrieval-augmented generation.
Prerequisites and setup
You need Python 3.6 or newer, an API key for the scraping service, and three Python libraries: the scraping client, BeautifulSoup and LangChain. Install them from a terminal or command prompt with pip.
The install command is supplied as an exact module and should be run as written. After installation, sign in to the scraping service, copy the API key from the dashboard, create a project directory and add an empty .py file for the script.
Account and trial terms: {{OWNER_TRIAL_TERMS}}
Step 1: initialise the client and build the search URL
Start by importing the scraping client and creating an instance with your API key. The key is a credential, so keep it out of version control and load it from an environment variable in real projects rather than pasting it into the file.
Next, accept two inputs — a business type and a location — and encode them into a YellowPages search URL. The base path and the two parameter names are fixed by the target site, so they are shown exactly in the supplied code module. The urlencode helper turns the input values into a valid query string.
Step 2: fetch rendered HTML
The fetch function defines a JavaScript scenario that waits 3,000 milliseconds before extraction, giving dynamic content time to appear. JavaScript rendering is on by default, so no extra flag is needed for that part.
The GET call passes three parameters that are worth understanding:
- js_scenario carries the pre-extraction instructions, such as the wait.
- block_resources skips images, CSS and fonts, which reduces bandwidth and speeds up the response.
- country_code routes the request through a proxy in the chosen country, which helps with region-specific content.
When the call succeeds, the response text is raw HTML. Printing it confirms the fetch worked before any parsing is added. In the original walkthrough the example searched for dentists in Orlando.
Step 3: strip markup and keep metadata
A parsing function takes the raw HTML and the source URL. BeautifulSoup reads the markup, get_text with a space separator and strip enabled produces visible text, and the page title is captured when present, falling back to a placeholder otherwise. The function returns the text, the title and a small metadata dictionary containing the source URL and title.
Call this function immediately after the response is received and assign its three return values. The printed output should now read as prose rather than tags and attributes. If it does not, check that the response actually contained the page you expected before adjusting the parser.
Step 4: split the text into chunks
Import RecursiveCharacterTextSplitter at the top of the script, then create a splitter with a chunk size of 1,000 characters and an overlap of 200. The overlap preserves context that would otherwise be cut at a boundary, which matters when neighbouring chunks are retrieved independently.
Run the splitter on the parsed text and store the resulting list. A loop then prints the title, the metadata, the number of chunks and each chunk with an index. That output is the hand-off point: the same list can be written to a vector store or passed to a retrieval chain.
Decision checklist before you scale up
Use this checklist to separate the common failure points:
- Empty or blocked response — revisit the fetch stage: confirm the API key, the target URL and any rendering wait.
- HTML returned but text looks wrong — revisit parsing: check that the selector-free text extraction matches the page structure.
- Text is clean but chunks are poor — revisit splitting: tune chunk size and overlap against your retrieval tests.
- Everything works but costs rise — review which request parameters are enabled, since credit charges depend on the configuration of the GET call.
- Volume increases — consider concurrency and retry handling rather than sending requests one at a time.
Operational options worth knowing
Concurrency is possible because the client uses the blocking requests library; the concurrent.futures approach is a common fit for HTTP work. Automatic retries exist for both the GET call and the JavaScript scenario, although behaviour depends on the error type and request configuration.
Billing on failed requests: {{OWNER_500_STATUS_NO_CHARGE_POLICY}}
Resource blocking can also be used to skip tracking scripts that slow responses or trigger bot protection.
Compliance and proxy handling: {{OWNER_GDPR_AND_PROXY_CLAIMS}}
Field and control map
The table below summarises the inputs and settings used in the walkthrough so you can adapt them deliberately.
| Control | Where it appears | What it changes |
|---|---|---|
| search_terms | URL query | Business type searched |
| geo_location_terms | URL query | Location searched |
| js_scenario | GET parameters | Pre-extraction browser instructions |
| block_resources | GET parameters | Skips images, CSS and fonts |
| country_code | GET parameters | Proxy country for the request |
| chunk_size | Splitter | Maximum characters per chunk |
| chunk_overlap | Splitter | Shared characters between chunks |
Frequently asked questions
Can LangChain scrape JavaScript-heavy sites by itself?
No. LangChain processes text after it has been retrieved. Rendering JavaScript and returning the page content requires a separate fetching tool, such as the scraping API client used in this walkthrough.
What reduces the chance of being blocked?
Proxy rotation, JavaScript rendering and resource blocking together make requests resemble ordinary browser traffic more closely. The original walkthrough relied on the scraping service's built-in parameters for these.
How are credit costs determined?
Charges depend on which parameters are enabled in the GET request. Credit costs per feature: {{OWNER_CREDIT_COST_TABLE}}
Can the resulting chunks feed a RAG system?
Yes. The chunk list produced by the splitter can populate a vector database and be retrieved by a RAG pipeline. Chunk size and overlap are the main settings to tune for retrieval quality.
Why keep the source URL in metadata?
It preserves provenance for each chunk, which helps when you need to trace an answer back to the page it came from or filter results by origin.