Skip to content

LangChain Web Scraper: Fetch, Parse and Chunk HTML for RAG

LangChain is strong at organising text for retrieval and generation, but it does not fetch or render web pages on its own. A practical pipeline therefore pairs it with a scraping layer that returns usable HTML, then converts that HTML into clean text and manageable chunks. This guide walks through that sequence in Python, using a scraping API client for the network side and LangChain's text splitter for the processing side.

What the pipeline actually does

The workflow has four distinct stages, and each one can fail independently. First, a request fetches the target page. Second, the response body is parsed so that markup is removed and visible text remains. Third, that text is divided into overlapping chunks. Fourth, the chunks are printed or passed to a downstream store such as a vector database.

Keeping the stages separate matters because a blank or blocked result is a fetch problem, while garbled text is usually a parsing problem, and oversized or fragmented chunks are a splitting problem. Diagnosing the stage first saves a lot of guesswork.

The scraping layer supplies what LangChain does not: JavaScript rendering, proxy rotation, geo-targeted routing and resource blocking. LangChain then handles the text-processing side that feeds embeddings and retrieval-augmented generation.

Prerequisites and setup

You need Python 3.6 or newer, an API key for the scraping service, and three Python libraries: the scraping client, BeautifulSoup and LangChain. Install them from a terminal or command prompt with pip.

The install command is supplied as an exact module and should be run as written. After installation, sign in to the scraping service, copy the API key from the dashboard, create a project directory and add an empty .py file for the script.

Account and trial terms: {{OWNER_TRIAL_TERMS}}

Step 1: initialise the client and build the search URL

Start by importing the scraping client and creating an instance with your API key. The key is a credential, so keep it out of version control and load it from an environment variable in real projects rather than pasting it into the file.

Next, accept two inputs — a business type and a location — and encode them into a YellowPages search URL. The base path and the two parameter names are fixed by the target site, so they are shown exactly in the supplied code module. The urlencode helper turns the input values into a valid query string.

Step 2: fetch rendered HTML

The fetch function defines a JavaScript scenario that waits 3,000 milliseconds before extraction, giving dynamic content time to appear. JavaScript rendering is on by default, so no extra flag is needed for that part.

The GET call passes three parameters that are worth understanding:

  • js_scenario carries the pre-extraction instructions, such as the wait.
  • block_resources skips images, CSS and fonts, which reduces bandwidth and speeds up the response.
  • country_code routes the request through a proxy in the chosen country, which helps with region-specific content.

When the call succeeds, the response text is raw HTML. Printing it confirms the fetch worked before any parsing is added. In the original walkthrough the example searched for dentists in Orlando.

Step 3: strip markup and keep metadata

A parsing function takes the raw HTML and the source URL. BeautifulSoup reads the markup, get_text with a space separator and strip enabled produces visible text, and the page title is captured when present, falling back to a placeholder otherwise. The function returns the text, the title and a small metadata dictionary containing the source URL and title.

Call this function immediately after the response is received and assign its three return values. The printed output should now read as prose rather than tags and attributes. If it does not, check that the response actually contained the page you expected before adjusting the parser.

Step 4: split the text into chunks

Import RecursiveCharacterTextSplitter at the top of the script, then create a splitter with a chunk size of 1,000 characters and an overlap of 200. The overlap preserves context that would otherwise be cut at a boundary, which matters when neighbouring chunks are retrieved independently.

Run the splitter on the parsed text and store the resulting list. A loop then prints the title, the metadata, the number of chunks and each chunk with an index. That output is the hand-off point: the same list can be written to a vector store or passed to a retrieval chain.

Decision checklist before you scale up

Use this checklist to separate the common failure points:

  1. Empty or blocked response — revisit the fetch stage: confirm the API key, the target URL and any rendering wait.
  2. HTML returned but text looks wrong — revisit parsing: check that the selector-free text extraction matches the page structure.
  3. Text is clean but chunks are poor — revisit splitting: tune chunk size and overlap against your retrieval tests.
  4. Everything works but costs rise — review which request parameters are enabled, since credit charges depend on the configuration of the GET call.
  5. Volume increases — consider concurrency and retry handling rather than sending requests one at a time.

Operational options worth knowing

Concurrency is possible because the client uses the blocking requests library; the concurrent.futures approach is a common fit for HTTP work. Automatic retries exist for both the GET call and the JavaScript scenario, although behaviour depends on the error type and request configuration.

Billing on failed requests: {{OWNER_500_STATUS_NO_CHARGE_POLICY}}

Resource blocking can also be used to skip tracking scripts that slow responses or trigger bot protection.

Compliance and proxy handling: {{OWNER_GDPR_AND_PROXY_CLAIMS}}

Field and control map

The table below summarises the inputs and settings used in the walkthrough so you can adapt them deliberately.

ControlWhere it appearsWhat it changes
search_termsURL queryBusiness type searched
geo_location_termsURL queryLocation searched
js_scenarioGET parametersPre-extraction browser instructions
block_resourcesGET parametersSkips images, CSS and fonts
country_codeGET parametersProxy country for the request
chunk_sizeSplitterMaximum characters per chunk
chunk_overlapSplitterShared characters between chunks

Frequently asked questions

Can LangChain scrape JavaScript-heavy sites by itself?

No. LangChain processes text after it has been retrieved. Rendering JavaScript and returning the page content requires a separate fetching tool, such as the scraping API client used in this walkthrough.

What reduces the chance of being blocked?

Proxy rotation, JavaScript rendering and resource blocking together make requests resemble ordinary browser traffic more closely. The original walkthrough relied on the scraping service's built-in parameters for these.

How are credit costs determined?

Charges depend on which parameters are enabled in the GET request. Credit costs per feature: {{OWNER_CREDIT_COST_TABLE}}

Can the resulting chunks feed a RAG system?

Yes. The chunk list produced by the splitter can populate a vector database and be retrieved by a RAG pipeline. Chunk size and overlap are the main settings to tune for retrieval quality.

Why keep the source URL in metadata?

It preserves provenance for each chunk, which helps when you need to trace an answer back to the page it came from or filter results by origin.