Skip to content

LangChain Web Scraping Tools Reference: Ten Agent Tools

This Python package turns each web scraping API endpoint into a LangChain Tool, so an agent can fetch a page, run a Google search, look up an Amazon or Walmart product, or pull YouTube data as one step in its reasoning loop. It is code you install into your own project, not a no-code builder integration.

The reference below covers the ten tools, their arguments, the params keys each one accepts, where results are written, and the two tools that break the shared conventions. Read the behaviour section first: most confusion with this package comes from misunderstanding what a tool call returns versus what it saves.

What the package gives you

The package ships ten tools, one for each web scraping API endpoint. Each is a LangChain Tool: you build it with your API key, then call it with .invoke(...).

Before you start you need Python >=3.9,<4.0 and an account with an API key from your dashboard. Install from PyPI with the package's documented install command (see the protected installation module), then either export your key as an environment variable or pass it explicitly to each tool constructor.

There is no global configuration step: every tool's constructor takes api_key explicitly, for example ScrapeUrlTool(api_key=api_key). If you prefer not to rely on the environment, set the key at runtime or hand it to each tool individually.

How every tool behaves (read this first)

Nine of the ten tools follow the same conventions. Understanding them up front prevents the most common surprises:

  • Every call is a real request. Each tool invocation makes an actual API request and consumes credits, whatever other arguments you pass.
  • Results are written to disk first, always. Output lands in a fresh, timestamped subfolder under results_folder (default "scraping_results"), for example scraping_results/20260706_083012/amazon_search_iphone_16.json. There is no option to skip the file write.
  • return_content only changes the return value. With the default False, the string you get back is a brief "saved successfully" note carrying the file path plus a handful of extracted highlights. Setting it to True additionally embeds the complete content or JSON inside that same string. Neither value changes whether the request is sent or the file is written, and neither writes to your terminal on its own.
  • A log is appended on every call. scraping_metadata.jsonl sits in the same timestamped subfolder and records the timestamp, input, params and result type, which is useful for auditing what an agent did across a session.
  • params and headers tolerate stringified input. If a model passes JSON, a Python dict-literal or a URL-query string instead of a real dict, the tool tries to parse it. When calling from your own code, always pass a real Python dict.

Two tools are exceptions. CheckUsageTool takes no arguments, writes no files and returns the raw usage response immediately. YouTubeMetadataTool accepts no params dict at all, only video_id.

The reason output is written to disk first: an agent that does not need to read a full result can keep the file path and hand it to another tool instead of pulling the whole payload into context, which saves tokens.

Common parameters

Every tool except CheckUsageTool and YouTubeMetadataTool accepts the same three parameters on top of its own primary argument. They are documented once here rather than repeated in each tool's section.

ScrapeUrlTool additionally accepts headers (dict, default {}) and custom_filename (str, default None).

Choosing a tool

Use this map to pick the right tool before reading the per-tool detail. The primary argument column is the one required input each tool expects.

ScrapeUrlTool

ScrapeUrlTool fetches any public URL and is the most flexible tool, wrapping the full HTML API. The url argument is required and must include http:// or https://.

Two extra arguments sit on top of the common three:

  • headers (dict, default {}) — custom headers forwarded to the target site. Each key is auto-prefixed with Spb- and forward_headers is set automatically.
  • custom_filename (str, default None) — overrides the auto-generated filename.

Here, params accepts any HTML API parameter. If the response is binary (a screenshot, PDF or image), the tool detects the content type, saves the binary file with the correct extension, and returns file info instead of trying to inline binary data.

The supported params keys below are the most commonly used; all standard HTML API parameters work.

GoogleSearchTool

GoogleSearchTool runs a classic, news, maps or images search and wraps the Google API. Its only required argument is search, the query string, plus the three common parameters.

Image searches are handled differently from text searches. Any base64-encoded image in the response is decoded and written out as its own image file, while plain image URLs are collected into image_links.txt; the complete JSON response is stored next to them. Everything lands in the same timestamped results folder as the rest of the call's output.

One known quirk: when search_type is left out, the underlying request still runs a standard ("classic") organic search, yet the tool records it as Type: web in both the summary and the filename rather than classic. The label is purely cosmetic; the search itself and the saved data are correct.

CheckUsageTool

CheckUsageTool reports your credit balance and concurrency usage. It is the simplest tool: no arguments, no file writes, no return_content. It maps directly to the usage endpoint and returns the raw JSON response (credits used and remaining, concurrency limits, account status) immediately.

AmazonSearchTool and AmazonProductTool

AmazonSearchTool searches Amazon for products and wraps the Amazon Search API. Its required argument is query, plus the common parameters.

AmazonProductTool gets detailed info and reviews for a single product and wraps the Amazon Product API. Its required argument is also named query, but despite the field name it must be an ASIN, not a search term. Its params accept the same keys as AmazonSearchTool: domain, country, currency, language, device, zip_code, light_request, add_html and autoselect_variant.

WalmartSearchTool and WalmartProductTool

WalmartSearchTool searches Walmart for products and wraps the Walmart Search API. Its required argument is query, plus the common parameters.

WalmartProductTool gets detailed info and reviews for a single Walmart product and wraps the Walmart Product API. Its required argument is product_id, the Walmart product ID. Its params accept domain, device, delivery_zip, store_id, light_request and add_html, with the same meanings as in WalmartSearchTool.

ChatGPTTool

ChatGPTTool sends a prompt to ChatGPT and wraps the ChatGPT API. Its required argument is prompt, plus the common parameters. The search param enables live web search so ChatGPT can use up-to-date information (default false); country_code localizes those results when search is on, and add_html includes full page HTML of searched pages.

YouTubeMetadataTool and YouTubeSearchTool

YouTubeMetadataTool returns title, description, views, likes, channel info and more for a video, wrapping the YouTube Metadata API. It is the simplest YouTube tool: no params dict. Its required argument is video_id, the part after v= in a YouTube URL, and it accepts only results_folder and return_content from the common set.

YouTubeSearchTool searches YouTube for videos, channels or playlists and wraps the YouTube Search API. Its required argument is search, plus the common parameters.

Example: an agent that picks its own tools

Pass these LangChain tools to a LangGraph agent and let the model choose which to call and in what order. The reference example uses Gemini through langchain-google-genai, which you install alongside langgraph — neither is bundled with this package — and sets GOOGLE_API_KEY.

In the captured reference run, the agent called AmazonSearchTool for "harry potter book", read ASIN 1546148507 from the result, then called AmazonProductTool to return full product details, with no manual chaining. Because each tool's description embeds its own supported params, a capable model can also pick sensible filters such as sort_by, device or country_code without you hardcoding them.

That run is a dated reference, not a guarantee that every model will chain the same way. Verify behaviour with your own model and key.

Credits and limits

A tool call draws down credits on the same schedule as the endpoint it wraps; the wrapper itself adds nothing on top. So GoogleSearchTool is billed like the Google API, and ScrapeUrlTool with params {"premium_proxy": True} is billed like the HTML API with that flag set. The credit-cost reference lists the exact per-endpoint figures.

An agent chooses both which tools to call and how many times, so one user prompt can fan out into several API requests. Run CheckUsageTool while developing and on a recurring basis in production to spot a runaway loop before it drains your balance.

For ScrapeUrlTool, escalate proxies only as needed:

  1. Start with no premium_proxy or stealth_proxy.
  2. If the site blocks you, retry with params {"premium_proxy": True}.
  3. If still blocked, escalate to params {"stealth_proxy": True}.

Each step costs more credits, so do not jump straight to the top of the ladder.

Troubleshooting

.invoke(...) only prints a "saved successfully" message, not the actual content. This is expected when return_content is False, the default. Every tool writes the full result to disk unconditionally, and the API call and credit spend happen either way; the return string is only a short file-info summary. Pass "return_content": True in the .invoke(...) call to include the actual scraped data, search results or API response in the return value, then call print(result) to display it. Assigning the return value to a variable without print(result) produces no terminal output even when return_content is True.

A scrape returns empty or blocked content. The default scraper is being refused by the target site. Retry ScrapeUrlTool with params={"premium_proxy": True}; if that is still refused, move up to params={"stealth_proxy": True}. Higher rungs cost more credits, so check the credit-cost reference before escalating.

The agent calls the wrong tool, or loops. Make the prompt or system message state plainly which tool to use and what condition ends the task. If it keeps choosing badly or repeating, a stronger model usually selects tools and stops more reliably. CheckUsageTool tells you whether the agent is genuinely making too many API calls.

Resources

  • Package source code on GitHub
  • Package on PyPI — installation and version history
  • A walkthrough of web scraping with LangChain
  • HTML API · Google API · Data Extraction · JavaScript Scenario · Amazon API · Walmart API · ChatGPT API · YouTube APIs
  • Dashboard — manage your API key, view usage, upgrade your plan
  • Knowledge Base — searchable troubleshooting articles
Tool-to-argument map
ToolRequired argumentWraps
ScrapeUrlToolurlHTML API
GoogleSearchToolsearchGoogle API
CheckUsageToolnoneusage endpoint
AmazonSearchToolqueryAmazon Search API
AmazonProductToolquery (an ASIN)Amazon Product API
WalmartSearchToolqueryWalmart Search API
WalmartProductToolproduct_idWalmart Product API
ChatGPTToolpromptChatGPT API
YouTubeMetadataToolvideo_idYouTube Metadata API
YouTubeSearchToolsearchYouTube Search API

Argument names and endpoint pairings come from the per-tool argument tables; CheckUsageTool has no required argument.

Frequently asked questions

Does return_content control whether the API call happens?

No. The API call and the file write happen on every invocation regardless of return_content. That flag only decides whether the full content or JSON is inlined in the return string; the default False returns a short summary with the file path and a few highlights.

Why does my script print nothing after .invoke(...)?

Assigning the return value to a variable does not print it. Add print(result). If you also want the actual data in that string rather than a file summary, pass "return_content": True.

What is the difference between AmazonSearchTool and AmazonProductTool?

AmazonSearchTool takes a search query and returns matching products. AmazonProductTool takes a single ASIN in its query field and returns detailed info and reviews for that one product.

Which tools skip the common params dict?

CheckUsageTool takes no arguments at all and writes no files. YouTubeMetadataTool accepts only video_id plus results_folder and return_content — it has no params dict.

How do I stop an agent from burning credits?

Call CheckUsageTool during development and periodically in production to see whether the agent is making excessive requests. Tighten the prompt or system message about which tool to use and when to stop, and consider a more capable model if tool selection or looping is poor.