Skip to content

Choose a web scraping integration for your workflow

Choose where a web scraping request belongs in your existing workflow: an application, a terminal, an AI assistant or an automation.

OUTPUT FORMATSHTMLJSONMarkdown

This directory covers API, CLI, MCP, Python and Node.js SDKs, coding agents, IDEs, AI builders and automation platforms. Use the categories to find the relevant setup guide and assess the connection before sending production data.

Browse integrations

51 entries

Featured

Product Integrations

SDKs

Coding Agents

IDEs

AI Development Frameworks

LLMs

Automation

AI Builders

MCP Marketplaces

How to choose a connection point

Start with where the fetched data will be used. Application code can call an HTTP API or a Python or Node.js SDK. Terminal workflows can use a CLI. AI assistants use an MCP server, while coding agents and IDEs can expose the scraping step inside the development environment.

For no-code workflows, compare the n8n, Make and Zapier setup guides. AI-builder entries cover Lovable, Replit, Base44, Bubble and v0. Docker Hub, Composio and Smithery are MCP distribution or discovery channels. Follow the relevant guide and check its current setup requirements; a directory entry is not an already configured connection.

What is in the directory

The directory is grouped into categories you can filter. Each entry links to a setup page or repository. The categories are:

  • Featured — Claude Code, Codex, Cursor, and VS Code.
  • Product Integrations — API, CLI, and MCP.
  • SDKs — Python SDK and Node.js SDK.
  • Coding Agents — Claude Code, Codex, Qwen, OpenCode, Goose, Amp, Warp, Gemini CLI, Augment, Conductor, Cline, Roo Code, Qodo, Amazon Q Developer, Zencoder, JetBrains AI Assistant, Copilot Coding Agent, Rovo Dev CLI, and Charmbracelet Crush.
  • IDEs — Cursor, VS Code, Visual Studio, Windsurf, Kiro, Zed, JetBrains, and Trae.
  • AI Development Frameworks — LangChain.
  • LLMs — ChatGPT, Claude, and Perplexity.
  • Automation — n8n, Make, and Zapier.
  • AI Builders — Lovable, Replit, Base44, Bubble, and v0.
  • MCP Marketplaces — Docker Hub, Composio, and Smithery.

Use the search box to filter by name, and the category buttons to narrow the list. The Clear all control clears selected categories while keeping the search text.

A worked example: adding a scrape step to an n8n workflow

Suppose an n8n workflow writes rows to a database and needs a page-fetch step first. Open the n8n integration guide, check the available node or HTTP-request configuration, and add the fetch before the database write. Supply credentials through the workflow platform's supported credential mechanism. Keep the fetch and write steps separate so you can reason about retries without unintentionally repeating a database write. The same planning applies to Make scenarios and Zapier Zaps.

Compliance posture before you commit

Before sending production data, check the scraping provider's current security and privacy documentation. If SOC 2 Type 2 evidence or GDPR-related terms matter to your organisation, confirm their actual scope, dates and applicability with the provider and your security or legal team. Review infrastructure security, access controls, data handling, endpoint security and vulnerability management. A badge or marketplace listing does not establish that every integration has the controls you need.

Three different 'it doesn't work' cases

Separate three cases when troubleshooting.

  • The tool is not listed. Check whether a standard HTTP API or MCP connection can meet your needs and identify the missing setup requirements.
  • The tool is listed but not connected. Follow the guide, configure credentials and check the request settings. A directory entry alone does not establish a live connection.
  • The connection runs but the target blocks it. Inspect the target response and scraping configuration before assuming the integration itself is broken.

If your tool is missing

If your tool is missing, document its name, the data you need, where the scraping request should run and which authentication or deployment constraints apply. Use that checklist to assess a generic HTTP or MCP connection or to discuss integration support with your chosen provider. Do not assume a native connector exists until its setup requirements are documented.

Frequently asked questions

Should I use the API or an SDK?

Use the API if you want a single HTTP call with no client library. Use the Python or Node.js SDK if you want the request wrapped in a language-native client. Both are listed under Product Integrations and SDKs respectively.

What is the MCP server for?

An MCP server exposes tools to AI assistants that support the Model Context Protocol. Check the setup guide and the distribution channel, such as Docker Hub, Composio or Smithery, then configure credentials and permissions for your environment.

Does the compliance badge cover every integration?

No. Assess the provider's documented compliance scope separately from the integration entry. Check the specific controls, data flows and contractual terms you need before routing production data.

My tool is not listed. What now?

Identify the tool, intended workflow and required authentication. Check whether a documented HTTP API or MCP connection can meet those requirements, and confirm native-connector availability with your chosen provider.