Build a Live Web Data Retriever in Dify with MCP
A Dify Agent node can answer questions about a public URL by fetching that page at request time, rather than relying on documents indexed earlier. This guide explains how to connect a remote MCP server, give the agent a single page-fetching tool, control its cost and iteration settings, and verify the result in the agent log.
What you will build
The finished retriever is a Chatflow with three nodes: User Input, Agent and Answer. The Agent node carries one tool that returns a target page as Markdown, and the model composes its reply from that returned text. A request therefore travels from the chat box into the Agent node, out to the remote MCP server and target page, and back through the Answer node to the user.
Fetching is often the difficult part on real sites because of blocked addresses, CAPTCHA pages or content that only appears after JavaScript runs. The remote MCP server used here wraps a web-scraping API that handles proxies and headless-browser rendering, exposing that capability as tools the agent can call.
Live retrieval differs from the Knowledge Retrieval node. The latter searches documents indexed in advance, so a pasted product URL asking for today's price needs a fresh fetch when no indexed copy exists or that copy is stale.
What you need
Before starting, have these four things ready:
- A Dify workspace, either Dify Cloud or self-hosted. The steps were exercised on self-hosted Dify 1.17.1 and on Dify Cloud.
- A model that supports tool calling, from a provider configured in your workspace. A small model can suffice; with the settings below, a locally run 9-billion-parameter model through Ollama and a hosted mini model on Dify Cloud both answered the test questions.
- The Dify Agent Strategies plugin (langgenius/agent), which supplies the FunctionCalling strategy. New workspaces may not include it, so install it from the Dify Marketplace and reload any open app page so FunctionCalling appears under Agentic Strategy.
- An API key for the scraping service, copied from your dashboard. Keep it in exactly one place — the MCP server header in Step 1 — and never in prompts, agent instructions or screenshots.
For a stable test, use a public scraping sandbox product page. The expected result has three fields: a price of £51.77, a stock level of In stock (22 available), and a UPC of a897fe39b1053632. A sandbox keeps those values steady so your run can be compared with the expected output.
Step 1: Connect the remote MCP server to Dify
The remote MCP server is hosted over Streamable HTTP and Dify connects to it directly by URL. Add it once per workspace so apps in that workspace can use the discovered tools. Open Integrations > Tools > MCP and choose Add MCP Server (HTTP). Enter the server URL, a display name and icon, and a server identifier in the three fields at the top of the dialog. The exact field values are held in protected module code0001, which uses the configurable owner origin and a placeholder for your own key.
The dialog first shows an Authentication tab intended for OAuth servers. Instead, open the Headers tab, choose Add Header, add the Authorization header with the Bearer value, and select Add & Authorize. Dify should save the server and load its tools. Leave the Configurations tab at its defaults — in Dify 1.17.1 those were 30 seconds for Timeout and 300 seconds for SSE Read Timeout.
After saving, open the server card's detail panel to confirm the discovered tool list. The Bearer header is the authentication method the MCP server page recommends; it connected successfully in the Dify 1.17.1 tests on 29 September 2026, on both self-hosted and Dify Cloud. Because MCP documentation can change, re-check the current authentication steps on the MCP server page before publishing, and let the first test question in Step 3 confirm that your key works.
If the server does not save or the tool list is empty or outdated, work through this checklist:
- Confirm the URL ends with the MCP path; a different ending makes Dify try the older SSE transport first.
- Confirm the header value begins with Bearer followed by a space.
- For self-hosted Dify, confirm the instance can reach the MCP host over HTTPS; self-hosted MCP calls pass through the SSRF proxy container in the Dify Docker setup.
- Open the detail panel and select Update so the discovered tool list reflects any tools the server has added or changed.
A custom tool built from an OpenAPI schema is the alternative to MCP, for example to call the underlying HTML API with your own forwarded headers. Use MCP when the provider hosts a server and you want Dify to discover tools and schemas for you.
Step 2: Build the agent and give it one tool
A new Chatflow starts as User Input, LLM, Answer. A standalone Agent app type also exists, but a Chatflow lets you add further nodes around the Agent node later. Replace the LLM node with an Agent node: right-click the LLM node or open its menu, hover over Change Node, and select Agent. Set Agentic Strategy to FunctionCalling and choose a tool-calling model.
In Tool list, select the add control, open the MCP tab, expand the connected server and select the page-text tool. Leave Allowed Tools empty. Then open that tool's settings and turn Auto off for the three cost-controlling parameters — auto_mode, premium_proxy and stealth_proxy — selecting False for each, even if False already appears selected. Paste the instruction held in protected module code0002 into Instruction. Insert the User Input query variable into Query, confirm Maximum Iterations is 3, and in the Answer node replace the old text variable (which Dify may flag with a warning after the node change) with the Agent node's text variable.
The instruction in code0002 has a deliberate shape: fetch once with JavaScript rendering off, retry once with rendering on only if the returned text lacks the requested information, answer only from returned page content quoting values exactly, and surface any tool error verbatim before stopping. Keep the phrase that forces the first fetch to skip rendering even when the page might need JavaScript — without it, the model in testing requested rendering on its first call to a JavaScript page. Adapt the field names in the quoting line to your own target fields. Specificity also guards against prompt injection: in testing, a page hid an instruction telling assistants to fetch a second URL. With this instruction the agent ignored it, while a generic instruction led the agent to fetch one page yet claim it had fetched the second as instructed.
Add only the tool the agent needs. The MCP tab offers an Add all option that sends every tool schema to the model on every round; in a like-for-like test with the same local model, the single-tool agent returned the same three field values with a smaller prompt.
Fix the cost-controlling parameters. With auto_mode off and set to False, each call uses the classic proxy and the premium and stealth proxy options stay off. Because those calls render JavaScript by default, the instruction asks for rendering off on the first call. Leave render_js on Auto so the model can request rendering when a page needs it. With Auto off, Dify hides the parameter from the model and sends your fixed value on every call; the agent log in Step 3 shows the model sending only the URL and render_js while Dify adds the three fixed values.
Set Maximum Iterations from the rounds you need. In Dify 1.17.1 with FunctionCalling, each iteration was one model round (one LLM call), and a round could contain several tool calls. A single-fetch question takes two rounds: the tool call in round 1 and the answer in round 2. The default of 3 allows one retry, such as a second fetch with rendering. At a limit of 1 the answer was raw scraped Markdown because no round remained to read it; at 2, a JavaScript page ended with a plan to retry rather than an answer, and Dify still reported the run as successful. When two book URLs were asked about, the model made three tool calls in round 1 including a duplicate, so state that the agent calls the tool once per URL and watch the log while testing.
Step 3: Test the retriever
Open Preview in the Chatflow and ask about the sandbox book page, using the question held in protected module code0003. The expected reply, held in code0004, reports the price, stock level and UPC exactly as the page shows them. Other models will word the answer differently, but the three values should match.
To confirm the values came from the page, expand Workflow Process above the answer, then open Agent, Agentic Strategy and ROUND 1, and expand the tool-call entry. Round 1 holds the tool call and round 2 holds the answer. The tool-call entry shows the exact input, including the three fixed values, and the Markdown returned. For this static page, the first fetch without rendering returned all three fields.
Then test the retry path with the JavaScript-rendered quotes sandbox question held in code0005. Its quotes load through JavaScript, so the first fetch returned the page without them; the instruction's retry line then triggered a second call with rendering on, and in round 3 the model answered that the author was Albert Einstein with tags change, deep-thoughts, thinking and world. Requesting rendering only when needed keeps static pages on the lower-cost fetch. When you adapt the instruction, name the fields the agent must find and ask it to say which fields it found.
If a test misses the expected fields, use this decision checklist:
- The agent answers without calling the tool: verify Query is wired to the User Input query variable and confirm the tool shows as enabled in Tool list. Explicitly mention the tool by name in the instruction; if the model still bypasses it, switch to a different model that supports tool calling.
- The agent calls the tool too often: state in the instruction when to stop calling tools, and set Maximum Iterations to the tool-calling rounds you allow plus one round for the answer.
- The answer mentions an invalid or missing key: open the server's detail panel, choose Edit, and check the header; then check your key and credits on the dashboard. A wrong key still produced a run reported as successful in testing, so read the agent log.
- The page needs JavaScript, clicks or premium proxies: allow render_js for JavaScript content, use a JavaScript scenario for clicks or scrolling, and set premium_proxy to True in Tool Settings for sites needing premium proxies.
Once both test questions return the expected fields, the retriever is tested end to end, and the dashboard shows the credits your tests consumed.
Optional: other options for the retriever
The retriever above is complete without these options; each one changes how the agent finds, fetches or extracts data.
- Selector-based extraction: a dedicated tool returns only the fields your CSS or XPath extract rules select, as JSON. Inspect the page DOM or fetch its HTML first, then create and test stable selectors before configuring the tool, and expect to update them when the target page changes.
- AI extraction: in the tested MCP schema, the page-text tool also exposed an AI query option that runs an AI web-scraping API on the page. Check the current tool schema before relying on it.
- Finding URLs: a fast search tool returns structured search results, so the agent can locate a page before fetching it.
- Product data: dedicated tools exist for Amazon and Walmart that fetch a product by its ID, such as a pricing tool.
- Automatic configuration: an Auto-Mode option picks the lowest-cost configuration that fetches each page. To use it instead of the manual retry, set auto_mode to True in Tool Settings, leave render_js, premium_proxy and stealth_proxy on Auto, and change the instruction so the model passes only the URL. Auto-Mode then controls rendering and proxies itself, with a maximum-cost setting limiting the configurations it may try.
Next steps: from live retrieval to a RAG pipeline
Because the retriever fetches pages live, each answer reflects the current page. When many questions reuse the same pages, a RAG pipeline can index those pages once instead. The pipeline shape is held in protected module code0006: website, scraping service, document ingestion, chunking and indexing, Dify knowledge base, Knowledge Retrieval node, then LLM. This arrangement operates separately from the agent: pages are collected on a recurring schedule, passed into Dify via its knowledge API or a knowledge pipeline, and indexed asynchronously in the background. The fetching step can be handled by a CLI, with an external scheduler triggering that CLI for periodic refreshes.
Choose between the two patterns by use case. Live retrieval fetches the page when the user asks, which suits answers that must match the live page or pages you never indexed. Knowledge-base retrieval searches pages ingested earlier, which suits a known set of pages queried often and refreshed on a schedule. The same data-acquisition layer can support both patterns.
Before publishing, test the agent with a page your users already ask about and read the agent log. Dify can mark a run as successful even when the answer is only an apology, so check the Logs for those answers after launch.
Frequently asked questions
Does Dify support MCP?
Yes. In Dify 1.17.1, remote MCP servers are added under Integrations > Tools > MCP, and Agent nodes can call the tools they discover. The connection dialog accepts custom headers, including the Bearer authorization header used by the remote MCP server described here.
How do I authenticate an MCP server in Dify?
Put the credential in a request header rather than in prompts or tool arguments. In the Add MCP Server (HTTP) dialog, open Headers and add Authorization with a Bearer value. In Dify 1.17.1, that key stays outside the text the model sees.
Dify workflow vs agent: what's the difference?
A workflow is a pre-designed graph of nodes and paths. An Agent node is a single step inside such a graph where the model decides which tools to call. Maximum Iterations caps how many model rounds that step may run.
Is live web retrieval the same as a Dify knowledge base?
No. Live retrieval fetches the page during the current run, so the answer reflects the page as it is now. A knowledge base searches material ingested, chunked and indexed earlier, and keeping it current requires a separate scheduled fetch and re-index.
Does MCP cost extra credits?
No extra surcharge is added for using MCP. Each tool call uses the same credits as the API endpoint it wraps, and the total for a question is the sum of its calls. Options such as JavaScript rendering affect that cost, and the agent log lists each call.
How do I stop a Dify agent from looping?
Set Maximum Iterations to the number of tool-calling rounds you allow plus one round for the final answer, and state clearly in the instruction when to stop calling tools. In Dify 1.17.1 the limit counted model rounds rather than individual tool calls, so inspect the agent log for duplicates.
Should I use MCP or a custom tool in Dify?
Use MCP when the provider hosts an MCP server, because Dify discovers the tools and schemas automatically. Use a custom tool built from an OpenAPI schema when you want to define each request yourself, for example to forward your own headers to the target page.