Skip to content

Playwright MCP Web Scraping in Cursor: Smithery Server List to JSON

Model Context Protocol gives a language model a standard way to call outside tools, and Playwright MCP turns that into browser control: open a URL, read the page, click, fill, screenshot, close. This guide walks through the whole chain in Cursor — install the Playwright MCP server, add a filesystem MCP so the model can write results, register both in Cursor's MCP configuration, and run a one-page scrape of the Smithery MCP server listing into a JSON file. It also shows where the approach breaks: hidden HTML attributes, repeated pagination, and non-deterministic model behaviour.

What MCP and Playwright MCP actually do

MCP is an open standard introduced by Anthropic in November 2024 that lets LLMs call external tools. It is supported by model providers including Google and OpenAI, and by apps such as Cursor, Windsurf and Claude Code. Each connected MCP server publishes a list of tools with parameters; the model chooses which tools to call and in what order.

Playwright MCP is one such server. It wraps Playwright, the browser automation framework, and exposes tools for navigation, reading visible page text, clicking, filling inputs, taking screenshots and closing the browser. Instead of writing selectors by hand, you describe the task in natural language and the model chains tool calls to complete it.

A typical scraping chain looks like this: playwright_navigate opens the URL, playwright_get_visible_text returns the page text, playwright_close ends the browser session, and write_file (from the filesystem MCP) saves the structured result. No custom scraping code and no DevTools selector hunting are required for that path.

Why an MCP-driven browser is different from pasting HTML into a chat

Without an MCP, the practical ceiling is pasting page HTML into a chat and asking for selectors or a snippet. With an MCP, the model can fetch fresh content at run time, act on it, and hand results to another tool through the same specification. That is the real shift: the model stops being a text generator and starts sequencing defined, reusable operations.

It also means the model never needs to understand the internals of a tool. It supplies inputs, receives outputs, and decides the next step. The trade-off is that the model, not your code, owns the control flow — which is exactly where reliability questions begin.

Before you start: prerequisites

You need an MCP-compatible app. This walkthrough uses Cursor, but the same principles apply to Claude Code and other compatible clients. You also need Node.js, npm and npx installed, because the Playwright MCP server is downloaded and run through npx. Keep those tools current so the model sees the latest MCP tool definitions.

The Playwright MCP used here is the Execute Automation server, which offers a broader tool set than the Microsoft Playwright MCP and is well suited to scraping. The filesystem MCP comes from the Model Context Protocol servers repository.

Setup walkthrough: five steps in Cursor

Step 1 — install the Playwright MCP globally. The install command is provided in the protected module code0001. The -g flag makes the package available to npx, and the install also pulls down the browsers Playwright needs, which can take a few minutes; the verbose log level lets you watch progress.

Step 2 — add the filesystem MCP. This server gives the model access to specific local directories so it can write the scraped JSON. You can let npx fetch it on first use; only the Playwright install is slow enough to warrant doing ahead of time.

Step 3 — register both servers in Cursor's MCP configuration. The exact JSON is in protected module code0002. On macOS and Linux the file lives at ~/.cursor/mcp.json; on Windows it is C:\Users\UserName\.cursor\mcp.json. The filesystem entry includes the directory the model is allowed to touch, so point it at a working folder rather than a broad path.

Step 4 — enable Agent mode, which Cursor calls YOLO mode. Without it, Cursor asks for approval on every MCP tool call. With it, the model can chain calls without interruption.

Step 5 — test the wiring. Restart Cursor if needed, then give an explicit instruction that names the MCP, for example: use the Playwright MCP to read the visible text of a page and save its headings to a text file. You should see a sequence of tool calls for opening the browser, reading content and writing the file. If the model says it cannot see the MCP tools, the registration failed; if it writes plausible-looking content without any tool calls, it hallucinated instead of using the tools.

What tools the two servers expose

Once both servers are registered, the model has a fixed menu of operations. The Playwright server covers navigation, screenshots, clicking, iframe clicks, form filling, selection, hovering, JavaScript evaluation, console logs, closing the browser, HTTP GET/POST/PUT/PATCH/DELETE calls, response waiting and assertion, custom user agents, visible text and HTML retrieval, history navigation, dragging, key presses and PDF export. The filesystem server covers reading one or many files, writing and editing files, creating directories, listing directories, moving files, recursive search, file metadata and listing the allowed directories.

The full tool inventory, with each tool name and its description, is preserved in the table below. Two entries are worth flagging because their descriptions are easy to misread: playwright_post is described as sending an HTTP PUT request, and playwright_put is also described as sending an HTTP PUT request. Treat the table as the authoritative record of what the captured server version advertised rather than assuming the description matches the verb in the name.

Running the Smithery scrape

The prompt that worked best for the Smithery homepage asked the model to use Playwright MCP to scrape the list of MCP servers, restricted the scope to the homepage, and specified the output shape: name, URL, description, type (remote or local), icon URL, and download count, saved to JSON with the URL as the unique key. The URL pattern was supplied explicitly as https://smithery.ai/server/@{username}/{mcp_name}.

Cursor ran the task with the claude-3.7-sonnet model. It opened a browser window, called playwright_get_visible_text, processed the text, and wrote mcp_servers.json to the Desktop. The run took just over a minute.

Under the hood nothing exotic happens: the model sees the available tools, picks a sequence, supplies inputs, reads outputs and decides the next action. It does not inspect what happens inside each tool.

What the output actually contained

The resulting file held details for 9 servers, keyed by URL. Name, description, URL and type came through correctly. Two fields did not. The downloads value stayed a display string such as "6.74k" rather than an integer, and icon_url was null for every entry because icon URLs live in HTML tag attributes, not in visible text.

A representative entry from the captured output is preserved in protected module code0003. The model's own summary confirmed the structure: URL as key, then name, description, type, downloads as shown on the site, and icon_url set to null because icons were not directly visible in the text content.

This is the clearest illustration of the visible-text boundary. If a field is not rendered as text, playwright_get_visible_text cannot see it. Recovering icon URLs or href values requires either playwright_get_visible_html (subject to context limits) or a rule that reconstructs the value from a known pattern.

Where MCP-based scraping breaks down

Two classes of limitation showed up in practice.

Context limits. The full HTML of a page may not fit in the model's context window, so the workflow falls back to visible text or a slice of the page. Anything embedded in attributes — links, page titles, descriptions, favicons — is then out of reach. In the Smithery run, icon URLs were lost for exactly this reason, and the first attempts could not recover linked-page URLs from href attributes. The workaround was to have the model construct URLs from the name using a supplied pattern.

Repetition limits. LLMs are not deterministic: the same prompt does not reliably produce the same output. When the task was extended to roughly 60 URLs with a page counter, Cursor typically stopped at 13–14 pages — asking for approval on each page, writing incomplete data, or halting outright. Claude Desktop managed 3–4 pages at best. Cursor also has a built-in cap that pauses MCP tool execution after 25 tool calls and asks the user to continue; a community script exists to work around it, but it requires extra installation steps.

The practical conclusion: code and APIs remain the reliable way to repeat a defined task at scale.

MCP or scraping API? A decision checklist

Use the questions below to pick the right tool before you invest in either setup.

  1. Is the job a one-off or exploratory? If you can inspect the output in minutes and re-prompt until it looks right, MCP is a good fit — no code, no selectors, fast iteration.
  2. Does every field you need appear as visible text? If any field lives in an attribute (icons, hrefs, data-* values, meta tags), expect to lose it or to reconstruct it from a pattern.
  3. Is the page count small? Single pages are comfortable. Dozens of pages with a counter tend to stall around the 13–14 page mark in Cursor, and the 25-tool-call cap adds friction.
  4. Does the job need to run on a schedule with identical output? Non-deterministic model behaviour makes MCP a poor fit for recurring pipelines.
  5. Will the target site push back with anti-bot defences or pagination? Those are exactly the bottlenecks a scraping API is built to absorb.

If the answers point to a repeatable, multi-page or attribute-heavy job, a scraping API with predictable output is the sturdier choice. If the answers point to a quick, visible-text, single-page extraction, the MCP route saves real setup time.

The API alternative, and how it compares on the same task

A scraping API moves the AI prompt inside the request itself. Instead of installing servers, editing configuration files and hoping the model chains the right tool calls, you send a URL plus an extraction instruction and receive structured output. The API handles proxies, browser rendering, pagination and anti-bot measures on its side.

The same Smithery task can be expressed as an API call with an AI query and extraction rules for server URL, name, description, type and downloads. The captured response for that approach returned HTTP 200 with a list of server objects, including download counts in the same display format (for example "22.39k") and relative URLs such as /server/@smithery/toolbox. A sample of that response is preserved in protected module code0005.

The comparison is not about which tool is more impressive. It is about where the control flow lives. With MCP, the model decides the sequence at run time. With an API, the sequence is fixed by the request, which is what makes repeated runs predictable.

Choosing between an MCP browser run and a scraping API
QuestionMCP browser runScraping API
Job typeOne-off, exploratory, inspectable in minutesRecurring, scheduled, or pipeline-driven
Field locationVisible text only; attributes are lostRules can target structured fields directly
Page countSingle pages comfortable; ~13–14 pages before stalling in CursorBuilt for pagination and multi-page runs
Output consistencyNon-deterministic across runsPredictable per request
Anti-bot handlingNot addressed by the MCP itselfHandled by the API
Setup effortInstall servers, edit mcp.json, enable Agent modeSend a request with an extraction instruction

The page-count and consistency rows reflect the captured Cursor and Claude Desktop runs described in this guide, not a benchmark of every MCP client.

Frequently asked questions

Can Playwright MCP take screenshots?

Yes. The server exposes playwright_screenshot, which captures the current page or a specified element via an optional CSS selector. For the screenshot to be useful to the model, the model and the app running it must support image input.

Can Playwright MCP record video?

No. The Playwright MCP server used in this walkthrough cannot record video of its browser interactions.

How do you integrate Playwright MCP with Cursor?

Install the server globally with the npm command in protected module code0001, then register it in Cursor's MCP configuration file at ~/.cursor/mcp.json (or the Windows equivalent). The exact JSON is in protected module code0002. Restart Cursor and enable Agent mode so tool calls run without per-call approval.

What is the difference between Chrome Dev MCP and Playwright MCP?

Chrome Dev MCP gives an AI agent access to a Chrome instance's developer tools console, aimed at debugging and performance work with low-level tools. Playwright MCP offers higher-level operations such as playwright_get_visible_text, which returns a page's text in a single call — a better fit for scraping tasks.

Why did the downloads field come out as a string like "6.74k"?

The model read the value as displayed on the page. Converting "6.74k" to an integer requires an explicit instruction or a post-processing step; the visible-text tool does not normalise numbers for you.

Why was icon_url null for every server?

Icon URLs sit in HTML tag attributes, not in visible text. playwright_get_visible_text cannot see them. Recovering them needs playwright_get_visible_html (subject to context limits) or reconstruction from a known URL pattern.

{
  "mcpServers": {
    "playwright": {
      "command": "npx",
      "args": ["-y", "@executeautomation/playwright-mcp-server"]
    },
    "filesystem": {
      "command": "npx",
      "args": [
        "-y",
        "@modelcontextprotocol/server-filesystem",
        "/Users/kd/Desktop"
      ]
    }
  }
}
{
 "https://smithery.ai/server/@smithery/toolbox": {
    "name": "Toolbox",
    "description": "Example post details.",
    "type": "Remote",
    "downloads": "6.74k",
    "icon_url": null
  },
  "https://smithery.ai/server/@smithery-ai/server-sequential-thinking": {
    "name": "Sequential Thinking",
    "description": "Example post details.",
    "type": "Remote",
    "downloads": "531.96k",
    "icon_url": null
  },
  "...7 more servers...": ""
}
Response HTTP Status Code:  200
[
  {
    "url": "/server/@smithery/toolbox",
    "name": "Toolbox",
    "description": "Example post details.",
    "type": "Remote",
    "downloads": "22.39k"
  },
  {
    "url": "/server/@wonderwhy-er/desktop-commander",
    "name": "Desktop Commander",
    "description": "Example post details.",
    "type": "Local",
    "downloads": "303.87k"
  },
  {
    "url": "/server/@smithery-ai/server-sequential-thinking",
    "name": "Sequential Thinking",
    "description": "Example post details.",
    "type": "Remote",
    "downloads": "133.86k"
  },
  {
    "url": "/server/@browserbasehq/mcp-browserbase",
    "name": "Browserbase",
    "description": "Example post details.",
    "type": "Remote",
    "downloads": "28.45k"
  },
  {
    "url": "/server/@smithery-ai/github",
    "name": "Github",
    "description": "Example post details.",
    "type": "Remote",
    "downloads": "32.87k"
  },
  "...39 more servers..."
]