Skip to content

Build an AutoGPT Agent That Scrapes Live Pages and Returns Ranked JSON

This guide assembles a self-hosted AutoGPT workflow that accepts a news-page URL, fetches the rendered page through an external scraping API, asks a language model to keep only strongly relevant stories, and returns structured JSON. The worked example is a World Cup news scout, but the block pattern transfers to other monitoring tasks.

How the pieces fit together

AutoGPT is an open-source platform for creating and running autonomous agents. Its visual builder chains single-purpose blocks: one gathers input, another calls an external service, another runs a model, and a final block emits the result. That separation matters because the agent still needs dependable live-web access. Plain requests can miss JavaScript-rendered content, trip anti-bot controls, or return location-dependent results. Delegating rendering, proxies and related handling to a scraping API keeps that infrastructure outside the agent canvas. Checklist before you start: Docker with Compose running, Git, Node.js with npm, local ports 3000, 8001 and 8006 free, a scraping API account and key, and a model-provider key. The example uses GPT-OSS 120B via OpenRouter, so no OpenAI key is needed for that path; other providers supported by the AI block are interchangeable with their own keys. Protected setup modules: code0001, code0002, code0003, code0004, code0005, code0006, code0007, code0008.

Assemble the five-block workflow

Start the platform from the autogpt_platform directory, confirm the containers are up, then open the builder at the local frontend address in protected module code0007. Create a local account — it stays on the bundled local service — open the Build tab, start a blank agent and save it under a descriptive name such as World Cup News Scout. Add an Agent Input block named page_url with a title and description explaining that it expects a football news page; leave its default value empty so the real URL is supplied at run time. Next add a Fill Text Template block with a page_url value and paste the request format from protected module code0012, which URL-encodes the target and enables JavaScript rendering. Connect the input block's Result pin to that value pin. Then add a Send Authenticated Web Request block: link the template output to its Url input, switch the method to GET, turn JSON format off because the first pass returns page content rather than JSON, and create an HTTP credential carrying the Authorization header from protected module code0014. Keeping the key in a header rather than the URL is the recommended pattern; the older query-parameter form is deprecated. Protected modules: code0009, code0010, code0011, code0012, code0014.

Add judgment, output and a run

Add an AI Structured Response Generator block. Create two empty prompt values, page_url and page_content, then wire Agent Input to the first and the web-request block's Response pin to the second. Select the model and attach its credential. Paste the editorial prompt from protected module code0015: it defines inclusion topics, excludes club and transfer coverage, scores relevance from 0 to 100, drops anything below 75, caps results at ten, removes duplicates, forbids invented details, and requires full URLs. Define the expected format with the five fields in the table below, each paired with a plain-language description so the model shapes its JSON accordingly. Finish with an Agent Output block named world_cup_news linked to the AI block's Response pin, save, then run and enter a news homepage such as the one in protected module code0016. A successful run returns an overview, a story count and ranked story objects, each with headline, summary, URL, date, score and reason; the shortened example in protected module code0017 shows the shape. Protected modules: code0015, code0016, code0017.

Troubleshooting by failure layer

Match the symptom to the layer that failed. Docker not starting: start the Docker daemon, verify it, then relaunch the stack (code0018, code0019). Port already in use: identify the process on port 3000 with the OS-appropriate check (code0020, code0021), stop it or remap the host port. Builder page not loading: check container status (code0022) and inspect recent logs (code0023). Scraping authentication error: confirm the credential is selected, the header format in protected module code0025 is exact, and credits remain. AI block 401 or missing authentication: verify the full provider key including its prefix, and that paid-model funds are available. Protected modules: code0018, code0019, code0020, code0021, code0022, code0023, code0025.

Cut token use with AI extraction, then reuse the pattern

Returning the whole page forces the model to wade through navigation, ads and unrelated stories, and on large pages the useful content can fall outside the context window. Adding an ai_rules value to the template and appending the extraction parameter from protected module code0027 makes the scraping service return only the requested story fields as JSON; then re-enable JSON format on the request block. In the captured run, roughly 120,000 characters of HTML shrank to about 14,000 characters of extracted JSON — approximately 30,000 tokens down to 3,500, an 88% reduction at the usual four-characters-per-token estimate. Savings vary by page, and extraction adds request credits, so weigh token price against page size and run frequency. The same pattern extends to news monitoring, competitor tracking, product monitoring, lead enrichment and research assistance: swap the input URL, rewrite the prompt's inclusion rules, and keep the block chain intact. Protected modules: code0026, code0027, code0028.

Expected output fields
FieldDescription
source_urlThe original football news webpage that was analyzed
site_nameThe name of the football news website or publication
overviewA concise summary of the World Cup coverage found on the page
story_countAn integer equal to the number of qualifying World Cup stories returned
storiesRanked array of qualifying story objects

Expected-format fields for the AI block; each story object must contain headline, summary, url, published_date, world_cup_relevance_score and relevance_reason, with scores of 75 or higher only.

Token comparison from the captured run
Input sent to the LLMSizeApproximate tokens
Complete rendered HTML120,000 characters30,000 tokens
Extracted JSON14,000 characters3,500 tokens

Captured single-run figures using the common estimate of about four characters per token; actual savings vary by page and extraction count.

Frequently asked questions

How do I build an AutoGPT agent?

Add blocks in the visual builder and link their input and output pins so data flows from entry to final result.

Is AutoGPT a no-code tool?

The builder is low-code and visual, so many workflows need no orchestration code. Self-hosting and custom blocks still need technical knowledge.

Can I self-host AutoGPT?

Yes. This guide runs the platform locally with Docker Compose, while a hosted option is also available.

Do I need an OpenAI API key to run AutoGPT?

No. The example routes GPT-OSS 120B through OpenRouter, and other supported providers can be used with their matching keys.

What is the AutoGPT Agent Builder?

It is the visual canvas for assembling reusable blocks covering inputs, API calls, models, processing and outputs.

What is the Agent Protocol?

It is a standard interface for talking to compatible agents. The visual workflow here does not require manual setup, but it can help when linking agents to other apps.

Can I create my own custom blocks?

Yes, developers can add custom blocks when built-ins are insufficient, for example for credential handling or bespoke parsing.

Can this workflow complete tasks other than news monitoring?

Yes. The same pattern suits competitor tracking, product monitoring, lead enrichment, research and other work needing current web data.

curl -fsSL https://setup.agpt.co/install.sh -o install.sh && bash install.sh
{{APP_BASE_URL}}/api/v1?url={{ page_url | urlencode }}&render_js=true
{
  "source_url": "https://espnfc.com",
  "site_name": "ESPN FC",
  "overview": "The page contains several stories covering the latest FIFA World Cup matches, teams, and knockout-stage developments.",
  "story_count": 10,
  "stories": [
    {
      "headline": "Argentina survive again to book England semifinal",
      "summary": "Julian Alvarez's extra-time strike earned Argentina a place in the World Cup semifinals.",
      "url": "https://espnfc.com/football/story/_/id/49339687/argentina-switzerland-2026-world-cup-julian-alvarez-england",
      "published_date": "",
      "world_cup_relevance_score": 92,
      "relevance_reason": "Direct match report of a World Cup knockout-stage game determining a semifinalist."
    }
  ]
}
{
  "stories": {
    "description": "Example post details.",
    "type": "list",
    "output": {
      "headline": "The story headline",
      "summary": "The story summary or teaser",
      "url": "The link to the full story",
      "published_date": "The publication date when available"
    }
  }
}
{{APP_BASE_URL}}/api/v1?url={{ page_url | urlencode }}&render_js=true&ai_extract_rules={{ ai_rules | urlencode }}