ScrapeGraphAI Tutorial: Local LLaMA Scraping with Ollama
ScrapeGraphAI lets you describe the data you want in ordinary language and have a language model work out which parts of a page or file to return. This guide walks through a local setup with Ollama and LLaMA, then shows four graph types: JSON extraction, website extraction, search-result extraction and an audio summary built with OpenAI models. The code modules below are the exact contracts to follow; the prose explains how the pieces fit together and where setups commonly go wrong.
Why use AI for scraping?
Classic scrapers depend on selectors and page structure staying stable. When a site changes its layout, those selectors break and the script has to be repaired. An AI-assisted scraper instead reads the page content and works out what matches your prompt, so it can tolerate more structural variation. That reduces maintenance effort, but it does not remove the need to check output quality, respect target-site policies and handle blocking.
Prerequisites
You need basic Python skills, an up-to-date Python 3 installation, macOS, Linux or Windows 10+, and a terminal plus code editor. The original walkthrough also points to a companion GitHub repository for the sample code.
Install Ollama and choose a model
Ollama handles downloading and running local language models. Install it from the official site for your operating system. Installing Ollama does not pull a model by itself, so you then choose one. The walkthrough lists LLaMA for general text work, Mistral for large datasets and pipelines, Phi for precision tasks, Gemma for multilingual sources and Gemini for frequently updated sites. Model benchmarks in the ScrapeGraphAI repository can help you compare options, and LLaMA is a reasonable starting point when you are unsure.
What the B in 8B or 70B means
The B stands for billion parameters: 8B is eight billion and 70B is seventy billion. More parameters usually mean more capacity to learn complex patterns and finer distinctions on difficult tasks, but they also raise compute, memory, storage, electricity and cooling demands. The walkthrough gives LLaMA 8B a minimum of 8GB RAM and a file size of about 4.7GB, while LLaMA 70B is about 40GB and needs over 32GB RAM. Larger is not automatically better; match the model to your hardware and task.
Pull the model and start the local server
Pull LLaMA 3 and the embedding model, then run the model once to check that it responds. Close the interactive prompt with /bye when you are done. Start the Ollama server so ScrapeGraphAI can reach it at 127.0.0.1:11434. Keep that terminal open. If the port is already busy, Ollama may have started automatically after installation.
Install ScrapeGraphAI and Playwright
Install the scrapegraphai package with pip, or add it to a Poetry project if you prefer isolated environments. The package brings in Playwright for browser automation. On a first Playwright install, download the browser components with the install command before running any graph.
Extract structured data from JSON
The JSON example uses a local demo.json file containing five products with prices, manufacturers, categories and reviews. You import JSONScraperGraph, build a graph_config with the Ollama model, temperature 0, JSON output format and the local base URL, and keep the embedding model set to ollama/nomic-embed-text even if you change the main model. Read the file with UTF-8 encoding, then pass a prompt asking for product names, average ratings and prices. The run returns a Python dictionary. In the supplied sample output, Product A is 4.5 at 25.99, Product B is 4 at 45.5, Product C is 4.5 at 15.75, Product D is 4.5 at 29.99 and Product E is 4.5 at 55.0. ScrapeGraphAI can also handle XML, CSV and PDF inputs.
Python example
{
"products": [
{"name": "Product A", "average_rating": 4.5, "price": 25.99},
{"name": "Product B", "average_rating": 4, "price": 45.5},
{"name": "Product C", "average_rating": 4.5, "price": 15.75},
{"name": "Product D", "average_rating": 4.5, "price": 29.99},
{"name": "Product E", "average_rating": 4.5, "price": 55.0},
]
}Scrape a website with SmartScraperGraph
SmartScraperGraph takes a URL and a prompt. The example uses Wired and asks for all titles, returning a dictionary under a titles key. The configuration is the same shape as the JSON example. Treat the returned list as model output that should be checked against the live page before you rely on it.
Proxy configuration
Repeated requests from one IP can be blocked, so proxies spread requests across addresses and can also reach geo-restricted content. ScrapeGraphAI supports automatic proxy rotation through the free-proxy library, with criteria for anonymity, security, country set, timeout and rotation frequency. That suits casual use with simple HTTP proxies. For heavier or more sensitive targets, a premium proxy service is the more reliable route. Proxy settings live under loader_kwargs.proxy in graph_config, either as a broker with criteria or as an explicit server with username and password.
Scrape search results with SearchGraph
SearchGraph runs a search from a prompt and returns structured results. The example asks for the best pizzerias in Riga and returns names, ratings and addresses. The configuration matches the earlier graphs. Expect a run to take a few minutes depending on hardware.
Python example
{
"best_pizzerias": [
{
"name": "Pizzaiolo",
"rating": 4.5,
"address": "Krišjāņa Valdemāra iela 13, Rīga",
},
{"name": "Pizza Marvella", "rating": 4.2, "address": "Skārņu iela 55, Rīga"},
{"name": "La Biga", "rating": 4.1, "address": "Kronvalda iela 10, Rīga"},
{"name": "Pizzeria Riga", "rating": 4.5, "address": "Riga, Latvia"},
]
}Generate an audio summary with OpenAI models
SpeechGraph combines a text model and a text-to-speech model to summarise a page and save the result as audio. You need an OpenAI API key and may need to buy credits under Billing. The example uses gpt-3.5-turbo for text, tts-1-hd for speech, the fable voice and an output file named audio_summary.mp3. Voice samples are available in the OpenAI documentation. The run returns both a text summary and the audio file, which can make content easier to consume for people who prefer listening or have visual impairments.
Setup checklist and field map
Use this checklist before running any graph:
- Ollama is installed and a model has been pulled.
- The model responds in the interactive prompt.
- ollama serve is running and port 11434 is reachable.
- scrapegraphai is installed in the active environment.
- playwright install has downloaded the browser components.
- The graph_config points at http://localhost:11434 and keeps ollama/nomic-embed-text as the embedding model.
The shared graph_config fields are llm.model, llm.temperature, llm.format and llm.base_url, plus embeddings.model, embeddings.temperature and embeddings.base_url. SpeechGraph replaces those with llm.api_key, llm.model, tts_model.api_key, tts_model.model, tts_model.voice and output_path. Proxy settings sit under loader_kwargs.proxy. If a run fails immediately, check whether the Ollama port is already in use before changing the model; if browser automation fails, check the Playwright install step.
Frequently asked questions
Can I change the embedding model in graph_config?
The walkthrough says to keep ollama/nomic-embed-text even when you switch the main language model. Change llm.model instead.
What should I check if ollama serve reports that the port is busy?
Ollama may already be running after installation. Check whether the local instance is up on 127.0.0.1:11434 before starting another server.
What does JSONScraperGraph return?
It returns a Python dictionary containing the fields requested in the prompt. The sample output groups product names, average ratings and prices under a products key.
Do I need paid OpenAI credits for the audio summary?
You need an OpenAI API key, and the walkthrough notes that you may need to purchase credits under Billing. It gives an approximate cost of around 5 cents per request, which you should verify against current OpenAI pricing.
[
{
"id": 1,
"name": "Product A",
"price": 25.99,
"availability": true,
"description": "Example post details.",
"categories": ["electronics", "gadgets"],
"manufacturer": {
"name": "GadgetPro",
"location": "USA"
},
"reviews": [
{
"user": "JohnDoe",
"rating": 4,
"comment": "Great value for the price!"
},
{
"user": "JaneSmith",
"rating": 5,
"comment": "Absolutely love this product!"
}
]
},
{
"id": 2,
"name": "Product B",
"price": 45.50,
"availability": false,
"description": "Example post details.",
"categories": ["home", "appliance"],
"manufacturer": {
"name": "HomeGoodsInc",
"location": "Canada"
},
"reviews": [
{
"user": "AliceBrown",
"rating": 3,
"comment": "Useful, but a bit overpriced."
},
{
"user": "DaveWilson",
"rating": 5,
"comment": "A must-have in every household!"
}
]
},
{
"id": 3,
"name": "Product C",
"price": 15.75,
"availability": true,
"description": "Example post details.",
"categories": ["sustainability", "eco-friendly"],
"manufacturer": {
"name": "EcoProducts",
"location": "Germany"
},
"reviews": [
{
"user": "ClaraKlein",
"rating": 4,
"comment": "Good for the environment and your wallet."
},
{
"user": "OmarFarouk",
"rating": 5,
"comment": "Excellent product with great eco credentials!"
}
]
},
{
"id": 4,
"name": "Product D",
"price": 29.99,
"availability": true,
"description": "Example post details.",
"categories": ["outdoors", "durable"],
"manufacturer": {
"name": "OutdoorGear",
"location": "Sweden"
},
"reviews": [
{
"user": "BethJones",
"rating": 4,
"comment": "Sturdy and reliable."
},
{
"user": "TomNorton",
"rating": 5,
"comment": "Perfect for any outdoor enthusiast!"
}
]
},
{
"id": 5,
"name": "Product E",
"price": 55.00,
"availability": true,
"description": "Example post details.",
"categories": ["beauty", "luxury"],
"manufacturer": {
"name": "LuxBeautyCo",
"location": "France"
},
"reviews": [
{
"user": "SandraLee",
"rating": 5,
"comment": "Feels luxurious and well worth the price."
},
{
"user": "RajPatel",
"rating": 4,
"comment": "Great quality, but a little pricey."
}
]
}
]