Getting Started with Apache Nutch
Apache Nutch is an open-source web crawler for collecting pages, storing what it finds, and building a searchable index. This guide walks through installing Nutch, pairing it with Apache Solr, running a crawl in segments, indexing the results, and querying them — with a clear checklist you can reuse for your own crawl.
Why Apache Nutch?
Crawling and scraping solve different problems. Scraping pulls chosen fields from pages you already know about. Crawling discovers pages: it starts from known addresses, follows links found in their content, queues new URLs, and repeats until the queue empties. Along the way it can save page content, meta tags and links for later use.
Nutch fits the discovery side. It collects pages, stores them, and builds an index you can search. It also supports incremental crawling — revisiting earlier pages to catch changes — and its plugin design lets you add parsers, URL filters, metadata extractors and storage backends. Hadoop support adds distributed processing for larger jobs, while segment-based storage keeps each crawl round manageable. For small runs the native file system is enough; for larger ones, Solr or Elasticsearch can take over storage and search.
Install Apache Nutch
Before you begin, check these prerequisites:
- A Unix environment, or WSL 2 on Windows
- Java Development or Runtime Environment version 11
- A code editor such as Visual Studio Code
Create a project directory, move into it, then download and unpack Nutch 1.19 using protected module code0001. Enter the unpacked folder and run the nutch command via protected module code0002 to confirm the installation lists its options. If you see a JAVA_HOME is not set error, set it for the current session with protected module code0003, adjusting the path to your Java 11 installation.
Configure Nutch with Solr for storage and indexing
This walkthrough uses Solr as the index and analysis layer. Download and unpack Solr 8.11.2 with protected module code0004, start the server from its bin folder with protected module code0005, and open the local Solr interface. Create a core named nutch with protected module code0006; the expected confirmation is shown in protected module code0007.
Next, open nutch-site.xml inside the Nutch conf folder and add three properties inside the configuration tags: the Solr store class (code0008), the Solr server URL pointing at the nutch core (code0009), and a specific crawler name (code0010). Naming the agent matters because robots.txt directives are matched against it — a default name can cause accidental violations. The combined file is shown in protected module code0011.
Set up seed URLs and URL filtering
Every crawl needs starting points, called seed URLs. Create a urls directory and a seed.txt file containing your target site — use {{OWNER_SITE_BASE_URL}} for the site you intend to crawl — via protected module code0012.
Then scope the crawl with the URL filter. In conf/regex-urlfilter.txt, the default accept-anything rule (code0013) would let the crawler wander beyond your target domain. Replace that rule with one that only accepts URLs under {{OWNER_SITE_BASE_URL}} so filtering stays tight from the first segment onward.
Crawl in segments: inject, generate, fetch, parse, update
The crawl runs as a repeatable pipeline. Inject the seed URLs into the crawl database with protected module code0015. Generate a segment by scanning that database and scheduling the next URLs with protected module code0016, then capture the newest segment name in a shell variable with protected module code0017. Fetch the pages in that segment (code0018) and parse them to extract useful data (code0019).
Update the central database with the segment results using protected module code0020. For a second, focused round, generate a segment limited to the top twenty pages, fetch and parse it (code0021), and update the database again (code0022). Keeping each round in its own segment makes failures easy to isolate: if fetching fails, parsing and indexing are unaffected.
Invert links and index into Solr
Before indexing, invert the crawled links into a link database with protected module code0023. This turns outlinks — links from one page to another site — into inlinks within the same site, adding relevance context. Then index into Solr with protected module code0024, which applies three safeguards: -filter applies your URL criteria, -normalize standardises URL format, and -deleteGone removes broken URLs from the index. Once you understand these steps, the crawl script can automate the whole sequence.
Analyze the indexed data in Solr
Open the nutch core overview in the Solr interface and confirm the document count matches your indexing output. Then use the Query panel as a practical checklist:
- Search a title field, for example title:amazon, to find pages by title term.
- Add a filter query such as title:alternative to narrow a content search, and watch the result count drop.
- Set the field list to title, url, content to control what each result returns.
- Search url:*.svg and sort by boost descending to surface the most relevant image URLs first.
The same index can be queried over HTTP from any language for deeper analysis. In short: configure once, crawl in segments, index with filters, and let Solr do the searching.
| Stage | What it does | Key output |
|---|---|---|
| Inject | Load seed URLs into crawldb | Crawl database ready |
| Generate | Select next URLs and create a segment | New segment under crawl/segments |
| Fetch / Parse | Download pages, then extract data | Parsed segment content |
| Update DB | Merge segment results into crawldb | Refreshed crawl database |
| Invert links | Build linkdb from outlinks to inlinks | Link relevance context |
| Index | Write filtered, normalised docs to Solr | Searchable nutch core |
Workflow summary based on the crawl sequence in the guide; exact commands are in the protected modules listed in the text.
Frequently asked questions
How is crawling different from scraping?
A crawler discovers and catalogues relevant sources by following links from known addresses, while scraping extracts specific data from pages. Nutch is presented as a crawler that collects, stores and indexes pages for searching.
What do you need before installing Apache Nutch?
Use a Unix environment (WSL 2 on Windows), Java 11, and an editor such as Visual Studio Code. If JAVA_HOME is not set, set it for the terminal session using protected module code0003.
How do you connect Nutch to Solr?
Start Solr, create a core named nutch, then in nutch-site.xml set the Solr store class, the Solr server URL, and a specific http.agent.name so robots.txt rules are honoured. The exact properties are in protected modules code0008 to code0011.
What is the crawl-to-index sequence?
Inject seed URLs into crawldb, generate a segment, fetch it, parse it, then update the database. Repeat for a second segment, invert links into linkdb, and index with filtering, normalisation and deletion of gone URLs. See protected modules code0015 to code0024.
How do you query the indexed data in Solr?
Use q for the main query, fq to narrow results, fl to choose returned fields, and sort to order results. Examples include searching a title field, filtering by another title term, returning title, url and content, and sorting SVG URL matches by boost descending.