Skip to content

Web Scraping in Go: A Practical Guide with Standard Library and Colly

Go's standard library can fetch a page and walk its HTML tree without any third-party dependency. When you need callbacks, selector helpers, and a structured crawl loop, Colly adds those conveniences on top of Go's HTTP and HTML tooling. This guide walks through both paths using a Wikipedia link scraper and a table-to-CSV example.

Project setup and prerequisites

You need a working Go installation, an editor or IDE, and basic familiarity with Go syntax. The examples were written against Go 1.23.2; check the current Go release before starting a new project.

Create a project directory and initialize a module. The go mod init command creates a go.mod file that records direct and indirect dependencies. When you later add Colly, a go.sum file records checksums and versions for those dependencies.

Before writing any scraper, decide whether the page is static HTML or rendered by JavaScript. The standard-library and Colly examples here target static HTML. If the content only appears after client-side rendering, you need a browser-based approach instead.

Building a scraper with the Go standard library

The standard-library path is useful when you want minimal dependencies and full control over the request and parse steps. It is a good fit for small, well-defined pages.

Start with package main so the compiler treats the file as an executable program. Then write a main function. The supplied starter code prints a greeting; replace that with the fetch and parse logic once the module is initialized.

Fetching the page uses http.Get. Always close the response body with defer resp.Body.Close() and read it with io.ReadAll. At this stage you have the raw HTML as a string, which is enough to confirm the request succeeded.

Parsing uses html.Parse from golang.org/x/net/html. It returns the root node of a parsed tree. To collect links, walk the tree recursively: check for element nodes whose data is a, read the href attribute, and append it to a slice. The complete example combines the fetch, parse, and recursive link collection in one file.

A useful distinction: an HTTP error means the request failed; a parse error means the response body was not valid HTML; an empty link slice means the page loaded and parsed but no matching anchors were found. Check these separately rather than treating every empty result as a network failure.

Installing and understanding Colly

Colly is a scraping and crawling library built on Go's net/http and the goquery package. goquery provides a jQuery-like selector syntax for targeting HTML elements; Colly adds the request lifecycle and callback structure around it.

The central component is the Collector. It manages network communication and the callbacks attached to a scraping job. You can configure it with options such as allowed domains, a custom User-Agent, or authentication headers.

Callbacks run at different points in the Collector lifecycle. OnRequest runs just before an HTTP request is made. OnHTML is the most common callback for scrapers: it registers a function that runs when the Collector reaches a matching HTML element. The callback receives the matched element and can read child attributes or text from it.

Install Colly with the module command shown in the supplied installation code. After installation, the dependency appears in go.mod and the checksum file is generated.

A Colly scraper for Wikipedia links

The Colly version of the link scraper initializes a Collector with en.wikipedia.org as an allowed domain. The allowed-domain option restricts which hosts the Collector will visit, which is a useful guardrail when you are learning or when a page contains external links.

Use the browser's inspect tools to find the container that holds the article body. On the Wikipedia page used in the example, the relevant container is the mw-parser-output div. The OnHTML callback targets that selector and calls ChildAttrs("a", "href") to collect all anchor href values inside it. The result is printed to the terminal.

Finally, call Visit with the target URL. The complete program is supplied as a code module. Running it with go run main.go prints the collected links.

A practical check: if the callback never fires, confirm the selector matches the current page markup. If it fires but returns an empty slice, the container exists but the expected child anchors are absent or use a different attribute.

Scraping table rows into a CSV file

The table example reuses the Colly pattern but writes rows to a CSV file instead of printing links. Go's encoding/csv package provides the writer.

The setup code creates data.csv with os.Create, defers file.Close, initializes a CSV writer, and defers writer.Flush. If file creation fails, the program logs the error and exits.

The scraping callback targets table#customers. Inside it, ForEach iterates over tr elements. For each row, the code reads the first, second, and third td cells using nth-child selectors and writes them as a CSV record. Because the callback does not target header cells, the output does not include column headings.

A field map for the example:

  • Target table: table#customers
  • Row selector: tr
  • Column 1: td:nth-child(1)
  • Column 2: td:nth-child(2)
  • Column 3: td:nth-child(3)
  • Output file: data.csv
  • Writer: encoding/csv

If the CSV file is created but contains no rows, check whether the table selector matches the live page and whether the rows use td cells rather than header or other cell types.

Choosing between the standard library and Colly

Use the standard library when the page is simple, you want no third-party dependency, and you are comfortable writing your own traversal. Use Colly when you want selector-based callbacks, allowed-domain controls, and a structure that scales to multiple pages or repeated jobs.

The standard library gives you http.Get and html.Parse; Colly gives you a Collector, lifecycle callbacks, and helpers such as ChildAttrs and ChildText. Neither approach renders JavaScript. For pages that depend on client-side rendering, a browser-based tool is the appropriate next step.

What you can build next

You now have two working patterns: a scraper written with Go's core libraries and a scraper built around Colly. Both can be extended, and Colly can also be paired with goquery to combine its crawl lifecycle with goquery's selector syntax for a more capable scraper.

Scrapers like these support keyword research, brand protection, promotion, and website testing, among other tasks. Treat the examples as starting points and adapt the selectors, allowed domains, and output format to the site you are working with.

Frequently asked questions

Can these Go examples scrape JavaScript-rendered pages?

No. The standard-library and Colly examples fetch and parse static HTML. If the content is inserted by client-side JavaScript, you need a browser-based approach that executes scripts before extraction.

What is the difference between Colly and goquery?

goquery provides jQuery-like selectors for HTML in Go. Colly builds on Go's HTTP client and goquery to provide a Collector, request lifecycle callbacks, and scraping helpers. You can use goquery directly for parsing, or use Colly when you want the surrounding crawl and callback structure.

Why does my scraper return no links or rows?

Check three things separately: whether the HTTP request succeeded, whether the response parsed as HTML, and whether the selector matched the current page markup. A successful request with an empty result usually means the selector or the expected child elements have changed.

Does the CSV example include table headers?

No. The callback reads only td cells in each row, so header cells are not written. If you need headers, add a separate write before the row loop.