Skip to content

Scrape Data in Go Using Colly: Hacker News Comments Walkthrough

Build a Go scraper that collects a Hacker News submission's title, link, site, score, identifier and every comment, then prints the result as structured JSON. The walkthrough uses Colly's collector and callbacks, declarative struct tags, and pagination handling.

What the finished program produces

The finished program reads a submission identifier, visits that Hacker News item, and writes one JSON object to the terminal. Top-level fields cover the submission itself, while a nested list holds each comment's author, permalink and text. The example output in module code0001 shows the shape to expect: title, URL, site, identifier, score, total comment count and the comments array.

Set up the project and verify it runs

Checklist before writing the scraper:

  • Confirm a Go 1.20.5 installation, or at least 1.19.1 or higher, using module code0002.
  • Create a project folder and initialise a module with module code0003.
  • Add Colly as a dependency with module code0004.
  • Create the main source file with module code0005; the folder should then match module code0006.
  • Run the smoke test in module code0007 with module code0008. Seeing the greeting confirms the collector can visit a page.

How Colly's collector and callbacks fit together

Colly's Collector is the object that issues requests. A new instance is created in module code0007. Visit handles the common case of opening a URL; Request is the alternative when extra request information is needed. Around those calls sit lifecycle callbacks: OnRequest before sending, OnError on failure, OnResponse after a reply arrives, OnHTML for HTML replies, OnXML for HTML or XML replies, and OnScraped at the very end. Any number of callbacks of each kind can be registered. For most extraction work, OnHTML matters most: give it a CSS selector and Colly runs your function for each matching element. Selector matching follows goquery, which is modelled on jQuery, so jQuery-compatible selectors apply.

Choose the fields, then locate them in the page

Decide the output shape first, then inspect the page. For the submission, capture identifier, title, URL, site, score and comments. For each comment, capture author, permalink and text. Start with plain structs as in module code0009, then refine them as selectors are discovered. In browser developer tools, the submission title sits in an anchor inside a titleline span, the site name in a sitestr span, and the score in a score span. Comments live under a comment-tree table, one row per comment, with the author in an hnuser anchor, the permalink in an age anchor, and the text in a comment div.

Extract the submission fields with Unmarshal

First prove the page loads: module code0010 visits the submission, and module code0011 registers an OnHTML callback on the html element, verified by module code0012. Then replace the probe with real extraction. Colly's Unmarshal feature maps struct tags to the page: a selector part picks the element, and an optional attr part reads an attribute instead of text. Module code0013 adds title, URL and site; module code0014 wires Unmarshal into the callback; module code0015 adds the score. The identifier is not in the markup, so module code0017 reads it from the request URL's query string.

Loop over comments and clean each value

Comments repeat, so iterate rather than unmarshal once. Module code0018 updates the comment struct, loops over each comment row with ForEach, unmarshals one comment at a time, and appends it to the submission. Permalinks arrive as relative references, so module code0019 converts each one to an absolute URL. Comment text needs a small cleanup: the raw extraction in module code0021 carries a trailing reply label from nested markup, and module code0022 trims that suffix and surrounding whitespace.

Follow pagination and run the complete scraper

Hacker News shows only some comments per page, with a morelink anchor leading to the next batch. Module code0023 registers a second OnHTML callback for that anchor and visits its target, so pagination is followed automatically. Module code0024 assembles everything: it accepts the submission identifier from the command line as shown in module code0025, unmarshals submission data only on the first page, accumulates comments and a running total across pages, and encodes the result as indented JSON. Run it with module code0026, supplying the identifier after the -id flag.

Limits to keep in mind

Colly's callback model handles straightforward pages and can also reach more complex sites and single-page applications, but it lacks a full JavaScript engine, so heavily scripted pages may require creative workarounds or a browser-based approach. Hacker News is an easy target because it applies no rate limiting or aggressive antibot checks to basic scrapers; larger projects should plan for blocks, proxies and fingerprinting. The official Colly examples for basic sites, Instagram and Google Groups are good next references for other shapes of page.

Field-to-selector map for the Hacker News example
FieldWhere it lives on the pageHow to capture it
Submission titleAnchor inside a titleline spanSelector for that anchor; read its text
Submission URLhref attribute of the same anchorSame selector plus attr for href
Sitesitestr spanSelector for that span; read its text
Scorescore spanSelector for that span; read its text
Submission IDid query parameter in the page URLParse the request URL query
Comment authorhnuser anchor inside each comment rowLoop rows, then selector inside each row
Comment permalinkhref of the age anchor inside each rowSelector plus attr, then convert to absolute URL
Comment textcomment div inside each rowSelector for that div, then trim the trailing reply label

Selectors are specific to the Hacker News submission page markup used in the walkthrough.

Frequently asked questions

Can Colly scrape JavaScript-heavy or complex sites?

Colly has no full JavaScript execution engine, so JavaScript-heavy pages may need a different approach or a browser-based tool. Hacker News worked simply here because it does not impose rate limiting or strong antibot measures on basic scrapers.

When should I use Visit instead of Request?

Use Visit for ordinary page loads. Use Request when the call must carry extra information beyond a simple visit.

How do I find the right selectors for a page?

Inspect the page in browser developer tools, identify the element holding each value, then map it with a CSS selector. For an attribute such as a link target, add an attr part; otherwise the matched element's text is extracted.

{
  "URL": "https://reddark.untone.uk/",
  "Title": "Reddit Strike Has Started",
  "Site": "reddark.untone.uk",
  "Id": "36283249",
  "Score": "1397 points",
  "TotalComments": 894,
  "Comments": [
	{
  	"Author": "wpietri",
  	"URL": "https://news.ycombinator.com/item?id=36283389",
  	"Comment": "Great! There's a picket line I won't be crossing. Not just for those subs, of course, but for all of Reddit.The reason Reddit is valuable is not the few execs making these (IMHO terrible) decisions. It's the thousands of mods and the millions of people creating and organizing the content that I go there to read. Until those people are happy with things, I'm not going back."
	},
	{
  	"Author": "jcims",
  	"URL": "https://news.ycombinator.com/item?id=36283543",
  	"Comment": "Reddit is basically successful despite itself.  It's wild."
	},
	{
  	"Author": "wpietri",
  	"URL": "https://news.ycombinator.com/item?id=36283661",
  	"Comment": "Twitter too! When I worked there they jokingly referred to the head of product as the defense against the dark arts teacher, in that every year or so they'd disappear mysteriously and there'd be a new one. Most of Twitter's successes came from watching what users were actually doing and supporting that (e.g., at mentions, retweets, quote tweets). Many of their failures have been trying to graft on something irrelevant or actively contradictory to user needs. Or just flat out ignoring things users liked, as with them closing down Vine and letting TikTok come in to win as the short, fun video platform.But network effects businesses are really hard to kill. Sure, Musk has set $20-30 billion on fire and Twitter is rapidly decaying. But imagine taking a resilient business like a McDonald's franchise and subjecting it to Musk levels of chaos. It would have been out of business long before, instead of merely shrinking significantly."
	},
	// ... additional comments ...
  ]
}