Skip to content

GoSpider Crawling Guide: Install, Flags, Filtering, and 403 Fixes

GoSpider is a Go-based command-line crawler built for speed. It maps a site by following links, parsing JavaScript files for URL-like strings, and optionally pulling in subdomains, sitemaps, and third-party sources. It is a discovery tool, not a data extractor: it will not render a page in a browser or return structured product fields. This guide walks through installation, the flags that matter most, a worked crawl against a Product Hunt leaderboard page, a URL filter, and the two failure modes that send most people looking for a proxy or scraping API.

What GoSpider does and does not do

GoSpider is an MIT-licensed CLI spider written in Go. It was originally built for security reconnaissance as part of the Osmedeus engine, which explains its emphasis on breadth: it can find ordinary links, URLs embedded in JavaScript files, subdomains, AWS S3 bucket references, sitemap entries, and archived URLs from third-party sources. It crawls targets in parallel, so a single run can cover many pages quickly.

The boundary to keep in mind is extraction. GoSpider returns URLs and assets, not product names, prices, or article text. If you need structured data, the usual pattern is to let GoSpider discover the URLs and then pass that list to a scraper such as Colly. A second boundary is rendering: GoSpider does not execute JavaScript or load pages like a browser, so content that appears only after client-side rendering can be missed even though the crawler follows links inside JavaScript files.

Install GoSpider and verify the CLI

GoSpider needs a working Go environment. Check the toolchain first, then install the binary and confirm the help menu appears. The commands below are the exact ones from the captured workflow; the version string you see will depend on your machine.

Core GoSpider flags and what they control

The general syntax is gospider [flags]. You use flags to choose the target, set crawl depth, control concurrency, decide where output goes, and select which discovery sources to enable. The table below lists the flags most readers will touch first, with their defaults. A shallow crawl is the safest starting point: -d 1 limits recursion to one level, while -d 0 enables infinite recursion and can expand quickly across a large site. Adding -k or -K introduces delays to reduce server load, and robots.txt is respected by default.

Reading GoSpider output tags

GoSpider's terminal output is tagged, and the tags tell you why a URL appeared. [url] means the crawler actually requested that page or asset. [href] means the link was found in the HTML. [javascript] marks a JavaScript file URL, and [linkfinder] marks a URL-like string found inside JavaScript. There is also a default blacklist for common static assets such as jpg, png, and css, though those links can still appear when they are discovered in page HTML.

That last point matters for filtering. A URL appearing in the output does not always mean GoSpider followed it. If you need a clean list, plan to filter the raw output rather than relying only on crawl-time flags.

Worked example: crawl a Product Hunt leaderboard page

The captured example targets a Product Hunt daily leaderboard page. The goal at this stage is not to extract product data but to find the product URLs on the page. The crawl command sets depth to one, keeps concurrency low, sets a 20-second timeout, and focuses on base HTML content. After the run, the output folder contains a file named for the target host, and that file holds the URLs and assets GoSpider discovered.

A typical output line shows a successful request followed by product links and a JavaScript chunk. The [url] line confirms the page was fetched; the [href] lines are the product pages you can carry forward. Raw output can still include navigation links, static assets, and other references you do not need, so the next step is to turn it into a clean list.

Filtering product URLs before scraping

GoSpider has --whitelist and --blacklist flags that control which URLs it is allowed to request during a crawl. In the captured test with GoSpider v1.1.6, those flags did not always stop matching links from appearing in the output as discovered [href] entries. A blacklisted URL found in page HTML could still show up as an [href] even though GoSpider did not follow it as a [url] request.

Because of that, the example uses a small custom filter program that reads the raw output, matches Product Hunt product-page URLs, strips query strings and fragments, checks the /products/slug path shape, and removes duplicates. The result is a clean product_urls.txt file ready for a scraper. This is a practical pattern for any site with a predictable URL pattern: filter after the crawl rather than assuming crawl-time flags produce a perfectly clean list.

Turning crawled URLs into data with Colly

Colly is a Go scraping framework that visits pages, handles responses, and extracts HTML. The example scraper reads the filtered URL list, checks for an API key in the environment, requests each page through Colly, parses the returned HTML, and writes a CSV. The product struct captures name, tagline, website, score, comments, and URL.

The reason the example routes page fetches through an API is that the target is protected by Cloudflare and uses dynamic rendering. A plain request is unlikely to return the product data reliably. The API request sets JavaScript rendering and premium proxies, which the captured example notes costs 25 API credits per request; premium proxies without JavaScript rendering cost 10 credits, and Stealth Mode costs 75 credits for harder targets. Those figures are a dated reference from the captured material and should be checked against current documentation before budgeting.

Extending discovery: subdomains, sitemaps, and third-party sources

The examples above focus on links found from one page, but GoSpider can look further. --sitemap checks sitemap URLs, and --robots crawls paths listed in robots.txt. --subs includes subdomains found from the target page, while -w expands that to subdomains found from third-party sources. The -a or --other-source flag pulls URLs from sources such as the Wayback Machine, Common Crawl, VirusTotal, and AlienVault, which can surface older paths and endpoints no longer linked from the current site. Custom headers and cookies can be supplied with -H and --cookie.

GoSpider vs. hakrawler

Both are fast Go-based crawlers, but they lean in different directions. hakrawler is simpler and fits neatly into shell pipelines: you pipe URLs in, take the output, and move on. GoSpider offers broader discovery, including third-party sources and more parsing options in a single run. In practice, many people run both because their results overlap only partially and one often finds URLs the other misses. The comparison table below summarizes the differences captured in the source material.

Why GoSpider gets a 403 and how to respond

A 403 usually means the target refused the request. Anti-bot systems check user agent, cookies, browser fingerprint, and IP reputation, so simply rerunning the same crawl or changing the target URL often will not help. In the captured example, GoSpider's default -u web setting sends a random browser-like user agent and got through; forcing a bot-like user agent with -u 'curl/8.7.1' produced a 403 response.

The first fix is to route GoSpider through a proxy using the -p flag. The captured example uses a proxy endpoint that keeps the GoSpider command almost unchanged and returns a successful response. Harder Cloudflare-protected targets may need more than a proxy, which is where a scraping API with rendering and stealth options comes in. The key distinction is between a block (the server refuses the request) and a rendering gap (the server returns a page but the content is built client-side and GoSpider cannot see it).

Decision checklist: GoSpider alone or with an API

Use GoSpider alone when the job is URL discovery on a static public site, when you need a fast map of links, or when you are preparing a URL list for another scraper. Add a rendering or proxy API when the target is a JavaScript-rendered single-page app, when Cloudflare or a similar system returns blocks, when you need structured data at scale, or when the page content only appears after client-side rendering. The table below lays out the scenarios captured in the source material.

GoSpider vs. hakrawler
DimensionGoSpiderhakrawler
Written inGoGo
Third-party sourcesYesnot the main focus
Subdomain discoveryYesscope-driven
JavaScript link findingYesmore minimal
Output formatTagged text, quiet URLs, JSONpipe-friendly URLs
Ease of pipingGood with -q, noisier by defaultvery good
Known gotchaLarge crawls need manual limitsfewer built-in discovery knobs

Comparison reflects the captured source material and is not an independent benchmark.

When to add a scraping API
ScenarioGoSpider aloneAdd a scraping API
Static public siteGood fit for URL discoveryUsually not needed
JS-rendered SPABad fitNeeded to render JavaScript
Cloudflare or DataDome-protected pageOften blockedUse Premium or Stealth proxies
Need structured dataNot enough by itselfGreat at scale
DiscoveryGood for discoveryAdd API for blocked or rendered pages

Scenario guidance is from the captured source material and should be validated against current target behavior.

Frequently asked questions

What is GoSpider used for?

GoSpider is used for fast URL and asset discovery. It crawls targets and finds links that can be passed to another scraper or used for testing.

Is GoSpider free?

Yes. GoSpider is free and open source under the MIT license. You can install it locally and run it from your terminal.

What is the difference between GoSpider and hakrawler?

Both are fast Go-based crawlers. hakrawler is simpler and more pipe-friendly, while GoSpider offers broader discovery features such as third-party sources and more parsing options.

Does GoSpider render JavaScript?

No. GoSpider neither runs JavaScript nor loads a page as a browser would. The --js flag merely scans JavaScript files for link-like strings, so content that only appears after client-side rendering can still be missed.

Why does GoSpider return a 403 error?

A 403 usually means the target refused the request. Anti-bot checks, poor IP reputation, or other factors can cause it. The usual fixes are to route requests through a proxy or fetch the page through a scraping API.

gospider -s https://www.producthunt.com/leaderboard/daily/2026/6/22 \
  -d 1 -c 1 -t 1 -m 20 -B -u 'curl/8.7.1' --debug
gospider -s https://www.producthunt.com/leaderboard/daily/2026/6/22 \
  -d 1 -c 1 -t 1 -m 30 -B -u 'curl/8.7.1' \
  -p "http://$SCRAPINGBEE_API_KEY:render_js=False&[email protected]:8886"
Core GoSpider flags
FlagWhat it doesDefault
-s, --siteCrawl one site–
-S, --sitesRead target sites from a file–
-o, --outputSave output to a folder–
-q, --quietSuppress extra output and show URLs onlyfalse
-d, --depthSet maximum recursion depth1
-c, --concurrentSet concurrent requests per matching domain5
-t, --threadsCrawl multiple sites in parallel1
-k, --delayAdd delay between requests in seconds0
--robotsCrawl robots.txttrue
--jsEnable link finding in JavaScript filestrue
--subsInclude subdomainsfalse
-a, --other-sourcePull URLs from third-party sourcesfalse
--jsonEmit JSON outputfalse
-p, --proxySend requests through a proxy–

Defaults and flag names are from the captured GoSpider help output; behavior may vary by version.