Make concurrent requests in Go
Run several scraping calls at once in Go, coordinate their completion, and save each returned page as an HTML file.
Why parallel calls help
Sequential requests leave the network idle while each page loads. Running calls in parallel can shorten a batch job when your account permits multiple simultaneous operations. Your effective ceiling is set by your concurrent-request allowance: a higher allowance means more calls can stay active together, while exceeding it can prevent the extra parallelism from helping.
How the example is structured
The pattern in protected module code0001 has three parts. First, a worker function builds a GET request, adds the API key and target URL as query parameters, checks for a successful response, reads the body, and writes it to a named HTML file. Second, main creates a sync.WaitGroup and registers two workers. Third, each worker is launched with the go keyword, and the program waits on the WaitGroup before exiting. In the captured example, the two targets are the blog and documentation pages, saved as blog.html and documentation.html.
Adaptation checklist
Use this checklist before adapting the example: confirm your API key and API base URL; confirm how many simultaneous calls your plan allows; set the WaitGroup count to match the number of goroutines you launch; give every target a distinct output file; check the HTTP status before writing the response body; and ensure the worker signals completion only after its file work is done. A mismatch between the WaitGroup count and the number of launched goroutines is the most common way this pattern hangs or exits early.
Goroutines still need a finish line
Concurrency here does not remove the need for coordination. Launching goroutines starts work in the background, but the main routine must still wait for all of them to finish. That waiting step is what the WaitGroup provides. Without it, the program could end before either HTML file is written.