chromedp in Go: Control Headless Chrome for Scraping
chromedp lets a Go program talk to a real Chrome or Chromium instance. That makes it a practical choice when a page needs JavaScript to render, when you must click or type before data appears, or when you want to inspect the DOM after a browser has done its work. This guide walks through the core actions in the order you will usually use them, then shows how to handle buttons, forms, and new tabs.
What you need before the first run
You need a working Go environment, the chromedp package, and a Chrome or Chromium browser installed on the machine that will run the program. Import the package alongside the standard context and fmt packages. The context package manages the lifecycle of browser operations; fmt prints results to the console; chromedp supplies the browser-driving actions.
A minimal program creates a context with chromedp.NewContext(context.Background()) and defers cancel() so the browser resources are released when the work finishes. Skipping the cancel call is a common way to leak processes during repeated runs.
Extracting text from a static page
For a page whose content is already in the HTML, the sequence is short: navigate to the URL, wait until the body is visible, then read the text content into a Go string. The captured example uses https://toscrape.com and prints the result.
A useful verification step is to compare the printed string with what the browser shows. In the captured run, the output begins with the site title and continues through the sandbox descriptions for Books and Quotes. If the string is empty, check that the selector matches an element that actually exists after load, and confirm that the navigation step returned no error.
The complete program is preserved in the accompanying code module so you can compare your version line by line.
Working with pages that need interaction
JavaScript-heavy pages often hide the data you want behind a click or a form submission. chromedp exposes actions for both. The pattern is to navigate, wait for a stable element, perform the interaction, wait again for the new state, and only then extract.
A decision checklist helps avoid the most common mistakes:
- Is the target present before you act? Wait for a selector you can see in the rendered DOM, not just in the raw HTML.
- Does the action change the page? Add a second wait after the click or submit.
- Does the action open a new tab? Capture the new target before you try to read from it.
- Are you extracting text or markup? TextContent returns visible text; OuterHTML and InnerHTML return markup around a selector.
- Are you logging from goroutines? log is safer than fmt for concurrent output and adds timestamps.
Clicking a button and reading the result
The button example navigates to https://go.dev, waits for the body, clicks the primary hero action with chromedp.Click and chromedp.ByQuery, waits again, and reads the OuterHTML of the .Learn-hero section. The captured output shows the Learn page markup, including the install heading and download link.
Use this pattern when a link or button changes the page without a full reload. The second WaitVisible is not decorative: it gives the new document time to settle before the extraction action runs.
Filling and submitting a form
The form example targets the W3Schools HTML forms page. It sets the first name field to Carl and the last name field to Jones using chromedp.SetValue, then submits the first form with chromedp.Submit. Both actions take a CSS selector and use chromedp.ByQuery to say that the selector is a CSS query.
Because the submission opens a new tab, the program calls chromedp.WaitNewTarget before running the fill task, then creates a new context with chromedp.WithTargetID(<-ch) to read the response. The captured output confirms that the server received fname=Carl and lname=Jones.
A field/control map for this example:
- SetValue selector: the field to fill, for example the first-name input inside the form.
- SetValue value: the string to type into that field.
- ByQuery option: tells the action to interpret the selector as a CSS query.
- Submit selector: the form element that should fire the submit event.
- WaitNewTarget: listens for the newly opened tab so the response can be read.
- InnerHTML: extracts the submitted page content for logging.
When chromedp is the right tool
chromedp runs a full browser, so it uses more memory and CPU than an HTTP client that only fetches HTML. It can also be slower because every page must be rendered and its JavaScript executed. Browser and protocol changes can require updates to your code. Those costs are worth paying when the data only exists after JavaScript runs or after a user-like interaction.
If your target pages are static, a simpler HTTP-based approach will usually be faster and easier to maintain. If you need browser behavior but do not want to manage proxies, user agents, and fingerprints yourself, a hosted scraping API is an alternative; any trial or pricing terms for such a service must come from its owner.
| Goal | Action | Typical arguments | Notes |
|---|---|---|---|
| Start a browser session | chromedp.NewContext | context.Background() | Defer cancel() to release resources. |
| Open a URL | chromedp.Navigate | url string | Returns an error if navigation fails. |
| Wait for an element | chromedp.WaitVisible | selector, chromedp.ByQuery | Use before acting on a selector. |
| Read visible text | chromedp.TextContent | selector, &string | Returns text inside the element. |
| Read markup | chromedp.OuterHTML / InnerHTML | selector, &string, chromedp.ByQuery | OuterHTML includes the element; InnerHTML is its contents. |
| Click an element | chromedp.Click | selector, chromedp.ByQuery | Triggers the element's click handler. |
| Set a field value | chromedp.SetValue | selector, value, chromedp.ByQuery | First argument is the CSS selector. |
| Submit a form | chromedp.Submit | selector, chromedp.ByQuery | Fires a submit event on the form. |
| Follow a new tab | chromedp.WaitNewTarget | predicate function | Returns a channel of target IDs. |
Action names and argument shapes are taken from the captured code examples. Selector values and page-specific behavior will differ by site.
Frequently asked questions
Why does chromedp need a context?
The context tracks the browser session and its operations. Creating it with chromedp.NewContext and deferring cancel() ensures the browser process and related resources are cleaned up when the program exits.
Why wait for an element before extracting?
A navigation call can return before the page has finished rendering. Waiting for a visible selector, such as the body or a specific container, reduces the chance that an extraction action runs against an incomplete DOM.
How do I read a page that opens in a new tab?
Call chromedp.WaitNewTarget before the action that opens the tab, then create a new context with chromedp.WithTargetID using the target ID from the returned channel. Run your extraction actions in that new context.
What does chromedp.ByQuery mean?
It tells the action to interpret the selector string as a CSS query. The form example uses it with SetValue and Submit so that selectors like the first-name input and the form element are matched as CSS selectors.
Should I use fmt or log for output?
Both print to the console. log is safer when multiple goroutines write at once and includes timestamps, which helps when you are diagnosing timing issues in browser automation.