Scraping AJAX Pages with Java: PhantomJS, Selenium and Smarter Waits
Many sites build their content in the browser after the first HTML response. A plain HTTP fetch therefore returns a shell, not the data. This guide shows one Java approach: run a headless browser, let it execute the JavaScript, and interact with the page as a user would. It also explains how to decide whether that is the right approach at all.
When a headless browser is the right tool
A browser-driven scraper is the pragmatic choice when the page needs many JavaScript calls, when the rendered DOM depends on scripts you cannot easily reproduce, or when you need to click, scroll or type before the data appears. It is heavier than a direct request, so treat it as the fallback, not the default. If a single visible button triggers one predictable request, intercepting and replaying that request is usually simpler and faster.
Prerequisites
You need a working Java environment and a basic grasp of HTML parsing and authentication. The original walkthrough builds on two earlier tutorials: an introduction to web scraping with Java and a guide to logging in to sites. Familiarity with Selenium's WebDriver API also helps, because the interaction code is standard WebDriver.
Setting up PhantomJS with Selenium
The stack is PhantomJS (a WebKit-based headless browser) piloted through Selenium and GhostDriver. PhantomJS renders the DOM much like a normal browser and is comparatively fast. Download the PhantomJS binary first, then add the two Maven dependencies below to your pom.xml. The versions shown are the ones used in the original walkthrough; check current compatibility before adopting them in a new project.
Driver setup code
The setup method below builds a DesiredCapabilities object, enables JavaScript, disables screenshots, points Selenium at the PhantomJS executable, sets a custom User-Agent header, and passes CLI arguments for web security, SSL handling and log level. It finishes by sizing the window to 1920×1080. Replace the executable path with the location of your own PhantomJS binary. The CLI arguments are worth reviewing against the PhantomJS documentation, since they control security and SSL behaviour.
Worked example: clicking a Load More button
The example page is Inshort, a news site whose Load More button fires an AJAX call that appends more article cards. After navigating to the base URL, the code counts the article divs inside the card-stack container, finds the button by its id, clicks it, waits, and counts again. The printed result shows the before/after difference. You could also dump the full DOM with driver.getPageSource() and compare it in a real browser to see exactly what changed.
Choosing a wait strategy
A fixed Thread.sleep(800) is the simplest option and the weakest: the number is arbitrary, so the scraper may run slower than necessary or race ahead of a slow response. Two better options exist. If the page uses jQuery for its AJAX calls, you can poll jQuery.active until it returns 0, which indicates no calls are in flight. If you know which element the response will render, you can wait until that element becomes clickable using an XPath expression. The first option depends on jQuery being present; the second depends on knowing the target element in advance.
Decision checklist
Use this sequence when you meet a JavaScript-rendered page:
- Inspect the network panel. If one request returns the data, replay it directly and skip the browser.
- If the page needs many scripted interactions, use a headless browser.
- If the page uses jQuery, prefer the jQuery.active wait over a fixed sleep.
- If you know the rendered element, wait on that element instead.
- Only fall back to a fixed sleep when neither condition is available, and document why.
Conclusion
PhantomJS plus Selenium gives Java scrapers a way to render and interact with AJAX-heavy pages. For simple cases, intercepting the underlying request is cheaper. For pages with many calls and complex rendering, the headless browser earns its overhead. The next step in the original series was to analyse the AJAX calls and issue the requests directly.
Frequently asked questions
Why does a plain HTTP request miss the data on an AJAX page?
The first response often contains only a shell. The content is inserted later by JavaScript, so a client that does not execute scripts never sees it. A headless browser executes the scripts and exposes the rendered DOM.
Is Thread.sleep a reliable way to wait for AJAX?
It is the least reliable option. The duration is arbitrary, so it can be too short on a slow response or waste time on a fast one. Prefer a condition-based wait such as jQuery.active returning 0 or a known element becoming clickable.
When does the jQuery.active wait work?
Only when the page's AJAX calls go through jQuery. The wait polls the internal counter until no calls are in flight. If the site uses another library or native fetch, this condition will not reflect its activity.
Why does the setup code fail with an executable path error?
The capability PHANTOMJS_EXECUTABLE_PATH_PROPERTY must point to your own PhantomJS binary. The example uses /usr/local/bin/phantomjs, which is specific to one machine.