Objective-C Web Scraping: Static Pages and Selenium
Objective-C remains a practical choice when you need to pull data inside a macOS or iOS app. This guide walks through a complete scraper: fetch a page with NSURLSession, parse it with HTMLReader, filter nodes with CSS selectors, read attributes, follow sub-links, and switch to a Selenium-controlled headless browser when the content only appears after JavaScript runs. You should already be comfortable with Objective-C and Xcode; the examples target a macOS App project but can be adapted to iOS.
Scraping a static page
The worked example uses the Wikipedia Physics article. The goal is to load the page, read its title, collect the text of span elements with class mw-headline, then read the href of the first a element inside every paragraph under the div with id mw-content-text and follow those links. The exact code for each stage is supplied in the protected modules referenced below; the prose here explains what each stage does and how to check it.
A useful decision rule before you write any code:
- If the HTML you need is present in the raw response, use the NSURLSession + HTMLReader path.
- If the content is injected by JavaScript after load (a single-page app or web component), skip straight to the Selenium path.
- If you only need a handful of fields from one known page, keep the helper class minimal; add callbacks only for the fields you actually consume.
Create the project
The sample is a macOS App project created from the standard App template in Xcode, which gives you an AppDelegate.m you can edit. A command-line tool also works, but it needs two extra calls: dispatch_main() at the end of main so the queue keeps processing, and exit() to terminate the program when done. If you run inside a sandbox, enable both Incoming Connections and Outgoing Connections in the App Sandbox settings, otherwise the request will fail before it reaches the network.
Load the URL with NSURLSession
The WebCrawl class declares a crawlURL: method and stores the response body in an NSMutableData instance variable. The implementation builds an NSURLRequest, uses the shared NSURLSession (appropriate for a static page), and creates an NSURLSessionDataTask whose completion handler receives data, response and error. The exact declarations and implementation are in the protected modules listed below.
Inside the completion handler, distinguish three outcomes before parsing:
- Transport error — the error parameter is non-null; log it and return.
- HTTP status other than 200 — cast the response to NSHTTPURLResponse, read statusCode, log it and return. The sample checks only 200-OK; add further checks as your target requires.
- Success — append the data to the response buffer and continue to parsing.
Call [task resume] after creating the task, or the request never starts. To test, invoke crawlURL: from applicationDidFinishLaunching: with the Wikipedia URL, then either add an NSLog or set a breakpoint to confirm _responseData holds the page.
Parse with HTMLReader
Several parsers work here, including libxml, HTMLReader and Ono. This walkthrough uses HTMLReader, added through File > Swift Packages > Add Package Dependency with the HTMLReader repository URL. After the package resolves, import HTMLReader and build an HTMLDocument from the response data. The Content-Type header from the NSHTTPURLResponse is required for correct parsing, so read it from allHeaderFields before constructing the document. The exact import, dependency and document-construction code is supplied in the protected modules below.
Read specific data
HTMLDocument exposes two core retrieval calls: firstNodeMatchingSelector: returns the first match, and nodesMatchingSelector: returns all matches. Both accept any CSS selector, not just tag names, so you can target an id, a class, a child relationship, or an attribute. The title example trims whitespace and newlines from textContent before logging; on the Physics page it prints "Physics - Wikipedia".
A short field map for the helper class described later:
- Title callback — receives the trimmed title string.
- Content callback — receives the element name, the textContent, and a running index.
- Attribute callback — receives the element name, index, attribute name and attribute value.
- Id-filtered content callback — receives the HTMLElement, the id name, the element type and an index.
Attributes live on HTMLElement.attributes, an ordered dictionary; read a value with objectForKey:. The exact selector and attribute code is in the protected modules below.
Wrap extraction in a helper class
Rather than repeating selector logic in every scraper, the sample defines a URLScrap class that holds an HTMLDocument and exposes four methods: scrapTitle:, scrapContent:contentHandler:, scrapContentWithId:elemType:contentHandler: and scrapAttributes:attributes:attributeHandler:. The handlers are blocks, so the caller decides what happens to each value. A separate DataHandler class receives the parsed results and can hold business logic; the sample only logs, but the split keeps parsing and processing independent. The exact interface and implementation are in the protected modules below.
When wiring this into crawlURL:, pass the DataHandler into the method and call the URLScrap methods from inside the completion handler. The sample scrapes span[class="mw-headline"] through scrapContent: and forwards each result to handleContent:content:index:.
Follow sub-links with a depth limit
Following links has three parts: select the a elements, read each href, and call crawlURL: on the resolved URL. The sample adds followLinksWithSelector:handler:element:selector:level:currentlevel: to URLScrap. It returns immediately when currentlevel reaches maxlevel, increments the level, and for each matching link reads href, scrubs it through a scrubURL: helper, and only recurses when the result starts with http:// or https://. The example starts from the div with id mw-content-text, uses the selector p > a:first-child, and sets maxlevel to 2 starting at currentlevel 1.
The same pattern applies to any node you can reach with a CSS selector, not just that div. One limitation to plan for: HTMLReader parses HTML only. If a link points to a stylesheet, a JavaScript file or another non-HTML resource, you need a different parser for that resource. The exact follow-links code is in the protected modules below.
Scraping JavaScript-rendered pages
The static path cannot see content that JavaScript builds at load time, such as a page whose body is just a web component placeholder. For those pages you need a headless browser driven programmatically. The sample uses Selenium: run a Grid hub with at least one node, enable the Safari driver, build the Selenium Objective-C framework from source, and drive it from your app. The exact commands, configuration files, framework build steps and driver code are in the protected modules below.
A practical checklist before writing the driver code:
- The Grid hub and node are both running with the same hostname configuration.
- The status endpoint reports ready: true and lists the platformName and browserName you will use.
- safaridriver is present and enabled for automation.
- The Selenium.framework you built is added to Frameworks, Libraries and Embedded Content, and its parent path is in Framework Search Paths.
- The SECapabilities platformName and browserName match what the Grid reported.
The sample loads a salaries page and finds elements by XPath with findElementsBy:. XPath can be copied from the browser's Inspect Element console. The exact driver code is in the protected modules below.
| Page behavior | Recommended path | Key tools |
|---|---|---|
| HTML present in the raw response | Static scraping | NSURLSession, HTMLReader, CSS selectors |
| Content injected by JavaScript after load | Dynamic scraping | Selenium Grid, Safari driver, Selenium Objective-C framework |
| Need to crawl related pages | Static scraping with link following | HTMLReader selectors plus a depth-limited crawlURL: loop |
| Only a few known fields | Static scraping with a minimal helper | firstNodeMatchingSelector: and nodesMatchingSelector: |
This table is a reader decision aid derived from the tutorial's two paths; it is not a benchmark.
Frequently asked questions
What is the difference between a transport error and an HTTP error in the completion handler?
A transport error arrives in the error parameter and means the request never completed at the network layer. An HTTP error means the request completed but the server returned a status other than 200; the sample reads statusCode from the NSHTTPURLResponse and logs it. Handle both before attempting to parse the data.
Why does a command-line tool exit before the request finishes?
Without dispatch_main() at the end of main, the program terminates and the queue never processes the completion handler. Call dispatch_main() to keep it alive, and call exit() when you want the program to end.
Why does the request fail inside the App Sandbox?
The sandbox blocks network access unless you enable both Incoming Connections and Outgoing Connections in the App Sandbox settings.
Can HTMLReader parse a linked stylesheet or JavaScript file?
No. HTMLReader reads HTML only. Linked stylesheets, scripts and other non-HTML resources need a separate parser.
Where do the XPath expressions for Selenium come from?
You can copy them from the browser's Inspect Element console and pass them to findElementBy: or findElementsBy:.
{
"value": {
"ready": true,
"message": "Selenium Grid ready.",
"nodes": [
{
"id": "7b160ac9-d615-4589-bc42-42279cb3cf8b",
"uri": "http:\u002f\u002f192.168.0.103:5555",
"maxSessions": 1,
"osInfo": {
"arch": "x86_64",
"name": "Mac OS X",
"version": "10.14.4"
},
"heartbeatPeriod": 60000,
"availability": "UP",
"version": "4.5.0 (revision fe167b119a)",
"slots": [
{
"id": {
"hostId": "7b160ac9-d615-4589-bc42-42279cb3cf8b",
"id": "d738c605-977d-4194-bf14-e8b27fe84dab"
},
"lastStarted": "2022-10-13T06:48:40.886555Z",
"session": null,
"stereotype": {
"browserName": "safari",
"platformName": "MAC"
}
}
]
}
]
}
}
"/usr/bin/safaridriver --enable"