Charles proxy for web scraping
Many modern sites and apps do not ship their data inside the HTML. They fetch it from a backend endpoint and render it in the browser or on the phone. Charles is an HTTP debugging proxy that sits between your machine or device and the internet, so you can watch those calls, read the headers and cookies, and see the JSON that comes back. This guide walks through installing Charles, decrypting HTTPS, locating the endpoint behind a JavaScript-heavy page, doing the same on a mobile app, and converting a working request into a Python script.
What Charles gives you that browser dev tools do not
Charles is a debugging proxy, not a scraper. It records every request that passes through it, including traffic from apps and tools outside your browser, and it can decrypt TLS so the bodies are readable. The features that matter most for data work are bandwidth throttling to imitate a slow connection, request editing to probe how a backend reacts, request replay, and full-text search across a long list of captured requests. That last one is what makes Charles practical: on a busy page you may have hundreds of calls, and searching the bodies for a field name is faster than clicking through them one by one.
Install and trust the certificate
Download Charles from its official download page. Builds exist for macOS, Windows and Linux; the walkthrough below follows macOS, and the menu names are close but not identical on other platforms.
After installing, add the Charles root certificate to your system trust store. Without that step Charles can see that a connection exists but cannot read the encrypted contents. On macOS the installer opens Keychain Access, where you open the certificate's Trust section, choose to always trust it, and confirm with your system password.
Next open Proxy > SSL Proxying Settings and add either a wildcard entry or the specific host you want to inspect. From that point Charles decrypts HTTPS for the hosts you listed. Keep the list narrow when you can: a wildcard captures far more traffic than you need and makes the session harder to read.
Why the API is often the better target
A server-rendered page arrives as finished HTML. A client-rendered app built with React, Vue or Angular instead ships a shell and then calls a backend API for the content. If you can call that same API yourself, you skip rendering JavaScript entirely: no headless browser, no large memory footprint, no waiting for the page to settle. You also get structured JSON instead of markup that changes whenever the design does.
That is the trade this guide is about. Rendering is the fallback when the API is protected or the data only exists after client-side computation. Calling the endpoint directly is the faster path when the request turns out to be reproducible.
Finding the endpoint behind a JavaScript page
Start Charles, open the site, and scroll far enough to trigger several loads. By default Charles records traffic from your whole system, so filter the session down to the domain you care about before reading anything.
In the ProductHunt example the interesting calls are POSTs to a GraphQL endpoint. Open one and compare it with the next: the request body is largely identical, and the value that moves is a cursor. That cursor is base64-encoded, and Charles can decode it in place — select the value, right-click, and read the decoded form.
Rather than rebuilding a request with dozens of headers and cookies by hand, right-click it and choose Compose. Change the cursor to a different page number encoded in base64, then execute. If the response still comes back, try deleting cookies one at a time and re-executing. When the endpoint answers without them, your eventual script gets much simpler.
From a working request to a Python call
Once the request succeeds in Compose, export it as a cURL command. A converter such as the cURL-to-Python service turns that command into a Requests snippet with the headers and body already filled in. The captured request for this example is preserved in full as a protected code module, including the GraphQL query and the base64 cursor value.
Two adjustments are worth making in the generated script. First, the example adds verify=False to silence TLS warnings; that disables certificate checking, so treat it as a debugging shortcut rather than something to keep in production. Second, the cursor is the pagination control: encode the page you want in base64 and swap it in, and the same POST returns a different slice of results.
Capturing traffic from a mobile app
Check the app's terms of service before you analyze its traffic; some prohibit it outright.
Older workflows required pointing a phone at a desktop Charles instance over the same Wi-Fi network. Since 2018 there is a native iOS Charles app, which removes most of that setup. Install it, switch it on from the main screen, allow the VPN configuration it requests, then enable SSL proxying in its settings and follow the on-screen prompts.
Expect breakage. Turning on SSL proxying means Charles presents its own root certificate to the app, which is effectively a man-in-the-middle position you have created against yourself. Apps that validate the certificate chain will refuse the connection — that behavior is called SSL pinning, and it is the usual reason a capture shows nothing useful.
Working through a mobile session
With Charles active, open the app and scroll to load more content, then filter to the app's domain so other apps' requests do not drown the view. Select the request sequence and share it to your Mac as a session file; the analysis is easier on the desktop, and the file ends in .chlsj. Double-click it to open the session in Charles for desktop.
In the dev.to example the useful calls hit a /search endpoint that behaves like a standard REST API. Before writing any code, confirm what the endpoint actually requires: create a fresh session, filter to the domain again, open the original request in Compose, strip the cookies and headers, and execute. If it still returns data, the endpoint needs no authentication. The parameters that matter are the page number and the per-page count, both of which you can vary freely.
The rebuilt mobile request
Exporting that request to cURL and converting it produces a short GET call with a parameter tuple: per_page, page, sort_by, sort_direction, approved and class_name. The complete script is preserved as a protected code module so the exact parameter names and values stay intact.
This is the payoff of the whole workflow. A page that is awkward to render becomes a single HTTP request returning JSON, and pagination is a loop over one integer.
Where Charles stops being enough
Two capabilities are worth learning once the basics work: breakpoints and response rewriting. Both let you alter traffic in flight, which is useful when you are testing how a front end handles an edge case rather than extracting data.
Charles is also close to essential for browser automation on complex sites. Login flows get hard to follow in ordinary dev tools once JWT tokens, CSRF tokens and OAuth are involved; being able to search across a series of requests and replay or modify them turns guesswork into observation. What Charles does not do is scale the extraction for you — it identifies the request, and your code has to run it.
A short decision checklist
Before you commit to an approach, answer these in order:
- Does the page return its content in the initial HTML? If yes, parse the HTML and stop here.
- Does a captured request return the data as JSON? If yes, prefer calling that endpoint directly.
- Does the endpoint answer after you remove cookies and headers in Compose? If yes, the script will be short and stable.
- Does the endpoint require a token, a signature or a cookie you cannot regenerate? Then you need session handling, or a rendering approach.
- Does the mobile capture show a certificate error rather than data? That is SSL pinning, and no amount of header editing will fix it.
- Are you permitted to make these requests? Check the terms of service and any applicable rules before you automate.
| Symptom | Likely cause | Check |
|---|---|---|
| Requests appear but bodies are unreadable | Root certificate not trusted | Open the certificate's Trust settings and confirm it is set to always trust |
| HTTPS host shows only a CONNECT line | Host not listed for decryption | Add the domain or a wildcard under Proxy > SSL Proxying Settings |
| Session is flooded with unrelated traffic | System-wide capture with no filter | Filter the session to the target domain |
| Mobile app fails to load once proxying is on | App validates the certificate chain (SSL pinning) | Confirm the failure is a certificate rejection before changing anything else |
| Endpoint returns an error after export | Required header or cookie was stripped | Re-add values in Compose one at a time until it succeeds |
This table reflects the setup steps and observed behaviors described in the walkthrough; exact menu labels vary by Charles version and platform.
| Field | Where it appears | What it controls |
|---|---|---|
| cursor | GraphQL variables, base64-encoded | Which page of results is returned |
| featured / visibleOnHomepage | GraphQL variables | Which content set the query targets |
| per_page | Query parameter | How many items come back per call |
| page | Query parameter | Which page of the result set to fetch |
| sort_by / sort_direction | Query parameter | Ordering of the returned items |
| class_name | Query parameter | Record type requested from the endpoint |
Field names and values are taken from the two example requests as captured; other endpoints will use different parameters.
Frequently asked questions
Do I need the Charles root certificate to read HTTPS?
Yes. Charles can only decrypt TLS traffic for hosts you have authorized if your system trusts its root certificate. Install it, set it to always trust, and then list the hosts under SSL Proxying Settings.
Why does a mobile app stop working when SSL proxying is on?
Charles presents its own certificate to the app. Apps that check the certificate chain reject it, which is known as SSL pinning. That is a deliberate defense, not a configuration mistake on your side.
How do I know whether an endpoint needs authentication?
Replay it in Compose with the cookies and headers removed. If the response still contains the data, the endpoint is open to that request shape. If it fails, add values back until you find the one that matters.
Is calling the API better than rendering the page?
Usually, when the request is reproducible. You avoid a headless browser, use far less memory, and parse JSON instead of markup. Rendering remains the fallback when the request cannot be rebuilt outside the app.
What does the base64 cursor actually contain?
In the example it encodes a page position. Decode it in Charles to read the value, then encode your replacement page number the same way before executing the request.