thermoptic

<!-- ![Rare image of the Major smiling]() -->

<a href="https://www.youtube.com/watch?v=qkXReli7eC0" target="_blank"><img src="_readme/gits.gif" width="100%"></a>

"I don't believe it, thermoptic camouflage!"

What is it?

This is an HTTP proxy designed to bypass services that use fingerprinting such as JA4+ to block certain HTTP clients. Using this proxy, you can use your preferred HTTP clients like curl and still have magically indistinguishable fingerprints from a real (Chrome/Chromium) web browser. thermoptic also comes with some fun features to mitigate JavaScript-based fingerprinting. It also makes it easy to do hybrid scraping using both a web browser and low-level HTTP clients together.

Even if you’re unfamiliar with JA4+ fingerprinting, if you’ve done any scraping you’ve probably been blocked by it before. Popular services such as Cloudflare use such techniques (and other tricks) to detect use of "non-human" HTTP clients to block requests. These services can also use this fingerprinting to detect if you start a session with a real browser and then switch to a low-level client like curl later. thermoptic solves all of these problems by presenting a unified "real" browser fingerprint for all scraping requests.

Example

Here's an example JA4H (HTTP) fingerprint of curl without the proxy:

$ curl https://ja4db.com/id/ja4h/
ge11nn090000_b6a016211e8a_000000000000_e3b0c44298fc

This is quite different from the fingerprint that Chrome produces when you visit the URL directly:

ge11cn19enus_f2808f0d04cf_9a10d4221160_7068f58def6e

However, when we use the proxy to make the request, our JA4H fingerprint is magically identical:

$ curl --proxy http://thermoptic:1234 https://ja4db.com/id/ja4h/
ge11cn19enus_f2808f0d04cf_9a10d4221160_7068f58def6e

(The same goes for our JA4 TLS fingerprint as well, etc).

Setup

To start a thermoptic proxy which cloaks your traffic through a containerized Chrome instance on Ubuntu 22.04:

Regular Docker setup (works on hosts without a GPU runtime):

docker compose up --build

That's all, now you can proxy traffic through it:

curl --proxy http://127.0.0.1:1234 --insecure https://ja4db.com/id/ja4h/

Important notes:

• By default the proxy runs without authentication. If you plan on exposing the proxy externally please make sure to set authentication with the environment variables PROXY_USERNAME and PROXY_PASSWORD.
• If you don't want to use ---insecure you need to use the generated CA file located in ./ssl/rootCA.crt. This is generated the first time you run thermoptic.
• You can connect thermoptic to any Chrome/Chromium instance launched with the --remote-debugging-port flag. This is essential as you'll want to set up and proxy through more commonly used environments to keep your fingerprint as low profile as possible (e.g. Chrome on Windows).
• The GPU compose override is intended for NVIDIA hosts that already have Docker's NVIDIA runtime/toolkit installed. It reserves the GPU device, mounts /dev/dri, and lets the bundled Chrome container switch to the NVIDIA/Vulkan rendering path. To use this, run docker compose -f docker-compose.yml -f docker-compose.gpu.yml up --build.

Features

• 🕵️ Browser-parity proxying that replays requests through a real Chrome session to match JA4 fingerprints byte-for-byte.
• 🤝 Little to no custom code required to integrate your HTTP client (e.g. curl, requests, etc) with thermoptic, just set the proxy and your fingerprints are taken care of.
• 🪝 Hook framework for before-request/after-request/on-start automation so you can drive the full browser to solve challenges, or capture artifacts.
  • 📘 An example Cloudflare turnstile solving hook can be seen in ./hooks/onstart.js.
• 🖥️ Web-browser control UI (at http://127.0.0.1:14111) to control the Dockerized Chrome browser window. Useful to manually log into sites manually and then seemlessly use the proxy to make requests as your logged-in session (and for debugging).
• 🔌 Set an upstream HTTP or SOCKS proxy URI via the UPSTREAM_PROXY environment variable in docker-compose.yml.
• 🛡️ Built-in health checks and restart control loop to detect frozen browsers and recover automatically without operator babysitting.
• ⚡ Supports HTTP/1.1 and HTTP/2, allowing traffic to be proxied over either protocol. (Note that you may talk HTTP/1.1 to the proxy, and the puppeted Chrome may negotiate with the end site over a different protocol.)

How does this cloaking work exactly?

• An HTTP request is made using an HTTP client such as curl with thermoptic set as a proxy.
• thermoptic analyzes the request to best determine what type of browser request it's supposed to be (e.g. manual URL visit? Form submission? A fetch() request?).
• thermoptic uses the Chrome Debugging Protocol (CDP) to puppet the browser and set up a page that mocks the request exactly as it normally would occur in a real web browser.
• thermoptic triggers the request via the mocked context and captures the HTTP response.
• thermoptic sends the HTTP response back to the client.

Due to the fact that the browser is actually making the request using its full stack, the resulting JA4 fingerprints are identitical.

NOTE: Due to many WAFs employing JavaScript-level fingerprinting of web browsers, thermoptic also exposes hooks to utilize the browser for key steps of the scraping process. See this section for more information on this.

Why this approach vs other solutions?

To put it bluntly: other approaches have fundamental flaws which prevent them from being a practical long term solution to the browser fingerprinting problem.

Many other attempts to "beat" browser JA4+ fingerprinting do so by reimplementing the various layers of the browser stack. This approach has a number of serious drawbacks, such as:

• Requiring great care to match the behavior of the "real" browser implementation perfectly. As a result, any quirks or discrepancies can be used to differentiate these clients from the "real" browser implementation.
• Attempting to only solve the problem at one layer of the stack. Chrome utilizes multiple protocols to provide a web browsing experience. As a result, even if you've made a perfectly-matching TLS layer your HTTP layer may give you away if it's not byte-perfect.
• As the "real" browsers regularly change their behavior, their fingerprints change, and as a result these tools constantly require more intensive development work to compensate.

In contrast, because thermoptic uses the browser itself to perform HTTP requests:

• Each layer of the stack such as TCP, TLS, HTTP are indistinguishable from the real browser because the request is made using a real browser in the way it would normally occur.
• Changes to browser behavior at various layers are minimally disruptive, the browser thermoptic controls just needs to be updated in order to match the latest set of fingerprints.

Of course, no solution is without drawbacks. See the DOWNSIDES.md docs for a fleshed out list of downsides to the thermoptic approach.

FAQ

Why the name thermoptic?

"Thermoptic" (short for "thermoptic camouflage") is a reference to the fictional camouflage used by the Major in the Ghost in the Shell (1995) anime. In the movie, this camouflage is shown to be able to hide the wearer across multiple spectrums of detection including both visual light and thermal radiation. Similarly, this tool attempts to cloak the user from fingerprinting across multiple channels (HTTP, TLS, etc).

JA4+ is a suite of fingerprints! Which ones does it spoof?

This tool will spoof the following JA4 fingerprints to be exactly like the Chrome/Chromium browser you're connected to:

• JA4 (TLS fingerprint)
• JA4H (HTTP fingerprint)
• JA4X (X509 TLS certificate fingerprint)
• JA4T (TCP fingerprint)

What if I want to use another upstream HTTP/SOCKS proxy with this?

thermoptic now routes the controlled Chrome instance through an internal proxyrouter service, so you can point Chrome at upstream HTTP or SOCKS proxies (including ones that require credentials). Set the upstream proxy URI by editing the UPSTREAM_PROXY value in docker-compose.yml under the proxyrouter service. When you leave it empty, Chrome talks directly to the internet through the unauthenticated in-cluster proxy.

Example for setting an upstream SOCKS proxy:

  proxyrouter:
    environment:
      UPSTREAM_PROXY: "socks5://username:password@1.2.3.4:1080"

Be aware that some upstream proxies can change low-level fingerprints (for example, TCP metadata) which may reduce parity with a residential browser.

What about cookies?

thermoptic will load the browser with the cookies your client specifies in the Cookie header. The request will then include these once it is executed in the browser context. This is done to ensure the server cannot fingerprint the ordering of the cookies or any other cheap tricks like that.

NOTE: these cookies will stick around after the request as well. If you'd like to implement cleanup logic for cookies, please write a thermoptic hook.

OK, but need a full web browser to click through X prompt to pass human verification!

Yep, thermoptic supports hybrid usage