<h1 align="center"> <img src="./docs/readme-header.webp" alt="Monastic Academy" height="450px"> <br> httptap </br> </h1> <p align="center"> <a href="https://pkg.go.dev/github.com/monasticacademy/httptap"><img src="https://img.shields.io/badge/go.dev-reference-007d9c?logo=go&logoColor=white&style=flat-square" alt="Documentation"></a> <a href="https://github.com/monasticacademy/httptap/actions"><img src="https://github.com/monasticacademy/httptap/workflows/Test/badge.svg" alt="Build Status"></a> </p> <br>

View the HTTP and HTTPS requests made by any linux program by running httptap -- <command>. For example, the following runs curl on "monasticacademy.org", which results in an HTTP status of 308 (Redirect):

$ httptap -- curl https://monasticacademy.org
---> GET https://monasticacademy.org/
<--- 308 https://monasticacademy.org/ (15 bytes)

Now let's try the same thing with an HTTP request from python. This time we see that python follows the redirect and gets a 200 OK response:

httptap -- python -c "import requests; requests.get('https://monasticacademy.org')"
---> GET https://monasticacademy.org/
<--- 308 https://monasticacademy.org/ (15 bytes)
---> GET https://www.monasticacademy.org/
<--- 200 https://www.monasticacademy.org/ (5796 bytes)

To run httptap you do not need to be the root user. You do not need to set up any kind of daemon or make any system-wide changes to your system (edit: on Ubuntu 23.10 and later you will need to run the sysctl documented below). It will not create any iptables rules or change your routing table, and generally will not affect any other processes running on the same system. The httptap executable is a static Go binary that runs without dependencies.

Httptap only runs on linux at present. It makes use of linux-specific system calls -- in particular network namespaces -- that will unfortunately make it very difficult to port to other operating systems. If you know how httptap could be ported to other operating systems then please get in touch!

Install pre-built binary

curl -L https://github.com/monasticacademy/httptap/releases/latest/download/httptap_linux_$(uname -m).tar.gz | tar xzf -

For all versions and CPU architectures see the latest releases page.

Install with Go

go install github.com/monasticacademy/httptap@latest

Ubuntu 23.10 and later

On Ubuntu 23.10 and later you will need to run the following in order to use httptap:

sudo sysctl -w kernel.apparmor_restrict_unprivileged_unconfined=0
sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0

What this does is disable a recent kernel feature that restricts unprivileged user namespaces. The above may also be needed on other distros that have disabled unprivileged user namespaces by default. I will update this documentation as I learn more. I am investigating ways to avoid the need for this entirely by shipping an apparmor profile with httptap.

Quickstart

Let's run a simple test:

httptap -- curl -s https://buddhismforai.sutra.co -o /dev/null
---> GET https://buddhismforai.sutra.co/
<--- 302 https://buddhismforai.sutra.co/ (117 bytes)

What happened here is that we ran curl -s https://buddhismforai.sutra.co -o /dev/null and it received a 302 redirect from the server. httptap printed summaries of the HTTP requests and their responses. Let's see how it changes if we tell curl to follow redirects by adding -L:

httptap -- curl -sL https://buddhismforai.sutra.co -o /dev/null
---> GET https://buddhismforai.sutra.co/
<--- 302 https://buddhismforai.sutra.co/ (117 bytes)
---> GET https://buddhismforai.sutra.co/space/cbodvy/content
<--- 200 https://buddhismforai.sutra.co/space/cbodvy/content (6377 bytes)

Now we can see that after receiving the 302 redirect, curl made an additional HTTP request to the URL to which it was redirected, which is what you expect when using -L with curl.

Let's see what HTTP endpoints the Google Cloud command line interface uses to list compute resources (this requires that you have gcloud installed and are signed in):

$ httptap -- gcloud compute instances list
---> POST https://oauth2.googleapis.com/token
<--- 200 https://oauth2.googleapis.com/token (997 bytes)
---> GET https://compute.googleapis.com/compute/v1/projects/maple-public-website/aggregated/instances?alt=json&includeAllScopes=True&maxResults=500&returnPartialSuccess=True
<--- 200 https://compute.googleapis.com/compute/v1/projects/maple-public-website/aggregated/instances?alt=json&includeAllScopes=True&maxResults=500&returnPartialSuccess=True (19921 bytes)
NAME       ZONE        MACHINE_TYPE  PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP     STATUS
<your cloud instances listed here>

What happened here is that we ran gcloud compute instances list, which lists the compute instances that the signed-in user has on Google Cloud. The bottom two lines of output were printed by gcloud; the rest were printed by httptap and show what HTTP requests gcloud used to get the information it printed.

Let's see what HTTP endpoints kubectl uses in a "get all" (this requires that you have kubectl installed and are authenticated to a cluster):

$ httptap --https 443 6443 -- kubectl get all --insecure-skip-tls-verify
---> GET https://cluster:6443/api/v1/namespaces/default/pods?limit=500
<--- 200 https://cluster:6443/api/v1/namespaces/default/pods?limit=500 (38345 bytes)
---> GET https://cluster:6443/api/v1/namespaces/default/replicationcontrollers?limit=500
<--- 200 https://cluster:6443/api/v1/namespaces/default/replicationcontrollers?limit=500 (2509 bytes)
---> GET https://cluster:6443/api/v1/namespaces/default/services?limit=500
<--- 200 https://cluster:6443/api/v1/namespaces/default/services?limit=500 (5586 bytes)
---> GET https://cluster:6443/apis/apps/v1/namespaces/default/daemonsets?limit=500
<--- 200 https://cluster:6443/apis/apps/v1/namespaces/default/daemonsets?limit=500 (3052 bytes)
---> GET https://cluster:6443/apis/apps/v1/namespaces/default/deployments?limit=500
<--- 200 https://cluster:6443/apis/apps/v1/namespaces/default/deployments?limit=500 (7438 bytes)
---> GET https://cluster:6443/apis/apps/v1/namespaces/default/replicasets?limit=500
<--- 200 https://cluster:6443/apis/apps/v1/namespaces/default/replicasets?limit=500 (47211 bytes)
---> GET https://cluster:6443/apis/apps/v1/namespaces/default/statefulsets?limit=500
<--- 200 https://cluster:6443/apis/apps/v1/namespaces/default/statefulsets?limit=500 (1416 bytes)
---> GET https://cluster:6443/apis/autoscaling/v2/namespaces/default/horizontalpodautoscalers?limit=500
<--- 200 https://cluster:6443/apis/autoscaling/v2/namespaces/default/horizontalpodautoscalers?limit=500 (2668 bytes)
---> GET https://cluster:6443/apis/batch/v1/namespaces/default/cronjobs?limit=500
<--- 200 https://cluster:6443/apis/batch/v1/namespaces/default/cronjobs?limit=500 (3134 bytes)
---> GET https://cluster:6443/apis/batch/v1/namespaces/default/jobs?limit=500
<--- 200 https://cluster:6443/apis/batch/v1/namespaces/default/jobs?limit=500 (2052 bytes)
<ordinary kubectl output here>

In the above, --insecure-skip-tls-verify is necessary because kubectl doesn't use the httptap-generated certificate authority, and --https 443 6443 says to treat TCP connections on ports 443 and 6443 as HTTPS connections, which is needed because my cluster's API endpoint uses port 6443.

Let's see how DNS-over-HTTP works when you use --doh-url with curl:

$ httptap -- curl -sL --doh-url https://cloudflare-dns.com/dns-query https://buddhismforai.sutra.co -o /dev/null
---> POST https://cloudflare-dns.com/dns-query
<--- 200 https://cloudflare-dns.com/dns-query (149 bytes)
---> POST https://cloudflare-dns.com/dns-query
<--- 200 https://cloudflare-dns.com/dns-query (150 bytes)
---> GET https://buddhismforai.sutra.co/
<--- 302 https://buddhismforai.sutra.co/ (117 bytes)
---> GET https://buddhismforai.sutra.co/space/cbodvy/content
<--- 200 https://buddhismforai.sutra.co/space/cbodvy/content (6377 bytes)

What happened here is that we told curl to request the url "https://buddhismforai.sutra.co", using the cloudflare DNS-over-HTTP service at cloudflare-dns.com. In the output we see that curl made 4 HTTP requests in total; the first two were DNS lookups, and then the second two were the ordinary HTTP requests for buddhismforai.sutra.co.

Let's print the contents of the DNS-over-HTTP payloads:

$ httptap --head --body -- curl -sL --doh-url https://cloudflare-dns.com/dns-query https://buddhismforai.sutra.co -o /dev/null
---> POST https://cloudflare-dns.com/dns-query
> Accept: */*
> Content-Type: application/dns-message
> Content-Length: 40
buddhismforaisutraco
<--- 200 https://cloudflare-dns.com/dns-query (149 bytes)
< Alt-Svc: h3=":443"; ma=86400
< Server: cloudflare
< Date: Tue, 24 Dec 2024 18:13:12 GMT
< Content-Type: application/dns-message
< Access-Control-Allow-Origin: *
< Content-Length: 149
< Cf-Ray: 8f7290631e334211-EWR
buddhismforaisutraco�
��w�4+#G�.           <wildcardsutraco	herokudnscom�4+!�=�4+
...

Here the --head option tells httptap to print the HTTP headers, and --body tells it to print the raw HTTP payloads. To keep it short I'm showing just the first request/response pair.

HAR output

You can dump the HTTP requests and responses to a HAR file like this:

$ httptap --dump-har out.har -- curl -Lso /dev/null https://monasticacademy.org

There are many HAR viewers out there that can visualize this dump file. For example here is how the above looks in the Google HAR Analyzer:

Again, what you're looking at here is one HTTP request to https://monasticacademy.or