tcpdp

tcpdp is TCP dump tool with custom dumper and structured logger written in Go.

tcpdp has 3 modes:

• TCP Proxy server mode
• Probe mode ( using libpcap )
• Read pcap file mode

Usage

tcpdp proxy : TCP proxy server mode

$ tcpdp proxy -l localhost:12345 -r localhost:1234 -d hex # hex.Dump()

$ tcpdp proxy -l localhost:55432 -r db.internal.example.com:5432 -d pg # Dump query of PostgreSQL

$ tcpdp proxy -l localhost:33306 -r db.example.com:3306 -d mysql # Dump query of MySQL

With server-starter

https://github.com/lestrrat-go/server-starter

$ start_server --port 33306 -- tcpdp proxy -s -r db.example.com:3306 -d mysql

With config file

$ tcpdp proxy -c config.toml

tcpdp probe : Probe mode (like tcpdump)

$ tcpdp probe -i lo0 -t localhost:3306 -d mysql # is almost the same setting as 'tcpdump -i lo0 host 127.0.0.1 and tcp port 3306'

$ tcpdp probe -i eth0 -t 3306 -d hex # is almost the same setting as 'tcpdump -i eth0 tcp port 3306'

tcpdp read : Read pcap file mode

$ tcpdump -i eth0 host 127.0.0.1 and tcp port 3306 -w mysql.pcap
$ tcpdp read mysql.pcap -d mysql -t 3306 -f ltsv

tcpdp config Create config

$ tcpdp config > myconfig.toml

Show current config

$ tcpdp config

config format

[tcpdp]
pidfile = "/var/run/tcpdp.pid"
dumper = "mysql"

[probe]
target = "db.example.com:3306"
interface = "en0"
bufferSize = "2MB"
immediateMode = false
snapshotLength = "auto"
internalBufferLength = 10000
filter = ""

[proxy]
useServerStarter = false
listenAddr = "localhost:3306"
remoteAddr = "db.example.com:3306"

[log]
dir = "/var/log/tcpdp"
enable = true
enableInternal = true
stdout = true
format = "ltsv"
rotateEnable = true
rotationTime = "daily"
rotationCount = 7
# You can execute arbitrary commands after rotate
# $1 = prev filename
# $2 = current filename
rotationHook = "/path/to/after_rotate.sh"
fileName = "tcpdp.log"

[dumpLog]
dir = "/var/log/dump"
enable = true
stdout = false
format = "json"
rotateEnable = true
rotationTime = "hourly"
rotationCount = 24
fileName = "dump.log"

Installation

$ go get github.com/k1LoW/tcpdp

Architecture

tcpdp proxy connection diagram

      client_addr
           ^
           |        tcpdp
+----------|---------------+
|          v               |
|  proxy_listen_addr       |
|         + ^              |
|         | |   +--------+ |
|         |<----+ dumper | |
|         | |<--+        | |
|         | |   +--------+ |
|         v +              |
|  proxy_client_addr       |
|          ^               |
+----------|---------------+
           |
           v
      remote_addr

tcpdp probe connection diagram

                    server
+--------------------------+
|                          |
|                      +---+---+
|       <--------------| eth0  |----------->
|            interface +---+---+
|            /target     ^ |
|                        | |
|         tcpdp          | |
|        +--------+      | |
|        | dumper +------+ |
|        +--------+        |
+--------------------------+

tcpdp read diagram

                  tcpdp
+--------+ STDIN +--------+ STDOUT
| *.pcap +------>+ dumper +-------->
+--------+       +--------+

tcpdp.log ( tcpdp proxy or tcpdp probe )

| key | description | mode | | --- | ----------- | ---- | | ts | timestamp | proxy / probe / read | | level | log level | proxy / probe | | msg | log message | proxy / probe | | error | error info | proxy / probe | | caller | error caller | proxy / probe | | conn_id | TCP connection ID by tcpdp | proxy / probe | | target | probe target | proxy / probe | | dumper | dumper type | proxy / probe | | use_server_starter | use server_starter | proxy | | conn_seq_num | TCP comunication sequence number by tcpdp | proxy | | client_addr | client address | tcpdp.log, hex, mysql, pg | proxy | | remote_addr | remote address | proxy | | proxy_listen_addr | listen address| proxy | | direction | client to remote: -> / remote to client: <- | proxy | | interface | probe target interface | probe | | mtu | interface MTU (Maximum Transmission Unit) | probe | | mss | TCP connection MSS (Max Segment Size) | probe | | probe_target_addr | probe target address | probe | | filter | BPF (Berkeley Packet Filter) | probe | | buffer_size | libpcap buffer_size | probe | | immediate_mode | libpcap immediate_mode | probe | | snapshot_length | libpcap snapshot length | probe | | internal_buffer_length | tcpdp internal packet buffer length | probe |

Dumper

mysql

MySQL query dumper

NOTICE: MySQL query dumper require --target option when tcpdp proxy tcpdp probe

| key | description | mode | | --- | ----------- | ---- | | ts | timestamp | proxy / probe / read | | conn_id | TCP connection ID by tcpdp | proxy / probe / read | | conn_seq_num | TCP comunication sequence number by tcpdp | proxy | | client_addr | client address | proxy | | proxy_listen_addr | listen address| proxy | | proxy_client_addr | proxy client address | proxy | | remote_addr | remote address | proxy | | direction | client to remote: -> / remote to client: <- | proxy | | interface | probe target interface | probe | | src_addr | src address | probe / read | | dst_addr | dst address | probe / read | | probe_target_addr | probe target address | probe | | proxy_protocol_src_addr | proxy protocol src address | probe / proxy /read | | proxy_protocol_dst_addr | proxy protocol dst address | probe / proxy /read | | query | SQL query | proxy / probe / read | | stmt_id | statement id | proxy / probe / read | | stmt_prepare_query | prepared statement query | proxy / probe / read | | stmt_execute_values | prepared statement execute values | proxy / probe / read | | character_set | character set | proxy / probe / read | | username | username | proxy / probe / read | | database | database | proxy / probe / read | | seq_num | sequence number by MySQL | proxy / probe / read | | command_id | command_id for MySQL | proxy / probe / read |

pg

PostgreSQL query dumper

NOTICE: PostgreSQL query dumper require --target option tcpdp proxy tcpdp probe

| key | description | mode | | --- | ----------- | ---- | | ts | timestamp | proxy / probe / read | | conn_id | TCP connection ID by tcpdp | proxy / probe / read | | conn_seq_num | TCP comunication sequence number by tcpdp | proxy | | client_addr | client address | proxy | | proxy_listen_addr | listen address| proxy | | proxy_client_addr | proxy client address | proxy | | remote_addr | remote address | proxy | | direction | client to remote: -> / remote to client: <- | proxy | | interface | probe target interface | probe | | src_addr | src address | probe / read | | dst_addr | dst address | probe / read | | probe_target_addr | probe target address | probe | | proxy_protocol_src_addr | proxy protocol src address | probe / proxy /read | | proxy_protocol_dst_addr | proxy protocol dst address | probe / proxy /read | | query | SQL query | proxy / probe / read | | portal_name | portal Name | proxy / probe / read | | stmt_name | prepared statement name | proxy / probe / read | | parse_query | prepared statement query | proxy / probe / read | | bind_values | prepared statement bind(execute) values | proxy / probe / read | | username | username | proxy / probe / read | | database | database | proxy / probe / read | | message_type | message type for PostgreSQL | proxy / probe / read |

hex

| key | description | mode | | --- | ----------- | ---- | | ts | timestamp | proxy / probe / read | | conn_id | TCP connection ID by tcpdp | proxy / probe / read | | conn_seq_num | TCP comunication sequence number by tcpdp | proxy | | client_addr | client address | proxy | | proxy_listen_addr | listen address| proxy | | proxy_client_addr | proxy client address | proxy | | remote_addr | remote address | proxy | | direction | client to remote: -> / remote to client: <- | proxy | | interface | probe target interface | probe | | src_addr | src address | probe / read | | dst_addr | dst address | probe / read | | probe_target_addr | probe target address | probe | | proxy_protocol_src_addr | proxy protocol src address | probe / proxy /read | | proxy_protocol_dst_addr | proxy protocol dst address | probe / proxy /read | | bytes | bytes string by hex.Dump | proxy / probe / read | | ascii | ascii string by hex.Dump | proxy / probe / read |

References

• https://github.com/jpillora/go-tcp-proxy
• https://github.com/dmmlabo/tcpserver_go