Kflowd

Kernel-based Process Monitoring on Linux Endpoints for File System, TCP and UDP Networking Events and optionally DNS, HTTP and SYSLOG Application Messages via eBPF Subsystem

Generate Convert Improve

Install / Use

/learn @tarsal-oss/Kflowd

About this skill

Quality Score

0/100

README

Kernel-based Process Monitoring on Linux Endpoints via eBPF

kflowd runs as agent on Linux endpoints to monitor processes via eBPF kernel subsystem for filesystem and TCP and UDP networking events, enabling immediate threat and anomaly detection on suspicious activities.

Advanced non-ebpf related features such as DNS, HTTP and SYSLOG application message decoding, checksum calculation for virus detection, process and file versioning for vulnerability detection and file device, network interface and user-group identification for files and processes can be enabled via open-binary plugin modules.<br>Pre-built kflowd and kflowd-plugins packages can be downloaded for quick installation from the Releases section.

If you would like to join our community Slack channel please send an email to devs@tarsal.co to receive an invitation. You can also contact us directly at kflow@tarsal.co for any questions.

kflowd contains an eBPF program running in kernel context and its control application running in userspace.<br> The eBPF program traces kernel functions to monitor processes based on file system and networking events. Events are aggregated into records and submitted into a ringbuffer where they are polled by the userspace control application. All Records are enriched with process information and then converted into a message in JSON output format.<br> Final messages are printed to stdout console and can be sent via UDP protocol to specified hosts for ingestion in a security data pipeline.

kflowd runs on Linux kernels 5.10+ and is built with the libbpf+CO-RE (Compile-Once-Run-Everywhere) eBPF development toolchain using BTF (BPF Type Format) to allow portability by avoiding dependencies on differences in kernel headers between kernel versions on deployment.

JSON Output

kflowd outputs JSON messages generated for each record of aggregated file system and TCP, UDP networking events and optionally DNS, HTTP and SYSLOG application messages in the formats as shown in the following examples:

<details open> <summary> Filesystem Record</summary>

{
  "InfoSequenceNumber": 1,
  "InfoTimestamp": "Thu, Apr 04 2024 15:34:35.643031330 UTC",
  "InfoMonitor": "filesystem",
  "InfoHostName": "dev.kflow.co",
  "InfoHostIP": "38.110.1.24",
  "InfoSystem": "Linux",
  "InfoKernel": "6.1.0-10-amd64",
  "InfoVersion": "kflowd-v0.9.1",
  "InfoUptime": 21.262713426,
  "ProcParent": "sshd",
  "Proc": "sftp-server",
  "ProcVersion": "1:9.2p1-2+deb12u1",
  "ProcUser": "dirk",
  "ProcGroup": "dirk",
  "ProcPPID": 183546,
  "ProcPID": 183547,
  "ProcTID": 183547,
  "ProcUID": 1002,
  "ProcGID": 1002,
  "ProcAge": 2.408293862,
  "FilePath": "/home/dirk/",
  "File": "malware",
  "FileVersion": "0.9",
  "FileMode": "regular",
  "FileEventCount": 4,
  "FileEvents": {
    "OPEN": 1,
    "MODIFY": 2,
    "CLOSE_WRITE": 1
  },
  "FileEventsDuration": 0.811829334,
  "FileInode": 19567988,
  "FileInodeLinkCount": 1,
  "FileDevice": "902h:/dev/md2:/:ext4",
  "FilePermissions": "0755/-rwxr-xr-x",
  "FileUser": "dirk",
  "FileGroup": "dirk",
  "FileUID": 1002,
  "FileGID": 1002,
  "FileSize": 41,
  "FileSizeChange": 41,
  "FileAccessTime": "Thu, Apr 04 2024 15:12:01.435718956 UTC",
  "FileStatusChangeTime": "Thu, Apr 04 2024 15:34:35.154106191 UTC",
  "FileModificationTime": "Thu, Apr 04 2024 15:34:35.154106191 UTC",
  "FileModificationTimeChange": 0.327993681,
  "FileMD5": "96760f46bd29ba986279f22bed9839f5",
  "FileSHA256": "72c58c2d02ae3a87f521594373433b7d05477c4994fc0ab4376827cadb29ba7e"
}

</details> <details open> <summary> UDP + DNS Networking Record</summary>

{
  "InfoSequenceNumber": 2,
  "InfoTimestamp": "Thu, Apr 04 2024 15:39:11.463732866 UTC",
  "InfoMonitor": "socket",
  "InfoHostName": "dev.kflow.co",
  "InfoHostIP": "38.110.1.24",
  "InfoSystem": "Linux",
  "InfoKernel": "6.1.0-10-amd64",
  "InfoVersion": "kflowd-v0.9.1",
  "InfoUptime": 23.972984597,
  "ProcParent": "bash",
  "Proc": "curl",
  "ProcVersion": "7.45.2-3",
  "ProcUser": "dirk",
  "ProcGroup": "dirk",
  "ProcPPID": 199853,
  "ProcPID": 199856,
  "ProcTID": 199857,
  "ProcUID": 1002,
  "ProcGID": 1002,
  "ProcAge": 0.044454620,
  "SockProtocol": "UDP",
  "SockRole": "CLIENT",
  "SockState": "UDP_ESTABLISHED",
  "SockFamily": "AF_INET",
  "SockLocalIP": "38.110.1.24",
  "SockLocalPort": 56664,
  "SockRemoteIP": "8.8.4.4",
  "SockRemotePort": 53,
  "SockTxInterface": "4:enp5s0:0c:c4:7a:88:84:c2",
  "SockTxPackets": 2,
  "SockTxDuration": 0.000048490,
  "SockTxBytes": 52,
  "SockTxInterface": "4:enp5s0:0c:c4:7a:88:84:c2",
  "SockRxPackets": 2,
  "SockRxPacketsQueued": 0,
  "SockRxPacketsDrop": 0,
  "SockRxPacketsFrag": 0,
  "SockRxDuration": 22.475134750,
  "SockRxBytes": 155,
  "SockRxTTL": 125,
  "SockAge": 0.043689149,
  "App": "DNS",
  "AppTxDns": [{
    "_Timestamp": 0.000001177,
    "TransactionId": 56864,
    "OpCode": "QUERY",
    "Flags": ["RD"],
    "ResourceRecords": [
      ["A", "kflow.co"]
    ]
  },
  {
    "_Timestamp": 0.000048627,
    "TransactionId": 52515,
    "OpCode": "QUERY",
    "Flags": ["RD"],
    "ResourceRecords": [
      ["AAAA", "kflow.co"]
    ]
  }],
  "AppRxDns": [{
    "_Timestamp": 0.037965555,
    "TransactionId": 52515,
    "ResponseCode": "NOERROR",
    "Flags": ["QR", "RD", "RA"],
    "AnswerCount": 0,
    "ResourceRecords": []
  },
  {
    "_Timestamp": 0.043688644,
    "TransactionId": 56864,
    "ResponseCode": "NOERROR",
    "Flags": ["QR", "RD", "RA"],
    "AnswerCount": 2,
    "ResourceRecords": [
      ["A", "kflow.co", 600, "IN", "15.197.142.173"],
      ["A", "kflow.co", 600, "IN", "3.33.152.147"]
    ]
  }]
}

</details> <details open> <summary> TCP + HTTP Networking Record</summary>

{
  "InfoSequenceNumber": 3,
  "InfoTimestamp": "Thu, Apr 04 2024 15:39:11.928989997 UTC",
  "InfoMonitor": "socket",
  "InfoHostName": "dev.kflow.co",
  "InfoHostIP": "38.110.1.24",
  "InfoSystem": "Linux",
  "InfoKernel": "6.1.0-10-amd64",
  "InfoVersion": "kflowd-v0.9.1",
  "InfoUptime": 24.873001288,
  "ProcParent": "bash",
  "Proc": "curl",
  "ProcVersion": "7.45.2-3",
  "ProcUser": "dirk",
  "ProcGroup": "dirk",
  "ProcPPID": 199853,
  "ProcPID": 216998,
  "ProcTID": 216998,
  "ProcUID": 1002,
  "ProcGID": 1002,
  "ProcAge": 0.114196829,
  "SockProtocol": "TCP",
  "SockRole": "CLIENT",
  "SockState": "TCP_CLOSE",
  "SockFamily": "AF_INET",
  "SockLocalIP": "38.110.1.24",
  "SockLocalPort": 43302,
  "SockRemoteIP": "15.197.142.173",
  "SockRemotePort": 80,
  "SockTxInterface": "4:enp5s0:0c:c4:7a:88:84:c2",
  "SockTxDataPackets": 1,
  "SockTxPackets": 6,
  "SockTxPacketsRetrans": 0,
  "SockTxPacketsDups": 0,
  "SockTxFlags": {
    "SYN": 1,
    "ACK": 3,
    "PSH-ACK": 1,
    "FIN-ACK": 1
  },
  "SockTxDuration": 0.057274799,
  "SockTxBytes": 72,
  "SockTxBytesAcked": 74,
  "SockTxBytesRetrans": 0,
  "SockTxRTO": 51,
  "SockTxInterface": "4:enp5s0:0c:c4:7a:88:84:c2",
  "SockRxDataPackets": 1,
  "SockRxPackets": 4,
  "SockRxPacketsQueued": 0,
  "SockRxPacketsDrop": 0,
  "SockRxPacketsReorder": 0,
  "SockRxPacketsFrag": 0,
  "SockRxFlags": {
    "SYN-ACK": 1,
    "ACK": 1,
    "PSH-ACK": 1,
    "FIN-ACK": 1
  },
  "SockRxDuration": 0.057197399,
  "SockRxBytes": 342,
  "SockRxTTL": 185,
  "SockRTT": 0.000450625,
  "SockAge": 0.057344028,
  "App": "HTTP",
  "AppTxHttp": [{
    "_Timestamp": 0.000876589,
    "_Method": "GET",
    "_Url": "/",
    "_Version": "HTTP/1.1",
    "Host": "kflow.co",
    "User-Agent": "curl/7.88.1",
    "Accept": "*/*"
  }],
  "AppRxHttp": [{
    "_Timestamp": 0.056237469,
    "_Version": "HTTP/1.1",
    "_Status": 301,
    "_Reason": "Moved Permanently",
    "Date": "Thu, 04 Apr 2024 15:49:12 GMT",
    "Content-Type": "text/html; charset=utf-8",
    "Content-Length": "59",
    "Connection": "keep-alive",
    "Location": "https://kflowd.github.io",
    "Server": "ip-10-123-123-119.ec2.internal",
    "X-Request-Id": "5c331cbe-fbe1-40ea-ba2b-989691e687a0",
    "_Body": "<a href=\"https://kflowd.github.io\">Moved Permanently</a>."
  }]
}

</details> <details open> <summary> UNIX Socket + SYSLOG Record</summary>

{
  "InfoSequenceNumber": 4,
  "InfoTimestamp": "Mon, Sep 16 2024 14:30:19.409100980 UTC",
  "InfoMonitor": "socket",
  "InfoHostName": "dev.kflow.co",
  "InfoHostIP": "38.110.1.24",
  "InfoSystem": "Linux",
  "InfoKernel": "6.1.0-10-amd64",
  "InfoVersion": "kflowd-v0.9.11",
  "ProcParent": "cron",
  "Proc": "cron",
  "ProcVersion": "3.0pl1-137",
  "ProcUser": "root",
  "ProcGroup": "root",
  "ProcPPID": 990,
  "ProcPID": 2122368,
  "ProcTID": 2122368,
  "ProcUID": 0,
  "ProcGID": 0,
  "ProcAge": 18.100170007,
  "SockRole": "CLIENT",
  "SockAddress": "/run/systemd/journal/dev-log",
  "SockFamily": "AF_UNIX",
  "SockTxBytes": 192,
  "SockAge": 18.095146325,
  "App": "SYSLOG",
  "AppTxSyslog": [{
    "Facility": "Security/Authorization",
    "Severity": "Notice (5)",
    "Priority": 85,
    "Version": 0,
    "Timestamp": "Sep 16 14:30:01",
    "Appname