Nfstream

---

[NFStream][repo] is a multiplatform Python framework providing fast, flexible, and expressive data structures designed to make working with online or offline network data easy and intuitive. It aims to be Python's fundamental high-level building block for doing practical, real-world network flow data analysis. Additionally, it has the broader goal of becoming a unifying network data analytics framework for researchers providing data reproducibility across experiments.

<table> <tr> <td><b>Live Notebook</b></td> <td> <a href="https://mybinder.org/v2/gh/nfstream/nfstream-tutorials/master?filepath=demo_notebook.ipynb"> <img src="https://img.shields.io/badge/notebook-launch-blue?logo=jupyter&style=for-the-badge" alt="live notebook" /> </a> </td> </tr> <tr> <td><b>Project Website</b></td> <td> <a href="https://www.nfstream.org/"> <img src="https://img.shields.io/website?down_color=red&down_message=down&label=nfstream.org&logo=github&up_color=blue&up_message=up&url=https%3A%2F%2Fnfstream.org%2F&style=for-the-badge" alt="website" /> </a> </td> </tr> <tr> <td><b>Discussion Channel</b></td> <td> <a href="https://gitter.im/nfstream/community"> <img src="https://img.shields.io/badge/chat-on%20gitter-blue?color=blue&logo=gitter&style=for-the-badge" alt="Gitter" /> </a> </td> </tr> <tr> <td><b>Latest Release</b></td> <td> <a href="https://pypi.python.org/pypi/nfstream"> <img src="https://img.shields.io/pypi/v/nfstream.svg?logo=pypi&style=for-the-badge" alt="latest release" /> </a> </td> </tr> <tr> <td><b>Supported Versions</b></td> <td> <a href="https://pypi.org/project/nfstream/"> <img src="https://img.shields.io/pypi/pyversions/nfstream?logo=python&style=for-the-badge" alt="python3" /> </a> <a href="https://pypi.org/project/nfstream/"> <img src="https://img.shields.io/badge/pypy-3.11-blue?logo=pypy&style=for-the-badge" alt="pypy3" /> </a> </td> </tr> <tr> <td><b>Project License</b></td> <td> <a href="https://github.com/nfstream/nfstream/blob/master/LICENSE"> <img src="https://img.shields.io/pypi/l/nfstream?logo=gnu&style=for-the-badge&color=blue" alt="License" /> </a> </td> </tr> <tr> <td><b>Continuous Integration</b></td> <td> <a href="https://github.com/nfstream/nfstream/actions/workflows/build_test_linux.yml"> <img src="https://img.shields.io/github/actions/workflow/status/nfstream/nfstream/build_test_linux.yml?branch=master&logo=linux&style=for-the-badge&label=linux" alt="Linux WorkFlows" /> </a> <a href="https://github.com/nfstream/nfstream/actions/workflows/build_test_macos.yml"> <img src="https://img.shields.io/github/actions/workflow/status/nfstream/nfstream/build_test_macos.yml?branch=master&logo=apple&style=for-the-badge&label=macos" alt="MacOS WorkFlows" /> </a> <a href="https://github.com/nfstream/nfstream/actions/workflows/build_test_windows.yml"> <img src="https://img.shields.io/github/actions/workflow/status/nfstream/nfstream/build_test_windows.yml?branch=master&logo=windows&style=for-the-badge&label=windows" alt="Windows WorkFlows" /> </a> </td> </tr> <tr> <td><b>Code Quality</b></td> <td> <a href="https://oss-fuzz-build-logs.storage.googleapis.com/index.html#nfstream"> <img src="https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fnfstream%2Foss-fuzz-status-endpoint%2Fmain%2Fstatus.json" alt="Coverage" /> </a> <a href="https://codecov.io/gh/nfstream/nfstream/"> <img src="https://img.shields.io/codecov/c/github/nfstream/nfstream?color=brightgreen&logo=codecov&style=for-the-badge" alt="Fuzzing" /> </a> <a href="https://www.codefactor.io/repository/github/nfstream/nfstream"> <img src="https://img.shields.io/codefactor/grade/github/nfstream/nfstream?label=codefactor%3A%20Python%2C%20C&logo=codefactor&style=for-the-badge&logoWidth=18)" alt="Quality" /> </a> </td> </tr> </table>

Table of Contents

• Table of Contents
• Main Features
• How to get it?
• How to use it?
  • Encrypted application identification and metadata extraction
  • System visibility
  • Post-mortem statistical flow features extraction
  • Early statistical flow features extraction
  • Pandas export interface
  • CSV export interface
  • Extending NFStream
  • Machine Learning models training and deployment
    • Training the model
    • ML powered streamer on live traffic
• Building from sources
• Contributing
• Ethics
• Credits
  • Citation
  • Authors
  • Supporting organizations
• Publications that use NFStream
• License

Main Features

• Performance: NFStream is designed to be fast: [AF_PACKET_V3/FANOUT][packet] on Linux, multiprocessing, native [CFFI based][cffi] computation engine, and [PyPy][pypy] full support.
• Encrypted layer-7 visibility: NFStream deep packet inspection is based on [nDPI][ndpi]. It allows NFStream to perform [reliable][reliable] encrypted applications identification and metadata fingerprinting (e.g. TLS, SSH, DHCP, HTTP).
• System visibility: NFStream probes the monitored system's kernel to obtain information on open Internet sockets and collects guaranteed ground-truth (process name, PID, etc.) at the application level.
• Statistical features extraction: NFStream provides state of the art of flow-based statistical feature extraction. It includes post-mortem statistical features (e.g., minimum, mean, standard deviation, and maximum of packet size and inter-arrival time) and early flow features (e.g. sequence of first n packets sizes, inter-arrival times, and directions).
• Flexibility: NFStream is easily extensible using [NFPlugins][nfplugin]. It allows the creation of a new flow feature within a few lines of Python.
• Machine Learning oriented: NFStream aims to make Machine Learning Approaches for network traffic management reproducible and deployable. By using NFStream as a common framework, researchers ensure that models are trained using the same feature computation logic, and thus, a fair comparison is possible. Moreover, trained models can be deployed and evaluated on live networks using [NFPlugins][nfplugin].

How to get it?

Binary installers for the latest released version are available on Pypi.

pip install nfstream

Windows Notes: NFStream does not include capture drivers on Windows (license restrictions). It is required to install [Npcap drivers][npcap] before installing NFStream. If Wireshark is already installed on Windows, then Npcap drivers are already installed, and you do not need to perform any additional action.

How to use it?

Encrypted application identification and metadata extraction

Dealing with a big pcap file and want to aggregate into labeled network flows? NFStream make this path easier in a few lines:

from nfstream import NFStreamer
# We display all streamer parameters with their default values.
# See documentation for detailed information about each parameter.
# https://www.nfstream.org/docs/api#nfstreamer
my_streamer = NFStreamer(source="facebook.pcap", # or live network interface
                         decode_tunnels=True,
                         bpf_filter=None,
                         promiscuous_mode=True,
                         snapshot_length=1536,
                         idle_timeout=120,
                         active_timeout=1800,
                         accounting_mode=0,
                         udps=None,
                         n_dissections=20,
                         statistical_analysis=False,
                         splt_analysis=0,
                         n_meters=0,
                         max_nflows=0,
                         performance_report=0,
                         system_visibility_mode=0,
                         system_visibility_poll_ms=100)
                         
for flow in my_streamer:
    print(flow)  # print it.

# See documentation for each feature detailed description.
# https://www.nfstream.org/docs/api#nflow
NFlow(id=0,
      expiration_id=0,
      src_ip='192.168.43.18',
      src_mac='30:52:cb:6c:9c:1b',
      src_oui='30:52:cb',
      src_port=52066,
      dst_ip='66.220.156.68',
      dst_mac='98:0c:82:d3:3c:7c',
      dst_oui='98:0c:82',
      dst_port=443,
      protocol=6,
      ip_version=4,
      vlan_id=0,
      tunnel_id=0,
      bidirectional_first_seen_ms=1472393122365,
      bidirectional_last_seen_ms=1472393123665,
      bidirectional_duration_ms=1300,
      bidirectional_packets=19,
      bidirectional_bytes=5745,
      src2dst_first_seen_ms=1472393122365,
      src2dst_last_seen_ms=1472393123408,
      src2dst_duration_ms=1043,
      src2dst_packets=9,
      src2dst_bytes=1345,
      dst2src_first_seen_ms=1472393122668,
      dst2src_last_seen_ms=1472393123665,
      dst2src_duration_ms=997,
      dst2src_packets=10,
      dst2src_bytes=4400,
      application_name='TLS.Facebook',
      application_category_name='SocialNetwork',
      application_is_guessed=0,
      application_confidence=4,
      requested_server_name='facebook.c