Nemea
System for network traffic analysis and anomaly detection.
Install / Use
/learn @CESNET/NemeaREADME
README outline
- Project status
- NEMEA System
- Dependencies
- Installation
- Vagrant
- Binary packages
- Source codes
- Packer
- Quick start and how to
- Further Information
- NEMEA Related publications
Project status
Travis CI build: 
NEMEA System
NEMEA (Network Measurements Analysis) system is a stream-wise, flow-based and modular detection system for network traffic analysis. It consists of many independent modules which are interconnected via communication interfaces and each of the modules has its own task. Communication between modules is done by message passing where the messages contain flow records, alerts, some statistics or preprocessed data.
Parts of the system
The following picture shows all important parts of the system.
- Modules - basic building blocks; separate system processes; receive stream of data on their input interfaces, process it and send another stream of data to their output interfaces; all modules are simply divided into two groups according to their task:
- Detectors (red) - detect some malicious traffic, e.g. DNS tunnel, DoS, scanning
- Modules (yellow) - export&storage of flow data, preprocess or postprocess the data (filter, aggregate, merge etc.)
- NEMEA Framework - set of libraries implementing features common for all modules
- TRAP (Traffic Analysis Platform) (blue) - implements communication interfaces and functions for sending/receiving the messages between interfaces
- UniRec (Unified Record) (orange) - implements efficient data format of the sent/received messages
- Common library (purple) - implements common algorithms and data structures used in modules
- Supervisor (green) - central management and monitoring tool of the NEMEA system. It takes care of running modules according to a specified configuration.
Repositories
The project is divided into four repositories added as submodules:
Dependencies
Building environment
- autoconf
- automake
- gcc
- gcc-c++
- libtool
- libxml2-devel
- libxml2-utils (contains xmllint on Debian)
- make
- pkg-config
Optional dependencies of modules and detectors
- rpm-build (build of RPM packages)
- libpcap (flow_meter)
- libnf (nfreader)
- libidn (blacklistfilter)
- bison and flex (unirecfilter)
How to install dependencies:
Debian/Ubuntu:
apt-get install -y gawk bc autoconf automake gcc g++ libtool libxml2-dev make pkg-config libpcap-dev libidn11-dev bison flex
RHEL/CentOS/Fedora:
yum install -y bc autoconf automake gcc gcc-c++ libtool libxml2-devel make pkg-config libpcap-devel libidn-devel bison flex
Note: Latest systems (e.g. Fedora) use dnf instead of yum.
Installation
There are three different ways of installation of the NEMEA system covered in this document: vagrant, binary packages and source codes.
Vagrant
To try the system "out-of-box", you can use Vagrant. For more information see ./vagrant/.
Binary packages
Latest RPM packages can be found in COPR: https://copr.fedorainfracloud.org/groups/g/CESNET/coprs/ The NEMEA repository is at https://copr.fedorainfracloud.org/coprs/g/CESNET/NEMEA/
The repository can be added trough dnf copr enable or by using the following commands (for CentOS/Fedora):
wget -O /etc/yum.repos.d/cesnet-nemea.repo https://copr.fedorainfracloud.org/coprs/g/CESNET/NEMEA/repo/epel-7/group_CESNET-NEMEA-epel-7.repo
rpm --import https://copr-be.cloud.fedoraproject.org/results/@CESNET/NEMEA/pubkey.gpg
After installation of the COPR repository, NEMEA can be installed as any other package (run as root/sudo):
yum install nemea
Note: Latest systems (e.g. Fedora) use dnf instead of yum.
For development purposes, there is nemea-framework-devel package that installs
all needed development files and docs. To install this package, also add the NEMEA-testing repository.
Currently, we do not have .deb packages (for Debian/Ubuntu/...) but we are working on it. Please follow installation from source codes
Source codes
The whole system is based on GNU/Autotools build system that makes dependency checking and building process much more easier.
To clone the NEMEA repositories, use:
git clone --recursive https://github.com/CESNET/nemea
After successful clone and dependencies installation (!), use:
./bootstrap.sh
that will create configure scripts and other needed files.
The configure script supplies various possibilities of
configuration and it uses some environmental variables that influence the build
and compilation process. For more information see:
./configure --help
We recommend to set paths according to the used operating system, e.g.:
./configure --enable-repobuild --prefix=/usr --bindir=/usr/bin/nemea --sysconfdir=/etc/nemea --libdir=/usr/lib64
After finishing ./configure, build process can be started by:
make
The make(1) tool has various parameters, to build the NEMEA package faster on multicore systems, we recommend to use parameter -j with the number of jobs that should be run in parallel.
When the compilation process ends without any error, the package can be installed
into paths that were set by configure. It is recommended NOT to change
target paths by passing variables directly to make(1).
The installation can be done by (usually it requires root / sudo):
make install
Congratulations, the whole NEMEA system should be installed right now... :-)
Quick start and how to
Try out NEMEA modules
Execute a module
NEMEA modules using have two implicit arguments. module -h for help
(optional) and module -i IFC_SPEC for communication interface (IFC)
specification. The -i parameter is mandatory for all NEMEA modules.
Module help -h
The example below shows part of help output of logger. It contains module's name, description, number of input and output IFC, modules parameters and TRAP library parameters (common for all modules).
TRAP module, libtrap version: 0.7.6 b258bb4
===========================================
Name: Logger
Inputs: variable
Outputs: 0
Description:
This module logs all incoming UniRec records to standard output or into a specified file. Each record
is written as one line containing values of its fields in human-readable format separated by chosen
delimiters (CSV format). If you use more than one input interface you have to specify output format by
parameter "-o".
Usage: logger [COMMON]... [OPTIONS]...
Parameters of module [OPTIONS]:
-------------------------------
-w --write <string> Write output to FILE instead of stdout (rewrite the file).
-a --append <string> Write output to FILE instead of stdout (append to the end).
-t --title Write names of fields on the first line.
-c --cut <uint32> Quit after N records are received, 0 can be useful in combination
with -t to print UniRec.
Common TRAP parameters [COMMON]:
--------------------------------
-h [trap,1] If no argument, print this message. If "trap" or 1 is given, print
TRAP help.
-i IFC_SPEC Specification of interface types and their parameters, see "-h trap"
(mandatory parameter).
-v Be verbose.
Environment variables that affects output:
------------------------------------------
LIBTRAP_OUTPUT_FORMAT If set to "json", information about module is printed in JSON format.
PAGER Show the help output in the set PAGER.
Interface specifier -i
The -i parameter with the interface specifier IFC_SPEC (module -i IFC_SPEC)
specifies modules interfaces - their types and parameters. The interface
specifier has the following format:
<IFC 1>,<IFC 2>,...,<IFC N>
where <IFC x> looks like
<type>:<par1>:<par2>:...:<parN>.
<type> can be one of the following: t - TCP socket (for remote
communication), u - UNIX socket (for local communication), b - blackhole to
drop all messages during sending, f - File IFC.
Interfaces are separated by , and their parameters are separated by :.
Input IFCs must be specified at first, output IFCs follow. Examples below show
Example:
module1 -i t:address:port1,t:port2
module1 uses TCP interfaces (for machine to machine communication). Let's assume it has one input and one output interface (number of input and output IFC
