High-rate Delay Tolerant Network

Overview

Delay Tolerant Networking (DTN) has been identified as a key technology to facilitate the development and growth of future space networks. DTN is an overlay network that uses the bundle protocol to connect once disparate one-to-one links. Bundles are the primary unit of data in a DTN, and can be of essentially any size. Existing DTN implementations have operated in constrained environments with limited resources resulting in low data speeds, and cannot utilize more than a fraction of available system capacity. However, as various technologies have advanced, data transfer rates and efficiency have also advanced. To date, most known implementations of DTN have been designed to operate on spacecraft.

High-rate Delay Tolerant Networking (HDTN) takes advantage of modern hardware platforms to substantially reduce latency and improve throughput compared to today’s DTN operations. HDTN maintains compatibility with existing deployments of DTN that conform to IETF RFC 5050. At the same time, HDTN defines a new data format better suited to higher-rate operation. It defines and adopts a massively parallel pipelined and message-oriented architecture, allowing the system to scale gracefully as its resources increase. HDTN’s architecture also supports hooks to replace various processing pipeline elements with specialized hardware accelerators. This offers improved Size, Weight, and Power (SWaP) characteristics while reducing development complexity and cost.

For more information, refer to the relevant documents attached to the releases. https://github.com/nasa/HDTN/releases

Architecture

HDTN is written in C++, and is designed to be modular. These modules include:

• Ingress - Processes incoming bundles.
• Storage - Stores bundles to disk.
• Router - Calculates the next hop for the bundle and sets links up/down based on contact plan.
• Egress - Forwards bundles to the proper outduct and next hop.
• Telemetry Command Interface - Web interface that displays the operations and data for HDTN.

Build Environment

Tested Platforms

• Linux
  • Ubuntu 20.04.2 LTS
  • Debian 10.13
  • Oracle Linux Server 8
  • Ubuntu 20.04.2 LTS (ARM64)
• Windows
  • Windows 11 (64-bit)
• macOS
  • Apple Silicon M2 on Sonoma
  • Intel x64 on Sonoma
• FreeBSD 14.3 Intel x64
• OpenBSD 7.7 Intel x64
• Raspbian

Dependencies

HDTN build environment requires:

• CMake version 3.12 minimum
• Boost library version 1.66.0 minimum, version 1.69.0 for TCPCLv4 TLS version 1.3 support. Version 1.86 is the maximum version supported.
• ZeroMQ (tested with version 4.3.4)
• OpenSSL (minimum version 1.1.1). If OpenSSL is not available, disable OpenSSL support via the CMake cache variable ENABLE_OPENSSL_SUPPORT:BOOL
• On Linux: gcc minimum version 9.3.0
• On Windows: see the Windows Build Instructions

These can be installed on Linux with:

• On Ubuntu

sudo apt-get install cmake build-essential libzmq3-dev libboost-dev libboost-all-dev openssl libssl-dev

• On RHel

sudo dnf install epel-release
sudo dnf install cmake boost-devel zeromq zeromq-devel gcc-c++ libstdc++-devel

macOS Dependencies

• ZeroMQ

On macOS, ZeroMQ needs to be built from source and installed in the /usr/local directory which can be done with:

sudo git clone https://github.com/zeromq/libzmq /usr/local/libzmq
cd /usr/local/libzmq
git checkout v4.3.4
sudo mkdir build
sudo chmod -R 777 .
cd build
cmake ..
make -j4
make install

• OpenSSL and gnutls

First, install Homebrew, which is a package manager for macOS:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Finally, install OpenSSL and gnutls using Homebrew:

brew install openssl gnutls

Known issue

• Ubuntu distributions may install an older CMake version that is not compatible
• Some processors may not support hardware acceleration or the RDSEED instruction, both ON by default in the cmake file

Notes on C++ Standard Version

HDTN build environment sets by default the CMake cache variable HDTN_TRY_USE_CPP17 to ON.

• If set to ON, CMake will test if the compiler supports C++17. If the test fails, CMake will fall back to C++11 and compile HDTN with C++11. If the test passes, CMake will compile HDTN with C++17.
• If set to OFF, CMake will compile HDTN with C++11.
• With Visual Studio 2017 version 15.7 and later versions, if C++17 is enabled, CMake will set the compile option /Zc:__cplusplus to make the __cplusplus preprocessor macro accurate for Visual Studio. (Otherwise, without this compile option, by default, Visual Studio always returns the value 199711L for the __cplusplus preprocessor macro.)
• Throughout the HDTN codebase, C++17 optimizations exist, usually within a #if(__cplusplus >= 201703L) block.

Optional X86 Hardware Acceleration

HDTN build environment sets by default two CMake cache variables to ON: USE_X86_HARDWARE_ACCELERATION and LTP_RNG_USE_RDSEED.

• If you are building natively (i.e. not cross-compiling) then the HDTN CMake build environment will check the processor's CPU instruction set as well as the compiler to determine which HDTN hardware accelerated functions will both build and run on the native host. CMake will automatically set various compiler defines to enable any supported HDTN hardware accelerated features.
• If you are cross-compiling then the HDTN CMake build environment will check the compiler only to determine if the HDTN hardware accelerated functions will build. It is up to the user to determine if the target processor will support/run those instructons. CMake will automatically set various compiler defines to enable any supported HDTN hardware accelerated features if and only if they compile.
• Hardware accelerated functions can be turned off by setting USE_X86_HARDWARE_ACCELERATION and/or LTP_RNG_USE_RDSEED to OFF in the CMakeCache.txt.
• If you are building for ARM or any non X86-64 platform, USE_X86_HARDWARE_ACCELERATION and LTP_RNG_USE_RDSEED must be set to OFF.

If USE_X86_HARDWARE_ACCELERATION is turned on, then some or all of the following various features will be enabled if CMake finds support for these CPU instructions:

• Fast SDNV encode/decode (BPv6, TCPCLv3, and LTP) requires SSE, SSE2, SSE3, SSSE3, SSE4.1, POPCNT, BMI1, and BMI2.
• Fast batch 32-byte SDNV decode (not yet implemented into HDTN but available in the common/util/Sdnv library) requires AVX, AVX2, and the above "Fast SDNV" support.
• Fast CBOR encode/decode (BPv7) requires SSE and SSE2.
• Some optimized loads and stores for TCPCLv4 requires SSE and SSE2.
• Fast CRC32C (BPv7 and a storage hash function) requires SSE4.2.
• The HDTN storage controller will use BITTEST if available. If BITTEST is not available, it will use ANDN if BMI1 is available.

If LTP_RNG_USE_RDSEED is turned on, this feature will be enabled if CMake finds support for this CPU instruction:

• An additional source of randomness for LTP's random number generator requires RDSEED. You may choose to disable this feature for potentially faster LTP performance even if the CPU supports it.

Storage Capacity Compilation Parameters

HDTN build environment sets by default two CMake cache variables: STORAGE_SEGMENT_ID_SIZE_BITS and STORAGE_SEGMENT_SIZE_MULTIPLE_OF_4KB.

• The flag STORAGE_SEGMENT_ID_SIZE_BITS must be set to 32 (default and recommended) or 64. It determines the size/type of the storage module's segment_id_t Using 32-bit for this type will significantly decrease memory usage (by about half).
  • If this value is 32, The formula for the max segments (S) is given by S = min(UINT32_MAX, 64^6) = ~4.3 Billion segments since segment_id_t is a uint32_t. A segment allocator using 4.3 Billion segments uses about 533 MByte RAM), and multiplying by the minimum 4KB block size gives ~17TB bundle storage capacity. Make sure to appropriately set the totalStorageCapacityBytes variable in the HDTN json config so that only the required amount of memory is used for the segment allocator.
  • If this value is 64, The formula for the max segments (S) is given by S = min(UINT64_MAX, 64^6) = ~68.7 Billion segments since segment_id_t is a uint64_t. Using a segment allocator with 68.7 Billion segments, when multiplying by the minimum 4KB block size gives ~281TB bundle storage capacity.
• The flag STORAGE_SEGMENT_SIZE_MULTIPLE_OF_4KB must be set to an integer of at least 1 (default is 1 and recommended). It determines the minimum increment of bundle storage based on the standard block size of 4096 bytes. For example:
  • If STORAGE_SEGMENT_SIZE_MULTIPLE_OF_4KB = 1, then a 4KB * 1 = 4KB block size is used. A bundle size of 1KB would require 4KB of storage. A bundle size of 6KB would require 8KB of storage.
  • If STORAGE_SEGMENT_SIZE_MULTIPLE_OF_4KB = 2, then a 4KB * 2 = 8KB block size is used. A bundle size of 1KB would require 8KB of storage. A bundle size of 6KB would require 8KB of storage. A bundle size of 9KB would require 16KB of storage. If STORAGE_SEGMENT_ID_SIZE_BITS=32, then bundle storage capacity could potentially be doubled from ~17TB to ~34TB.

For more information on how the storage works, see module/storage/doc/storage.pptx in this repository.

Logging Compilation Parameters

Logging is controlled by CMake cache variables:

• LOG_LEVEL_TYPE controls which messages are logged. The options, from most verbose to least verbose, are TRACE, DEBUG, INFO, WARNING, ERROR, FATAL, and NONE. All log statements using a level more verbose than the provided level will be compiled out of the application. The default value is INFO.
• LOG_TO_CONSOLE controls whether log messages are sent to the console. The default value is ON.
• LOG_TO_ERROR_FILE controls whether all