In-Situ OAM (IOAM)

In-situ OAM (sometimes also referred to as inband OAM, which was the term originally used before IETF chose "in-situ OAM" for the technology) is a technology to record operational information in the packet while the packet traverses a path between two points in the network. In-situ OAM is to complement current out-of-band OAM (sometimes also called "active" OAM) mechanisms based on ICMP or other types of probe packets.

Overview

"In-situ" OAM describes an approach to record OAM and telemetry information within the data packet while the data packet traverses a network or a particular network domain. The term "in-situ" refers to the fact that the OAM and telemetry data is carried within data packets rather than being sent within packets specifically dedicated to OAM. In-situ OAM mechanisms, which are sometimes also referred to as embedded network telemetry are a current topic of discussion. In-band network telemetry has been defined for [P4]. The SPUD prototype [SPUD] uses a similar logic that allows network devices on the path between endpoints to participate explicitly in the tube outside the end-to-end context. Even the IPv4 route-record option defined in [RFC0791] can be considered an in-situ OAM mechanism. In-situ OAM complements "out-of-band" mechanisms such as ping or traceroute, or more recent active probing mechanisms, as described in [I-D.lapukhov-dataplane-probe]. In-situ OAM mechanisms can be leveraged where current out-of-band mechanisms do not apply or do not offer the desired characteristics or requirements, such as proving that a certain set of traffic takes a pre-defined path, strict congruency is desired, checking service level agreements for the live data traffic, detailed statistics on traffic distribution paths in networks that distribute traffic across multiple paths, or scenarios where probe traffic is potentially handled differently from regular data traffic by the network devices. [RFC7276] presents an overview of OAM tools.

Compared to probably the most basic example of "in-situ OAM" which is IPv4 route recording [RFC0791], an in-situ OAM approach has the following capabilities:

• A flexible data format to allow different types of information to be captured as part of an in-situ OAM operation, including not only path tracing information, but additional operational and telemetry information such as timestamps, sequence numbers, or even generic data such as queue size, geo-location of the node that forwarded the packet, etc.

• A data format to express node as well as link identifiers to record the path a packet takes with a fixed amount of added data.

• The ability to detect whether any nodes were skipped while recording in-situ OAM information (i.e., in-situ OAM is not supported or not enabled on those nodes).

• The ability to actively process information in the packet, for example to prove in a cryptographically secure way that a packet really took a pre-defined path using some traffic steering method such as service chaining or traffic engineering.

• The ability to include OAM data beyond simple path information, such as timestamps or even generic data of a particular use case.

• The ability to include OAM data in various different transport protocols.

A detailed description in-situ OAM concepts, capabilities, data-formats and transport encapsulations can be found in the following IETF internet drafts:

• Requirements for In-band OAM discusses the motivation and requirements for including specific operational and telemetry information into data packets while the data packet traverses a path between two points in the network.

• Data Formats for In-Situ OAM discusses the data types and data formats for in-situ OAM data records.

• Encapsulations for IOAM data. These drafts describe how IOAM data fields are encapsulated in "parent" protocols:

  • NSH encapsulation for IOAM data
  • Geneve encapsulation for IOAM data
  • VXLAN-GPE encapsulation for IOAM data
  • IPv6 encapsulation for IOAM data
  • SRv6 encapsulation for IOAM data
  • Encapsulations for protocols which use Ethertype for next protocol (e.g. GRE, Geneve). Note: For IPv4, it is suggested to use a GRE header sequenced in with an IOAM header to carry IOAM data fields.

• Proof of Transit defines mechanisms to securely prove that traffic transited the defined path. Several technologies such as traffic engineering, service function chaining, or policy based routing, are used to steer traffic through a specific, user-defined path. The mechanisms described in this document allow to securely verify whether all packets traversed all those nodes of a given path that they are supposed to visit.

• Export of IOAM data in raw format describes how IOAM information can be exported in raw, i.e. uninterpreted, format from network devices to systems, such as monitoring or analytics systems using IPFIX.

A wide variety of use-cases can leverage IOAM:

• Service/Quality Assurance - Fabric OAM
  • Prove traffic SLAs, as opposed to probe-traffic SLAs; Overlay/Underlay
  • Service/Path Verification (Proof of Transit) - prove that traffic follows a pre-defined path
• Micro-Service/NFV deployments
  • Smart service selection based on network criteria - "M-Anycast" (intelligent micro-service selection and load-balancing): https://github.com/CiscoDevNet/iOAM/tree/master/M-Anycast. For the VPP implementation of m-anycast, see https://docs.fd.io/vpp/17.04/ioam_manycast_doc.html
• Operations Support - Fabric Visibility
  • Network Fault Detection and Fault Isolation through efficient network probing: By using IOAM's loopback option an issue can be identified within a single packet roundtrip time.
  • Path Tracing - debug ECMP, brown-outs, network delays
  • Derive Traffic Matrix
  • Custom/Service Level Telemetry

Code

Dataplane implementation in FD.io/VPP

• The in-situ OAM dataplane is implemented as a plugin in VPP: https://git.fd.io/cgit/vpp/tree/plugins/ioam-plugin.
• Documentation for the in-situ OAM in VPP:
  • Overview user guide: https://docs.fd.io/vpp/16.12/md_plugins_ioam-plugin_ioam_Readme.html
  • Command line references:
    • https://docs.fd.io/vpp/17.04/ioam_plugin_doc.html
    • https://docs.fd.io/vpp/16.12/plugins_ioam-plugin_ioam_encap.html
    • https://docs.fd.io/vpp/16.12/plugins_ioam-plugin_ioam_export.html
    • https://docs.fd.io/vpp/16.12/plugins_ioam-plugin_ioam_lib-pot.html
    • https://docs.fd.io/vpp/16.12/plugins_ioam-plugin_ioam_lib-trace.html
• FD.io wiki on in-situ OAM configuration: https://wiki.fd.io/view/VPP/Command-line_Interface_(CLI)_Guide#Inline_IPv6_OAM_Commands.

Dataplane implementation in the Linux Kernel

The University of Liege in Belgium created an IOAM implementation for the Linux Kernel: https://github.com/IurmanJ/kernel_ipv6_ioam

Dataplane implementation in Cisco IOS

The dataplane implementation in Cisco IOS is focused on IPv6 only.

• Documentation for in-situ OAM is found in the In-band OAM for IPv6 guide of the "IPv6 Network Management Configuration Guide, Cisco IOS Release 15M&T".

• The IOS software can be downloaded from here. IPv6 in-situ OAM is supported on Cisco 1900/2900/3900/3900e Integrated Services Routers and vIOS on Cisco [VIRL] (http://virl.cisco.com/).

• To configure Proof-of-Transit for IOS, a series of configuration parameters are needed. To help with computing the appropriate values, you can use the scripts provided here, see https://github.com/CiscoDevNet/iOAM/tree/master/scripts/config_generator. Note that those apply mostly for the IOS dataplane application. For VPP, one can use the App for OpenDaylight.

In-band OAM Controller Application in OpenDaylight

In-band OAM is reflected in two applications within OpenDaylight:

• SFC: In-band OAM "Proof of Transit" can be used as part of OpenDaylight Service Function Chaining (SFC). The extensions to ODL SFC enable ODL to serve POT control data (secrets etc.) required for POT. Full support is expected as part of the OpenDaylight Carbon release.

  • The following features are supported.
    • Utilizes Java-based libraries to generate iOAM parameters.
    • REST/Yang based APIs for north-bound to configure iOAM via OSS like postman or via programmatic triggers.
    • Augments and enhances base SFC application.
    • NETCONF/Yang based APIs for south-bound.
    • Feature capability is split into two modules to be installed: sfc-pot and sfc-pot-netconf-renderer.
  • The following are the feature commits.
    • https://git.opendaylight.org/gerrit/#/c/40669/
    • https://git.opendaylight.org/gerrit/#/c/48766/
    • https://git.opendaylight.org/gerrit/#/c/49636/

• Path-tracing: Configuration application to enable and control in-situ OAM tracing. The tracing application will be included in a future version of OpenDaylight as a separate module. For now, it is dependent on the SFC git (but not SFC functionality) and is available as below.

  • https://github.com/CiscoDevNet/iOAM/tree/master/sfc

In-band