snap <sup><sub>A powerful telemetry framework</sub></sup>

Here are blog posts related to snap by the team:

Nickapedia.com - What if collecting data center telemetry was a snap?

Overview
Getting Started

In Active Development
System Requirements
Installation
Running snap
Load Plugins
Running Tasks
Building Tasks
Plugin Catalog

Documentation

Examples
Roadmap

Community Support
Contributing

Author a Plugin

License
Contributors

Initial Authors
Maintainers

Thank You

Overview

snap is a framework for enabling the gathering of telemetry from systems. The goals of this project are to:

Empower systems to expose a consistent set of telemetry data
Simplify telemetry ingestion across ubiquitous storage systems
Improve the deployment model, packaging and flexibility for collecting telemetry
Allow flexible processing of telemetry data on agent (e.g. machine learning)
Provide powerful clustered control of telemetry workflows across small or large clusters

The key features of snap are:

Plugin Architecture: snap has a simple and smart modular design. The three types of plugins (collectors, processors, and publishers) allow snap to mix and match functionality based on user need. All plugins are designed with versioning, signing and deployment at scale in mind. The open plugin model allows for loading built-in, community, or proprietary plugins into snap.
- Collectors - Collectors consume telemetry data. Collectors are built-in plugins for leveraging existing telemetry solutions (Facter, CollectD, Ohai) as well as specific plugins for consuming Intel telemetry (Node, DCM, NIC, Disk) and can reach into new architectures through additional plugins (see Plugin Authoring below). Telemetry data is organized into a dynamically generated catalog of available data points.
- Processors - Extensible workflow injection. Convert telemetry into another data model for consumption by existing consumption systems (like OpenStack Ceilometer). Allows encryption of all or part of the telemetry payload before publishing. Inject remote queries into workflow for tokens, filtering, or other external calls. Implement filtering at an agent level reducing injection load on telemetry consumer.
- Publishers - Store telemetry into a wide array of systems. snap decouples the collection of telemetry from the implementation of where to send it. snap comes with a large library of publisher plugins that allow exposure to telemetry analytics systems both custom and common. This flexibility allows snap to be valuable to open source and commercial ecosystems alike by writing a publisher for their architectures.
Dynamic Updates: snap is designed to evolve. Each scheduled workflow automatically uses the most mature plugin for that step, unless the collection is pinned to a specific version (e.g. get /intel/psutil/load/load1/v1). Loading a new plugin automatically upgrades running workflows in tasks. Load plugins dynamically, without a restart to the service or server. This dynamically extends the metric catalog when loaded, giving access to new measurements immediately. Swapping a newer version plugin for an old one in a safe transaction. All of these behaviors allow for simple and secure bug fixes, security patching, and improving accuracy in production.
snap tribe: snap is designed for ease of administration. With snap tribe, nodes work in groups (aka tribes). Requests are made through agreement- or task-based node groups, designed as a scalable gossip-based node-to-node communication process. Administrators can control all snap nodes in a tribe agreement by messaging just one of them. There is auto-discovery of new nodes and import of tasks and plugins from nodes within a given tribe. It is cluster configuration management made simple.

Some additionally important notes about how snap works:

Multiple management modules including: CLI (snapctl) and REST API (each of which can be turned on or off)
Secure validation occurs via plugin signing, SSL encryption for APIs and payload encryption for communication between components
CLI control from Linux or OS X

snap is not intended to:

Operate as an analytics platform: the intention is to allow plugins for feeding those platforms
Compete with existing metric/monitoring/telemetry agents: snap is simply a new option to use or reference

Getting Started

In Active Development

The master branch is used for new feature development. Our goal is to keep it in a deployable state. If you're looking for the most recent binary that is versioned, please see the latest release.

System Requirements

Snap deploys as a binary, which makes requirements quite simple. We've tested on a subset of Linux and OS X versions.

Installation

You can get the pre-built binaries for your OS and architecture at snap's GitHub Releases page. This isn't the comprehensive list of plugins, but they will help you get started. Right now, snap only supports Linux and OS X (Darwin).

Building snap

If you're looking for the bleeding edge of snap, you can build it by getting the master branch using go get github.com/intelsdi-x/snap. Otherwise you can just use the binaries. To build snap from source, you will need Golang >= 1.4 and GNU Make. For more on building snap check out BUILD_AND_TEST.md.

Running snap

Start a standalone snap agent (snapd):

If you cloned down the repository and aren't using just downloaded binaries, you can set the following. Also, follow the steps in BUILD_AND_TEST.md:

$ export SNAP_PATH=<snapDirectoryPath>/build

And then you can use $SNAP_PATH/bin for your <snapBinPath>.

$ <snapBinPath>/snapd --plugin-trust 0 --log-level 1

This will bring up a snap agent without requiring plugin signing and set the logging level to debug. snap's REST API will be listening on port 8181. To learn more about the snap agent and how to use it look at SNAPD.md and/or run <snapBinPath>/snapd -h.

Running snap in tribe (cluster) mode

Start the first node:

$ <snapBinPath>/snapd --tribe

All other nodes who join will need to select any existing member of the cluster.

$ <snapBinPath>/snapd --tribe-seed <ip or name of another tribe member>

Checkout the tribe doc for more info.

Load Plugins

snap gets its power from the use of plugins. The plugin catalog is a collection of all known plugins for snap.

Next, open a new shell and lets load a few of the demo plugins. You can do this via cURL, or snapctl, snap's CLI.

Using cURL

$ cd $SNAP_PATH
$ curl -X POST -F plugin=@plugin/snap-collector-mock1 http://localhost:8181/v1/plugins
$ curl -X POST -F plugin=@plugin/snap-processor-passthru http://localhost:8181/v1/plugins
$ curl -X POST -F plugin=@plugin/snap-publisher-file http://localhost:8181/v1/plugins

Or:

Using snapctl (snap CLI)

cd $SNAP_PATH
$ <snapBinPath>/snapctl plugin load plugin/snap-collector-mock1
$ <snapBinPath>/snapctl plugin load plugin/snap-processor-passthru
$ <snapBinPath>/snapctl plugin load plugin/snap-publisher-file

Let's look at what plugins we have loaded now:

$ <snapBinPath>/snapctl plugin list
NAME             VERSION         TYPE            SIGNED          STATUS          LOADED TIME
mock             1               collector       false           loaded          Tue, 17 Nov 2015 14:08:17 PST
passthru         1               processor       false           loaded          Tue, 17 Nov 2015 14:16:12 PST
file             3               publisher       false           loaded          Tue, 17 Nov 2015 14:16:19 PST

Running Tasks

Tasks can be in JSON or YAML format. Let's start one of the example tasks from the examples/tasks/ directory:

$ cd <snapDirectoryPath>
$ <snapBinPath>/snapctl task create -t examples/tasks/mock-file.yaml
Using task manifest to create task
Task created
ID: 8b9babad-b3bc-4a16

Snap

Install / Use

README