Exometer - Erlang instrumentation package

Copyright (c) 2014 Basho Technologies, Inc. All Rights Reserved.

Version: Feb 1 2015 23:02:37

Authors: Ulf Wiger (ulf.wiger@feuerlabs.com), Magnus Feuer (magnus.feuer@feuerlabs.com).

NOTE: Exometer has been split into exometer_core, and exometer (as well as separate reporter applications). The latest monolithic version of Exometer is 1.1.

The Exometer package allows for easy and efficient instrumentation of Erlang code, allowing crucial data on system performance to be exported to a wide variety of monitoring systems.

Exometer comes with a set of pre-defined monitor components, and can be expanded with custom components to handle new types of Metrics, as well as integration with additional external systems such as databases, load balancers, etc.

This document gives a high level overview of the Exometer system. For details, please see the documentation for individual modules, starting with exometer.

Note the section on Dependency Management for how to deal with optional packages, both users and developers.

<a name="Table_of_Content">Table of Content</a>

1. Concept and definitions
   1. Metric
   2. Data Point
   3. Metric Type
   4. Entry Callback
   5. Probe
   6. Caching
   7. Subscriptions and Reporters
2. Built-in entries and probes
   1. counter (exometer native)
   2. fast_counter (exometer native)
   3. gauge (exometer native)
   4. exometer_histogram (probe)
   5. exometer_uniform (probe)
   6. exometer_spiral (probe)
   7. exometer_folsom [entry]
   8. exometer_function [entry]
3. Built in Reporters
   1. exometer_report_graphite
   2. exometer_report_opentsdb
   3. exometer_report_amqp
   4. exometer_report_snmp
4. Instrumenting Erlang code
   1. Exometer Start
   2. Creating metrics
   3. Deleting metrics
   4. Setting metric values
   5. Retrieving metric values
   6. Setting up subscriptions
   7. Set metric options
5. Configuring Exometer
   1. Configuring type - entry maps
   2. Configuring statically defined entries
   3. Configuring static subscriptions
   4. Configuring reporter plugins
   5. Configuring opentsdb reporter
   6. Configuring amqp reporter
   7. Configuring graphite reporter
   8. Configuring snmp reporter
6. Creating custom exometer entries
7. Creating custom probes
8. Creating custom reporter plugins
9. Dependency management

<a name="Concepts_and_Definitions">Concepts and Definitions</a>

Exometer introduces a number of concepts and definitions used throughout the documentation and the code.

<a name="Metric">Metric</a>

A metric is a specific measurement sampled inside an Erlang system and then reported to the Exometer system. An example metric would be "transactions_per_second", or "memory_usage".

Metrics are identified by a list of terms, such as given below:

[ xml_front_end, parser, file_size ]

A metric is created through a call by the code to be instrumented to exometer:new(). Once created, the metric can be updated through exometer:update(), or on its own initiative through the exometer_probe:sample behavior implementation.

<a name="Data_Point">Data Point</a>

Each metric can consist of multiple data points, where each point has a specific value.

A typical example of data points would be a transactions_per_second (tps) metric, usually stored as a histogram covering the last couple of minutes of tps samples. Such a histogram would host multiple values, such as min, max, median, mean, 50_percentile, 75_percentile, etc.

It is up to the type of the metric, and the data probe backing that type (see below), to specify which data points are available under the given metric.

<a name="Metric_Type">Metric Type</a>

The type of a metric, specified when the metric is created through exometer:new(), determines which exometer_entry callback to use.

The link between the type and the entry to use is configured through the exometer_admin module, and its associated exometer defaults configuration data.

The metric type, in other words, is mainly used to map a metric to a configurable exometer_entry callback, but it can also be referenced in queries using exometer:select/1. An entry callback can also support multiple types (the type is provided as an argument in the callback functions).

Exometer provides default mappings for a number of metric types. It is possible to select different callbacks for each metric instance, as well as modify metrics using callback-specific options. Please see Configuring type - entry maps for details on how to do this.

<a name="Entry_Callback">Entry Callback</a>

An exometer entry callback will receive values reported to a metric through the exometer:update() call and compile it into one or more data points. The entry callback can either be a counter (implemented natively in exometer), or a more complex statistical analysis such as a uniform distribution or a regular histogram.

The various outputs from these entries are reported as data points under the given metric.

An entry can also interface external analytics packages. exometer_folsom, for example, integrates with the folsom_metrics package found at https://github.com/boundary/folsom.

<a name="Probe">Probe</a>

Probes are a further specialization of exometer entries that run in their own Erlang processes and have their own state (like a gen_server). A probe is implemented through the exometer_probe behavior.

A probe can be used if independent monitoring is neede