Prometheus Pushgateway

[ Docker Repository on Quay ][quay] [ Docker Pulls ][hub]

The Prometheus Pushgateway exists to allow ephemeral and batch jobs to expose their metrics to Prometheus. Since these kinds of jobs may not exist long enough to be scraped, they can instead push their metrics to a Pushgateway. The Pushgateway then exposes these metrics to Prometheus.

Non-goals

First of all, the Pushgateway is not capable of turning Prometheus into a push-based monitoring system. For a general description of use cases for the Pushgateway, please read When To Use The Pushgateway.

The Pushgateway is explicitly not an aggregator or distributed counter but rather a metrics cache. It does not have statsd-like semantics. The metrics pushed are exactly the same as you would present for scraping in a permanently running program. If you need distributed counting, you could either use the actual statsd in combination with the Prometheus statsd exporter, or have a look at the prom-aggregation-gateway. With more experience gathered, the Prometheus project might one day be able to provide a native solution, separate from or possibly even as part of the Pushgateway.

For machine-level metrics, the textfile collector of the Node exporter is usually more appropriate. The Pushgateway is intended for service-level metrics.

The Pushgateway is not an event store. While you can use Prometheus as a data source for Grafana annotations, tracking something like release events has to happen with some event-logging framework.

A while ago, we decided to not implement a “timeout” or TTL for pushed metrics because almost all proposed use cases turned out to be anti-patterns we strongly discourage. You can follow a more recent discussion on the prometheus-developers mailing list.

Run it

Download binary releases for your platform from the release page and unpack the tarball.

If you want to compile yourself from the sources, you need a working Go setup. Then use the provided Makefile (type make).

For the most basic setup, just start the binary. To change the address to listen on, use the --web.listen-address flag (e.g. "0.0.0.0:9091" or ":9091"). By default, Pushgateway does not persist metrics. However, the --persistence.file flag allows you to specify a file in which the pushed metrics will be persisted (so that they survive restarts of the Pushgateway).

Using Docker

You can deploy the Pushgateway using the prom/pushgateway Docker image.

For example:

docker pull prom/pushgateway

docker run -d -p 9091:9091 prom/pushgateway

Use it

Configure the Pushgateway as a target to scrape

The Pushgateway has to be configured as a target to scrape by Prometheus, using one of the usual methods. However, you should always set honor_labels: true in the scrape config (see below for a detailed explanation).

Libraries

Prometheus client libraries should have a feature to push the registered metrics to a Pushgateway. Usually, a Prometheus client passively presents metric for scraping by a Prometheus server. A client library that supports pushing has a push function, which needs to be called by the client code. It will then actively push the metrics to a Pushgateway, using the API described below.

Command line

Using the Prometheus text protocol, pushing metrics is so easy that no separate CLI is provided. Simply use a command-line HTTP tool like curl. Your favorite scripting language has most likely some built-in HTTP capabilities you can leverage here as well.

Note that in the text protocol, each line has to end with a line-feed character (aka 'LF' or '\n'). Ending a line in other ways, e.g. with 'CR' aka '\r', 'CRLF' aka '\r\n', or just the end of the packet, will result in a protocol error.

Pushed metrics are managed in groups, identified by a grouping key of any number of labels, of which the first must be the job label. The groups are easy to inspect via the web interface.

For implications of special characters in label values see the URL section below.

Examples:

Push a single sample into the group identified by {job="some_job"}:
```
  echo "some_metric 3.14" | curl --data-binary @- http://pushgateway.example.org:9091/metrics/job/some_job
```
Since no type information has been provided, some_metric will be of type untyped.

Push something more complex into the group identified by {job="some_job",instance="some_instance"}:

  cat <<EOF | curl --data-binary @- http://pushgateway.example.org:9091/metrics/job/some_job/instance/some_instance
  # TYPE some_metric counter
  some_metric{label="val1"} 42
  # TYPE another_metric gauge
  # HELP another_metric Just an example.
  another_metric 2398.283
  EOF

Note how type information and help strings are provided. Those lines are optional, but strongly encouraged for anything more complex.

Delete all metrics in the group identified by {job="some_job",instance="some_instance"}:

  curl -X DELETE http://pushgateway.example.org:9091/metrics/job/some_job/instance/some_instance

Delete all metrics in the group identified by {job="some_job"} (note that this does not include metrics in the {job="some_job",instance="some_instance"} group from the previous example, even if those metrics have the same job label):
```
  curl -X DELETE http://pushgateway.example.org:9091/metrics/job/some_job
  
```
Delete all metrics in all groups (requires to enable the admin API via the command line flag --web.enable-admin-api):
```
  curl -X PUT http://pushgateway.example.org:9091/api/v1/admin/wipe
```

Note for MS Windows users

MS Windows users can send HTTP requests to the Pushgateway using Powershell.

Example:

     Invoke-WebRequest -Uri http://localhost:9091/metrics/job/MyJob -Method POST -Body "Some_Metric 3.14`n"

Double quotes should be used to enclose the metric text, and the back tick (`) character should come before the line ending character (n) at the end.

About the job and instance labels

The Prometheus server will attach a job label and an instance label to each scraped metric. The value of the job label comes from the scrape configuration. When you configure the Pushgateway as a scrape target for your Prometheus server, you will probably pick a job name like pushgateway. The value of the instance label is automatically set to the host and port of the target scraped. Hence, all the metrics scraped from the Pushgateway will have the host and port of the Pushgateway as the instance label and a job label like pushgateway. The conflict with the job and instance labels you might have attached to the metrics pushed to the Pushgateway is solved by renaming those labels to exported_job and exported_instance.

However, this behavior is usually undesired when scraping a Pushgateway. Generally, you would like to retain the job and instance labels of the metrics pushed to the Pushgateway. That's why you have to set honor_labels: true in the scrape config for the Pushgateway. It enables the desired behavior. See the documentation for details.

This leaves us with the case where the metrics pushed to the Pushgateway do not feature an instance label. This case is quite common as the pushed metrics are often on a service level and therefore not related to a particular instance. Even with honor_labels: true, the Prometheus server will attach an instance label if no instance label has been set in the first place. Therefore, if a metric is pushed to the Pushgateway without an instance label (and without instance label in the grouping key, see below), the Pushgateway will export it with an empty instance label ({instance=""}), which is equivalent to having no instance label at all but prevents the server from attaching one.

About metric inconsistencies

The Pushgateway exposes all pushed metrics together with its own metrics via the same /metrics endpoint. (See the section about exposed metrics for details.) Therefore, all the metrics have to be consistent with each other: Metrics of the same name must have the same type, even if they are pushed to different groups, and there must be no duplicates, i.e. metrics with the same name and the exact same label pairs. Pushes that would lead to inconsistencies are rejected with status code 400.

Inconsistent help strings are tolerated, though. The Pushgateway will pick a winning help string and log about it at info level.

Legacy note: The help string of Pushgateway's own push_time_seconds metric has changed in v0.10.0. By using a persistence file, metrics pushed to a Pushgateway of an earlier version can make it into a Pushgateway of v0.10.0 or later. In this case, the above mentioned log message will show up. Once each previously pushed group has been deleted or received a new push, the log message will disappear.

The consistency check performed during a push is the same as it happens anyway d

Pushgateway

Install / Use

README