Feedgnuplot

=head1 TALK

I just gave a talk about this at L<SCaLE 17x|https://www.socallinuxexpo.org/scale/17x>. Here are the L<video of the talk|https://www.youtube.com/watch?v=Qvb_uNkFGNQ&t=12830s> and the L<"slides"|https://github.com/dkogan/talk-feedgnuplot-vnlog/blob/master/feedgnuplot-vnlog.org>.

=head1 NAME

feedgnuplot - General purpose pipe-oriented plotting tool

=head1 SYNOPSIS

Simple plotting of piped data:

$ seq 5 | awk '{print 2*$1, $1*$1}' 2 1 4 4 6 9 8 16 10 25

$ seq 5 | awk '{print 2*$1, $1*$1}' | feedgnuplot
--lines
--points
--title "Test plot"
--y2 1
--unset key
--unset grid

=for html <p><img src="documentation-header-plot.svg">

Simple real-time plotting example: plot how much data is received on the wlan0 network interface in bytes/second. This plot updates at 1Hz, and shows the last 10sec of history. The plot shown here is the final state of a sample run

$ while true; do sleep 1; cat /proc/net/dev; done
| gawk '/wlan0/ {if(b) {print $2-b; N++; fflush()} b=$2} N==15 {exit}'
| feedgnuplot
--lines
--title "wlan0 throughput"
--stream
--xlen 10
--ylabel 'Bytes/sec'
--xlabel seconds
--unset key
--unset grid

=for html <p><img src="documentation-header-network-throughput-plot.svg">

=head1 DESCRIPTION

This is a flexible, command-line-oriented frontend to Gnuplot. It creates plots from data coming in on STDIN or given in a filename passed on the commandline. Various data representations are supported, as is hardcopy output and streaming display of live data. For a tutorial and a gallery please see the guide at Lhttps://github.com/dkogan/feedgnuplot/blob/master/guide/guide.org

A simple example:

$ seq 5 | awk '{print 2*$1, $1*$1}' | feedgnuplot

You should see a plot with two curves. The C<awk> command generates some data to plot and the C<feedgnuplot> reads it in from STDIN and generates the plot. The C<awk> invocation is just an example; more interesting things would be plotted in normal usage. No commandline-options are required for the most basic plotting. Input parsing is flexible; every line need not have the same number of points. New curves will be created as needed.

The most commonly used functionality of gnuplot is supported directly by the script. Anything not directly supported can still be done with options such as C<--set>, C<--cmds> C<--style>, etc. Arbitrary gnuplot commands can be passed in with C<--cmds>. For example, to turn off the grid, you can pass in C<--cmds 'unset grid'>. Commands C<--set> and C<--unset> exists to provide nicer syntax, so this is equivalent to passing C<--unset grid>. As many of these options as needed can be passed in. To add arbitrary curve styles, use C<--style curveID extrastyle>. Pass these more than once to affect more than one curve.

To apply an extra style to I<all> the curves that lack an explicit C<--style>, pass in C<--styleall extrastyle>. In the most common case, the extra style is C<with something>. To support this more simply, you can pass in C<--with something> instead of C<--styleall 'with something'>. C<--styleall> and C<--with> are mutually exclusive. Furthermore any curve-specific C<--style> overrides the global C<--styleall> or C<--with> setting.

=head2 Data formats

By default, each value present in the incoming data represents a distinct data point, as demonstrated in the original example above (we had 10 numbers in the input and 10 points in the plot). If requested, the script supports more sophisticated interpretation of input data

=head3 Domain selection

If C<--domain> is passed in, the first value on each line of input is interpreted as the I<X>-value for the rest of the data on that line. Without C<--domain> the I<X>-value is the line number, and the first value on a line is a plain data point like the others. Default is C<--nodomain>. Thus the original example above produces 2 curves, with B<1,2,3,4,5> as the I<X>-values. If we run the same command with C<--domain>:

$ seq 5 | awk '{print 2*$1, $1*$1}' | feedgnuplot --domain

we get only 1 curve, with B<2,4,6,8,10> as the I<X>-values. As many points as desired can appear on a single line, but all points on a line are associated with the I<X>-value at the start of that line.

=head3 Curve indexing

We index the curves in one of 3 ways: sequentially, explicitly with a C<--dataid> or by C<--vnlog> headers.

By default, each column represents a separate curve. The first column (after any domain) is curve C<0>. The next one is curve C<1> and so on. This is fine unless sparse data is to be plotted. With the C<--dataid> option, each point is represented by 2 values: a string identifying the curve, and the value itself. If we add C<--dataid> to the original example:

$ seq 5 | awk '{print 2*$1, $1*$1}' | feedgnuplot --dataid --autolegend

we get 5 different curves with one point in each. The first column, as produced by C<awk>, is B<2,4,6,8,10>. These are interpreted as the IDs of the curves to be plotted.

If we're plotting C<vnlog> data (Lhttps://www.github.com/dkogan/vnlog) then we can get the curve IDs from the vnlog header. Vnlog is a trivial data format where lines starting with C<#> are comments and the first comment contains column labels. If we have such data, C<feedgnuplot --vnlog> can interpret these column labels if the C<vnlog> perl modules are available.

The C<--autolegend> option adds a legend using the given IDs to label the curves. The IDs need not be numbers; generic strings are accepted. As many points as desired can appear on a single line. C<--domain> can be used in conjunction with C<--dataid> or C<--vnlog>.

=head3 Multi-value style support

Depending on how gnuplot is plotting the data, more than one value may be needed to represent the range of a single point. Basic 2D plots have 2 numbers representing each point: 1 domain and 1 range. But if plotting with C<--circles>, for instance, then there's an extra range value: the radius. Many other gnuplot styles require more data: errorbars, variable colors (C<with points palette>), variable sizes (C<with points ps variable>), labels and so on. The feedgnuplot tool itself does not know about all these intricacies, but they can still be used, by specifying the specific style with C<--style>, and specifying how many values are needed for each point with any of C<--rangesizeall>, C<--tuplesizeall>, C<--rangesize>, C<--tuplesize>. These options are required I<only> for styles not explicitly supported by feedgnuplot; supported styles do the right thing automatically.

Specific example: if making a 2d plot of y error bars, the exact format can be queried by running C<gnuplot> and invoking C<help yerrorbars>. This tells us that there's a 3-column form: C<x y ydelta> and a 4-column form: C<x y ylow yhigh>. With 2d plots feedgnuplot will always output the 1-value domain C<x>, so the rangesize is 2 and 3 respectively. Thus the following are equivalent:

$ echo '1 2 0.3 2 3 0.4 3 4 0.5' | feedgnuplot --domain --rangesizeall 2 --with 'yerrorbars'

$ echo '1 2 0.3 2 3 0.4 3 4 0.5' | feedgnuplot --domain --tuplesizeall 3 --with 'yerrorbars'

$ echo '1 2 1.7 2.3 2 3 2.6 3.4 3 4 3.5 4.5' | feedgnuplot --domain --rangesizeall 3 --with 'yerrorbars'

=head3 3D data

To plot 3D data, pass in C<--3d>. C<--domain> MUST be given when plotting 3D data to avoid domain ambiguity. If 3D data is being plotted, there are by definition 2 domain values instead of one (I<Z> as a function of I<X> and I<Y> instead of I<Y> as a function of I<X>). Thus the first 2 values on each line are interpreted as the domain instead of just 1. The rest of the processing happens the same way as before.

=head3 Time/date data

If the input data domain is a time/date, this can be interpreted with C<--timefmt>. This option takes a single argument: the format to use to parse the data. The format is documented in 'set timefmt' in gnuplot, although the common flags that C<strftime> understands are generally supported. The backslash sequences in the format are I<not> supported, so if you want a tab, put in a tab instead of \t. Whitespace in the format I<is> supported. When this flag is given, some other options act a little bit differently:

=over

=item

C<--xlen> and C<--binwidth> are I<integers> in seconds

=item

C<--xmin> and C<--xmax> I<must> use the format passed in to C<--timefmt>

=back

Using this option changes both the way the input is parsed I<and> the way the x-axis tics are labelled. Gnuplot tries to be intelligent in this labelling, but it doesn't always do what the user wants. The labelling can be controlled with the gnuplot C<set format> command, which takes the same type of format string as C<--timefmt>. Example:

$ sar 1 -1 | awk '$1 ~ /..:..:../ && $8 ~/^[0-9.]*$/ {print $1,$8; fflush()}' | feedgnuplot --stream --domain --lines --timefmt '%H:%M:%S' --set 'format x "%H:%M:%S"'

This plots the 'idle' CPU consumption against time.

Note that while gnuplot supports the time/date on any axis, I<feedgnuplot> currently supports it I<only> as the x-axis domain. This may change in the future.

=head3 'using' expressions

We just described how feedgnuplot parses its input data. When passing this data to gnuplot, each curve is sent independently. The domain appears in the leading columns followed by C<--rangesize> columns to complete each row. Without C<--domain>, feedgnuplot explicitly writes out sequential integers. gnuplot then knows how many values it has for each point, and it knows which style we're using, so it's able to interpret the data appropriately, and to make the correct plot.

As an example, if gnuplot is passed 2 columns of data, and it is plotting C<with points