Rstats

Author: Libor Spacek

Usage

This crate is written in 100% safe Rust.

Use in your source files any of the following structs, as and when needed:

use Rstats::{RE,RError,Params,TriangMat,MinMax};

and any of the following traits:

use Rstats::{Stats,Vecg,Vecu8,MutVecg,VecVec,VecVecg};

and any of the following auxiliary functions:

use Rstats::{
    fromop,sumn,tm_stat,unit_matrix,nodata_error,data_error,
    arith_error,other_error };

or just simply use everything:

use Rstats::*;

The latest (nightly) version is always available in the github repository Rstats. Sometimes it may be (only in some details) a little ahead of the crates.io release versions.

It is highly recommended to read and run tests.rs for examples of usage. To run all the tests, use a single thread in order not to print the results in confusing mixed-up order:

cargo test --release -- --test-threads=1 --nocapture

However, geometric_medians, which compares multithreading performance, should be run separately in multiple threads, as follows:

cargo test -r geometric_medians -- --nocapture

Alternatively, just to get a quick idea of the methods provided and their usage, read the output produced by an automated test run. There are test logs generated for each new push to the github repository. Click the latest (top) one, then Rstats and then Run cargo test ... The badge at the top of this document lights up green when all the tests have passed and clicking it gets you to these logs as well.

Any compilation errors arising out of rstats crate indicate most likely that some of the dependencies are out of date. Issuing cargo update command will usually fix this.

Introduction

Rstats has a small footprint. Only the best methods are implemented, primarily with Data Analysis and Machine Learning in mind. They include multidimensional (nd or 'hyperspace') analysis, i.e. characterising clouds of n points in space of d dimensions.

Several branches of mathematics: statistics, information theory, set theory and linear algebra are combined in this one consistent crate, based on the abstraction that they all operate on the same data objects (here Rust Vecs). The only difference being that an ordering of their components is sometimes assumed (in linear algebra, set theory) and sometimes it is not (in statistics, information theory, set theory).

Rstats begins with basic statistical measures, information measures, vector algebra and linear algebra. These provide self-contained tools for the multidimensional algorithms but they are also useful in their own right.

Non analytical (non parametric) statistics is preferred, whereby the 'random variables' are replaced by vectors of real data. Probabilities densities and other parameters are in preference obtained from the real data (pivotal quantity), not from some assumed distributions.

Linear algebra uses generic data structure Vec<Vec<T>> capable of representing irregular matrices.

Struct TriangMat is defined and used for symmetric, anti-symmetric, and triangular matrices, and their transposed versions, saving memory.

Our treatment of multidimensional sets of points is constructed from the first principles. Some original concepts, not found elsewhere, are defined and implemented here (see the next section).

Zero median vectors are generally preferred to commonly used zero mean vectors.

In n dimensions, many authors 'cheat' by using quasi medians (one dimensional (1d) medians along each axis). Quasi medians are a poor start to stable characterisation of multidimensional data. Also, they are actually slower to compute than our gm ( geometric median), as soon as the number of dimensions exceeds trivial numbers.

Specifically, all such 1d measures are sensitive to the choice of axis and thus are affected by their rotation.

In contrast, our methods based on gm are axis (rotation) independent. Also, they are more stable, as medians have a maximum possible breakdown point.

We compute geometric medians by our method gmedian and its parallel version par_gmedian in trait VecVec and their weighted versions wgmedian and par_wgmedian in trait VecVecg. It is mostly these efficient algorithms that make our new concepts described below practical.

Additional Documentation

For more detailed comments, plus some examples, see rstats in docs.rs. You may have to go directly to the modules source. These traits are implemented for existing 'out of this crate' rust Vec type and unfortunately rust docs do not display 'implementations on foreign types' very well.

New Concepts and their Definitions

• zero median points (or vectors) are obtained by moving the origin of the coordinate system to the median (in 1d), or to the gm (in nd). They are our alternative to the commonly used zero mean points, obtained by moving the origin to the arithmetic mean (in 1d) or to the arithmetic centroid (in nd).

• median correlation between two 1d sets of the same length.
  We define this correlation similarly to Pearson, as cosine of an angle between two normalised samples, interpreted as coordinate vectors. Pearson first normalises each set by subtracting its mean from all components. Whereas we subtract the median (cf. zero median points above). This conceptual clarity is one of the benefits of interpreting a data sample of length d as a single vector in d dimensional space.

• gmedian, par_gmedian, wgmedian and par_wgmedian
  our fast multidimensional geometric median (gm) algorithms.

• madgm (median of distances from gm)
  is our generalisation of mad (median of absolute differences from median), to n dimensions. 1d median is replaced in nd by gm. Where mad was a robust measure of 1d data spread, madgm becomes a robust measure of nd data spread. We define it as: median(|pi-gm|,i=1..n), where p1..pn are a sample of n data points, each of which is now a vector.

• tm_stat
  We define our generalized tm_stat of a single scalar observation x as: (x-centre)/spread, with the recommendation to replace mean by median and std by mad, whenever possible. Compare with common t-stat, defined as (x-mean)/std, where std is the standard deviation.
  These are similar to the well known standard z-score, except that the central tendency and spread are obtained from the sample (pivotal quantity), rather than from any old assumed population distribution.

• tm_statistic
  we now generalize tm_stat from scalar domain to vector domain of any number of dimensions, defining tm_statistic as |p-gm|/madgm, where p is a single observation point in nd space. For sample central tendency now serves the geometric median gm vector and the spread is the madgm scalar (see above). The error distance of observation p from the median: |p-gm|, is also a scalar. Thus the co-domain of tm_statistic is a simple positive scalar, regardless of the dimensionality of the vector space in question.

• contribution
  one of the key questions of Machine Learning is how to quantify the contribution that each example (typically represented as a member of some large nd set) makes to the recognition concept, or outcome class, represented by that set. In answer to this, we define the contribution of a point p as the magnitude of displacement of gm, caused by adding p to the set. Generally, outlying points make greater contributions to the gm but not as much as to the centroid. The contribution depends not only on the radius of p but also on the radii of all other existing set points and on their number.

• comediance
  is similar to covariance. It is a triangular symmetric matrix, obtained by supplying method covar with the geometric median instead of the usual centroid. Thus zero mean vectors are replaced by zero median vectors in the covariance calculations. The results are similar but more stable with respect to the outliers.

• outer_hull is a subset of all zero median points p, such that no other points lie outside the normal plane through p. The points that do not satisfy this condition are called the internal points.

• inner_hull is a subset of all zero median points p, that do not lie outside the normal plane of any other point. Note that in a highly dimensional space up to all points may belong to both the inner and the outer hulls, as, for example, when they all lie on the same hypersphere.

• depth is a measure of likelihood of a zero median point p belonging to a data cloud. More specifically, it is the projection onto unit p of a sum of unit vectors that lie outside the normal through p. For example, all outer hull points have by their definition depth = 0, whereas the inner hull points have high values of depth. This is intended as an improvement on Mahalanobis distance which has a similar goal but says nothing about how well enclosed p is. Whereas tm_statistic only informs about the probability pertaining to the whole cloud, not to its local shape near p.

• sigvec (signature vector)
  Proportional projections of a cloud of zero median vectors on all hemispheric axis. Whe