<img src="img/permutools_logo.png">

PERMUTOOLS is a statistical software package for multivariate permutation testing and effect size measurement. It is optimised for multivariate data and uses efficient resampling algorithms to generate the sampling distribution empirically, providing distribution-free, nonparametric hypothesis testing and effect size analysis.

PERMUTOOLS offers permutation-based hypothesis testing and confidence interval estimation for a range of statistical tests, including the ANOVA (one-way, two-way), t-test (one-sample, paired-sample, two-sample) F-test (two-sample), Z-test (one-sample), and correlation coefficient (Pearson, Spearman, rankit). Multiple comparison correction is automatically applied using the max correction method, which is less prone to type II errors than conventional methods.

PERMUTOOLS also offers effect size measurement and bootstrapped confidence interval estimation for a range of standardised and unstandardised effect sizes, including Cohen's d, Hedges' g, Glass' Δ, Cliff's d, unstandardised mean difference and unstandardised median difference. Inflation of standardised effect size measures and their CIs due to sample size is controlled by applying a bias correction factor.

• Installation
• Documentation
• Contents
• Correction Features
• Examples
• Citation
• References
• License

Installation

MATLAB

Download and unzip PERMUTOOLS to a local directory, then in the MATLAB command window enter:

addpath(genpath('directory/PERMUTOOLS-1.1/permutools'))
savepath

Alternatively, use the MATLAB dialog box to install PERMUTOOLS. On the Home tab, in the Environment section, click Set Path. In the Set Path dialog box, click Add Folder with Subfolders and search for PERMUTOOLS in your local directory and select the permutools subfolder.

Octave

Download and unzip PERMUTOOLS to a local directory, then in the Octave command window enter:

addpath(genpath('directory/PERMUTOOLS-1.1/octave'))
savepath
pkg load statistics

Alternatively, use the Octave dialog box to install PERMUTOOLS. In the Edit menu, click Set Path. In the Set Path dialog box, click the Add Folder... dropdown menu and select Folder With Subfolders and search for PERMUTOOLS in your local directory and select the octave subfolder.

Documentation

For documentation and citation, please refer to the arXiv paper:

• Crosse MJ, Foxe JJ, Molholm S (2024) PERMUTOOLS: A MATLAB Package for Multivariate Permutation Testing. arXiv 2401.09401. https://doi.org/10.48550/arXiv.2401.09401

For usage, please see the example code provided in the Examples section below, as well the M-files in the examples folder. For detailed usage, please see the help documentation in each of the function headers.

Contents

PERMUTOOLS consists of the following set of functions:

| Function | Description | | --- | --- | | permuanova1() | One-way permutation-based analysis of variance (ANOVA) | | permuanova2() | Two-way permutation-based analysis of variance (ANOVA) | | permuttest() | One/paired-sample permutation-based t-test with max correction | | permuttest2() | Two-sample permutation-based t-test with max correction | | permuvartest2() | Two-sample permutation-based F-test with max correction | | permuztest() | One-sample permutation-based Z-test with max correction | | permucorr() | Permutation-based correlation analysis with max correction | | booteffectsize() | Effect size & bootstrapped confidence intervals with bias correction |

Correction Features

Max Correction for Multiple Comparisons

Max correction, also referred to as t_max or joint correction, is an effective way of controlling family-wise error rate (FWER) when conducting multivariate permutation tests (Blair et al., 1993, 1994; Westfall and Young, 1993). It works as follows: on each permutation of the data, the test statistic is computed for each variable and the maximum absolute value (or most extreme positive or negative value) is taken. Repeating this procedure thousands of times produces a single, more-conservative permutation distribution, against which the actual test statistic is compared (see figure below). Thus, the more tests there are to take the maximum across, the more conservative the permutation distribution naturally becomes. This highly intuitive approach provides strong control of FWER, even for small sample sizes, and is much more powerful than traditional correction methods (Gondan, 2010; Groppe et al., 2011a,b; Rousselet, 2023). PERMUTOOLS automatically applies max correction to multivariate tests, unless specified otherwise.

Bias Correction for Small Samples

A common effect size measure is the standardised mean difference, known as Cohen's d (Cohen, 1969). Standardised effect sizes have the advantage of being metric-free, meaning that they can be directly compared across different studies (Hentschke & Stuttgen, 2011). However, Cohen's d has been shown to have an upwards bias of up to about 4% for sample sizes of less than 50. This bias is somewhat reduced by using the pooled weighted standard deviation of the samples, instead of that of either sample. In addition, a bias correction factor can be applied to the effect size estimate, which is approximately equal to $1−3/(4n−9)$ (Hedges, 1985). When this correction factor is applied, it is usual to refer to the resulting estimate as Hedges' g. PERMUTOOLS automatically applies bias correction to measures of Cohen's d and Glass' Δ, unless specified otherwise.

<img src="img/fig_permutation_distribution.png">

The above figure shows two permutation distributions based on the t-statistic – one with max correction (red), the other without (blue) – for synthetically generated data with 20 variables (i.e. corrected for 20 comparisons).

Examples

The example code used to generate the results and plots below can be found in run_github_examples.m, as well as more extensive examples for each function in the examples folder.

Permutation tests for multivariate data

The following example demonstrates how to test whether two independent samples come from distributions with equal means in PERMUTOOLS, and compares the test results to those of the equivalent parametric tests in MATLAB/Octave.

First, we generate random multivariate data for 2 "independent" samples X and Y. Each sample has 20 variables, each with a mean value of approximately 0, except for the first 10 variables of Y which have a mean value of approximately -1. Each variable has 30 observations.

% Generate random data
rng(42);
x = randn(30,20);
y = randn(30,20);

% Make the first 10 variables of Y have a mean of -1
y(:,1:10) = y(:,1:10)-1;

Let's assume that we do not know whether the data in X and Y come from distributions with equal variances and thus whether we should use a two-sample Student's t-test or a Welch's t-test. To establish this, we compare the variances of each corresponding variable in X and Y using two-tailed tests based on the F-statistic, first using the standard parametric approach (i.e. F-tests), and then using the equivalent nonparametric approach (i.e. permutation tests). For demonstration purposes, the permutation tests are conducted both with and without correction for multiple comparisons.

% Run MATLAB's two-sample parametric variance test (F-test)
[~,p,ci] = vartest2(x,y);

% Run PERMUTOOLS' two-sample permutation variance test (uncorrected)
[f,pu,ciu] = permuvartest2(x,y,'correct',0);

% Run PERMUTOOLS' two-sample permutation variance test (max-corrected)
[~,pc,cic] = permuvartest2(x,y,'correct',1);

To demonstrate the benefit of permutation tests with max correction over traditional parametric tests, we plot the F-statistic along with the parametric and permutation CIs for each test. Variables that are found to be significantly different (p < 0.05) are indicated by black o's (parametric tests) and red x's (permutation tests). We see that applying max correction widens the CIs of the test statistic such that none of the spuriously significant results survive.

% Set up figure
figure('Name','Two-sample permutation F-test','NumberTitle','off')
set(gcf,'color','w')
xaxis = 1:size(f,2);
alpha = 0.05;

% Plot parametric & uncorrected permutation CIs
subplot(2,2,1), hold on
plot(xaxis,f,'LineWidth',3)
plot(xaxis,ci,'k',xaxis,ciu,'--r','LineWidth',1)
plot(xaxis(p<=alpha),f(p<=alpha),'ok','LineWidth',2)
plot(xaxis(pu<=alpha),f(pu<=alpha),'xr','LineWidth',2)
xlim([0,21]), ylim([0,6]), box on, grid on
title('Uncorrected'), ylabel('{\itF}-value')
legend('{\itF}-statistic','parametric CI','','permutation CI','Location','northeast')

% Plot parametric & corrected permutation