niseq

For a full description of the permutation-based approach to sequential hypothesis testing implemented in niseq, check out our NeuroImage paper! (We also appreciate a citation if you use this package in your research.)

niseq provides sequential generalizations of common statistical tests used in neuroimaging. That is, you can analyze your data multiple times throughout your experiment and stop data collection when there is enough evidence to reject the null hypothesis without inflating your false positive rate using an approach called alpha spending.

The alpha spending approach to sequential analysis was first introduced by Lan and DeMets (1983) and has become common practice in clinical trials due to its substantial efficiency advantage over fixed-sample designs (i.e. fewer observations required on average to acheive the same statistical power). However, the original alpha spending approach relies on normality assumptions to derive adjusted significance thresholds, limiting its applicability to statistical tests used in neuroimaging. Our permutation-based approach to alpha spending relaxes these assumptions, allowing essentially any fixed-sample permutation test (e.g. cluster-based permutation test, threshold-free cluster enhancement, network-based statistic, t-max and F-max) to be generalized to a sequentially-valid permutation test.

You may be interested in using niseq if you want to run a well-powered neuroimaging study and

• don't currently have a good way of estimating the sample size you need for your study a priori. You can use a sequential stopping rule to determine your final sample size without inflating your false positive rate.
• do have a good way of estimating a fixed sample size, but would prefer to collect fewer observations if justified by the data. Sequential designs can acheive the same statistical power as a fixed-sample design using, on average, fewer observations.
• want to use an adaptive design in case so you can adjust mid-experiment if you've underestimated the needed sample size. You can conduct a conditional power analysis at an interim look at the data and adjust your design accordingly without inflating your false positive rate.

Installation

A stable version can be installed using

pip install niseq

and the development version using

pip install git+https://github.com/john-veillette/niseq.git

Usage

See our API documentation and example notebooks for usage instructions.

Tutorial notebooks currently include:

• Sequential cluster-based permutation tests and sequential threshold-free cluster enhancement on EEG data
• Sequential cluster-based permutation tests and sequential t-max on fMRI data
• Sequential network-based statistic for fMRI connectivity data
• Power analysis (both a priori & conditional) by bootstrap for fixed-sample, sequential, and adaptive designs