Interpreting permutation p-values

The permutation test reduces a lot of computation to one number, the p-value. It is easy to read too much into it. This page explains what the number says, where its limits are, and how it relates to the CV metrics.

Null hypothesis

The test compares the real result with results on data where the features and the labels have been made independent by shuffling the labels. The null hypothesis is:

The features contain no information about the class; any performance of the model is due to chance.

The p-value is the probability, under this hypothesis, of getting a CV metric at least as good as the one you got. In Chrometrica it is estimated as

p = \frac{\#\{\text{permutations with a metric} \ge \text{original}\} + 1}{m + 1},

with $m$ permutations. A small p-value means that a result this good rarely happens by chance, so the null hypothesis is unlikely.

The test is conditional on the whole procedure. The shuffled labels go through the same CV scheme, the same folds and (with the recommended mode) the same feature selection. This is its strength: optimism built into the procedure, such as feature selection on all rows, affects the original and the shuffled results alike.

Minimum achievable p-value

Because the original labelling is counted as one of the permutations, $p$ can never be smaller than

p_\text{min} = \frac{1}{m + 1} .

Permutations $m$	Smallest $p$
50	0.020
100	0.0099
200 (default)	0.0050
500	0.0020

If you get $p = p_\text{min}$ , no permutation reached the original metric. Report it as $p \le p_\text{min}$ (for example, " $p \le 0.005$ , 200 permutations"), not as $p = 0$ .

Ties count against the model. A shuffled result equal to the original one increases $p$ . With accuracy on a small data set this happens often: with 10 samples, accuracy can only take the values 0, 0.1, …, 1, and a shuffled model can hit 1.0 by luck. AUC takes more distinct values and gives a finer test.

Common misreadings

Misreading	Correct reading
" $p = 0.01$ : the model is 99 % correct."	$p$ says nothing about accuracy. A model with 60 % CV accuracy can have a tiny $p$ on a large data set; a model with 90 % can have a large $p$ on a tiny one. Report both.
" $p < 0.05$ : the model will work on new samples."	The test uses the same data and the same CV. It cannot detect batch effects, a plate or day confounded with the class, or samples that are not representative.
" $p > 0.05$ : the features are useless."	With few samples the test has little power. A large $p$ means "not demonstrated", not "no effect".
"The train p-value is large, so the model is bad."	Flexible models fit shuffled labels on the training data perfectly, so the train p-value is often large. Use the CV p-value.
"I tried five methods; one has $p = 0.03$ ."	With several tests, one small $p$ is expected by chance. Decide on the method before testing, or account for multiple comparisons.

What this means in Chrometrica

Read the CV p-value of the permutation plot, together with the CV metrics on the analysis card.
Use at least 100 permutations when you report $p$ below 0.01.
If the analysis used feature selection, keep Feature Selection in Permutations on Repeated for each permutation.
Prefer AUC (or R² / Q² for PLS-DA) over accuracy on small data sets.

Null hypothesis​

Minimum achievable p-value​

Common misreadings​

What this means in Chrometrica​

Further reading​

Null hypothesis

Minimum achievable p-value

Common misreadings

What this means in Chrometrica

Further reading