Cross-validation and data leakage

A model is useful only if it works on samples it has never seen. Cross-validation estimates exactly that, but only if the test rows are truly unseen. Any path by which information from the test rows reaches the model, directly or through a choice you made, is data leakage. Leakage does not produce an error message; it produces numbers that look better than they are, and a model that disappoints on the next plate.

Training vs validation error

A model fitted on some rows and evaluated on the same rows measures how well it can reproduce them, not how well it can predict. The more flexible the model, the larger the gap: a random forest or a 1-nearest-neighbour classifier can reach 100 % training accuracy on random labels.

Cross-validation splits the rows into folds, fits a model without one fold, tests it on that fold, and repeats for every fold. Every row is predicted by a model that did not see it. The CV estimate is slightly pessimistic (each model is trained on less data than the final one), but it is honest, as long as nothing leaks.

A useful rule: if the training accuracy is much higher than the CV accuracy, the model overfits; if the CV accuracy is surprisingly high, look for leakage.

Sources of leakage

Source	What happens	Typical effect
Replicates split across folds	A replicate of the test sample is in the training set; the model recognises the sample, not the class	Very large, especially for k-NN and random forest
Preprocessing on all rows	Scaling, feature selection or any fitted transformation uses the test rows	Small for scaling; large for supervised feature selection with few features kept out of many
Tuning on the test folds	You try several settings (number of components, $k$ , number of features, …) and keep the one with the best CV score	Grows with the number of settings tried
Early stopping on the test fold	The number of boosting rounds is chosen by the score on the test fold	Moderate
Batch effects	All samples of one class were measured on one plate or day; the model learns the plate, not the class	Large, and CV cannot detect it

The last one deserves emphasis: if the classes and the plates are confounded, every CV scheme that mixes rows of the same plate will be fooled. Only the design of the experiment (every plate contains every class) or a validation on a new plate can reveal it.

How Chrometrica prevents leakage

What is protected:

Replicates. With Group K-Fold (the default for class labels), all rows of a sample_name stay in one fold. With replicate averaging, each sample is a single row.
Scaling. Where a method scales the data (logistic regression, k-NN, PLS-DA, SIMCA), the scaler is part of the model and is refitted in every fold, on the training rows only.
Reproducible folds. Shuffling uses a fixed seed, so reruns give the same folds and differences between runs come from your settings, not from luck.
A fair permutation test. The permutation test repeats the procedure, including feature selection, for every shuffle. Whatever leaks through feature selection in the original analysis leaks equally into the shuffled ones, so the p-value stays honest even when the CV numbers are somewhat optimistic. (The search for the number of PLS-DA components is not repeated: every shuffle uses the number chosen for the original labels.)

What is not protected, and how to deal with it:

Situation	Where	What to do
Feature selection runs once on all rows before CV	Feature selection	Keep $k$ moderate; confirm with the permutation test (mode Repeated for each permutation)
PLS-DA picks the number of components by the CV score it reports	PLS-DA	Keep the search range small; confirm with the permutation test
XGBoost early stopping uses the test fold	XGBoost	Set Early Stopping Rounds to 0
Random K-Fold or Stratified K-Fold with replicates	Cross-validation	Use Group K-Fold or averaging
You compare many methods and settings and report the best	Everywhere	Keep a few samples aside, or measure a new plate, for a final check

What this means in Chrometrica

Keep the default Group K-Fold when your table has replicates.
Read the CV metrics, not the training ones. See Cross-validated vs training metrics.
For every model you intend to use, run a permutation test and report the CV p-value together with the CV metrics.
Before relying on a model, test it with prediction on samples from a plate that was not used for training.

Training vs validation error​

Sources of leakage​

How Chrometrica prevents leakage​

What this means in Chrometrica​

Further reading​

Training vs validation error

Sources of leakage

How Chrometrica prevents leakage

What this means in Chrometrica

Further reading