Replicates and Group K-Fold
A plate usually holds several wells of the same sample: technical replicates. They are the best protection against a bad well, but for cross-validation they are a trap. If replicates of one sample end up on both sides of a split, the CV metrics measure how well the model recognises a sample it has already seen, not how well it classifies a new one.
The problem
Imagine 10 samples, 5 of class A and 5 of class B, each measured in 4 wells: 40 rows. The 4 wells of a sample differ only by measurement noise; different samples of the same class differ much more (concentration, preparation, matrix).
With Random K-Fold on the 40 rows, a test fold contains, say, well 3 of sample 7, while wells 1, 2 and 4 of sample 7 are in the training set. A nearest-neighbour model finds well 1 of sample 7 at a tiny distance and copies its class. It is right, but for the wrong reason: it would have been right even if the class of sample 7 had nothing to do with its colour.
The CV accuracy is then close to the training accuracy, and both are misleading. The problem is worst for methods that can memorise single points (k-NN, random forest, SVM with a narrow kernel), but it affects every method: the model is always evaluated on points closer to its training data than new samples will be.
Group K-Fold
Group K-Fold assigns groups, not rows, to folds: all rows of one group go
to the same fold. In Chrometrica the group is the sample_name column. Every
test fold therefore contains only samples that the model has never seen, in
any replicate.
With one fold per sample (the default number of folds), the scheme becomes leave-one-sample-out: each sample is predicted by a model trained on all the other samples. This is the closest CV can get to "a new sample arrives".
Group K-Fold does not stratify: a fold can contain one class only. That is expected and harmless for the overall numbers, but it makes some per-fold metrics meaningless. See Metrics → Small folds.
Averaging vs grouping
The alternative is to average the replicates before the analysis: every sample becomes one row, and the problem disappears with any CV scheme.
| Average replicates | Keep replicates, Group K-Fold | |
|---|---|---|
| Unit of the model | Sample (averaged wells) | Single well |
| Noise | Reduced before modelling | The model sees and must tolerate it |
| Rows for fitting | Number of samples | Number of wells |
| Effective information | The same | The same: replicates add little new information about the class |
| Prediction on new data | Average the new wells the same way | Works on single wells |
| Outlier wells | Pull the mean; check the data first | Visible as separate points |
Choose by how the model will be used:
- New samples will be measured in replicates, and you want one answer per sample: average.
- You want a prediction for every well, or you want to see how consistent the replicates of a sample are: keep them and use Group K-Fold.
What you should not do is keep the replicates and use Random or Stratified K-Fold.
What this means in Chrometrica
- Fill in
sample_namefor every row: it defines the groups, whatever label column you analyse. - For a class label, the dialog selects Group K-Fold with one fold per sample. Keep it. See Cross-validation.
- To average, turn on Average technical replicates in the analysis dialog, and the same option in the prediction dialog.
Further reading
- Roberts D. R. et al. Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure. Ecography, 40, 913–929 (2017). doi:10.1111/ecog.02881
- scikit-learn user guide: Cross-validation iterators for grouped data.