Skip to main content

Metrics

This page defines every metric on the analysis card and in the results. The numbers come from the run_<method>_analysis functions in chrometrica/analysis/analysis.py; the formulas are those of sklearn.metrics.

Classification metrics​

For one class treated as "positive": TPTP are the positive rows predicted as positive, FNFN the positive rows predicted as something else, FPFP the other rows predicted as positive, TNTN the other rows predicted as other.

MetricCard labelFormulaMeaning
AccuracyCV Accuracycorrectall\dfrac{\text{correct}}{\text{all}}Share of correctly classified rows
PrecisionCV PrecisionTPTP+FP\dfrac{TP}{TP + FP}Of the rows predicted as this class, how many really are
RecallCV RecallTPTP+FN\dfrac{TP}{TP + FN}Of the rows of this class, how many were found
F1CV F1-Score2⋅precision⋅recallprecision+recall\dfrac{2 \cdot \text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}}Harmonic mean of precision and recall
SensitivityCV SensitivityTPTP+FN\dfrac{TP}{TP + FN}Recall of the positive class. Binary only.
SpecificityCV SpecificityTNTN+FP\dfrac{TN}{TN + FP}Recall of the negative class. Binary only.
ROC AUCCV AUC, CV AUC AggArea under the ROC curveProbability that a random positive row gets a higher score than a random negative row. 0.5 = chance, 1 = perfect ranking.
  • Precision, recall and F1 are macro averages: computed for every class separately, then averaged with equal weight per class. A small class counts as much as a large one. A class with no predicted rows gets precision 0 (zero_division=0).
  • Positive class for sensitivity, specificity and AUC: the second class in alphabetical order of the labels. For labels neg and pos, pos is positive; for 0 and 1, 1 is positive.
  • AUC needs a continuous score, not only a class. Each method uses its own score: the probability of the positive class (logistic regression, SVM, random forest, k-NN, PLS-DA), the decision function (LDA), or a distance (SIMCA). AUC is computed for binary problems; PLS-DA and SIMCA also give a macro one-vs-rest AUC for more classes. XGBoost, Balanced Bagging and Balanced Random Forest do not compute AUC.
  • Confusion matrix: rows are the true classes, columns the predicted classes, both in alphabetical order.

The full per-class numbers are in classification_report (training) and cv_class_reports (one per fold); the card shows the CV averages in Class Metrics (CV Average).

Cross-validated vs training metrics​

Every classification analysis reports two sets of metrics:

Cross-validated (cv_*)Training (no prefix)
ModelA new model per fold, fitted without the test foldThe final model, fitted on all rows
Tested onThe rows of the test fold, never seen in fittingThe same rows it was fitted on
Tells youHow the method is expected to perform on new samplesHow well the model fits its own data
Fieldscv_accuracy, cv_precision, …, cv_confusion_matrix_aggregatedaccuracy, sensitivity, specificity, auc, confusion_matrix, classification_report, predictions

Judge a model by the cross-validated metrics. Training metrics are always optimistic, and for flexible models (random forest, k-NN with distance weights, XGBoost) they are often 100 % regardless of the data. A large gap between training and CV accuracy means overfitting. Why this matters and what else can inflate the CV numbers is explained in Cross-validation and data leakage.

Three ways to combine folds​

With FF folds there are three ways to turn per-fold results into one number, and the results contain all three:

KindFieldsHow it is computed
Mean over foldscv_mean (= cv_accuracy), cv_precision, cv_recall, cv_f1, cv_sensitivity, cv_specificity, cv_auc, and their _stdThe metric is computed on every test fold, then averaged. The ± value on the card is the standard deviation over folds. Per-fold values: cv_scores, cv_sensitivity_scores, cv_auc_scores.
Pooled (aggregated)cv_confusion_matrix_aggregated, cv_sensitivity_aggregated, cv_specificity_aggregated, cv_auc_aggregated (CV AUC Agg)The predictions of all test folds are put together (every row once), and the metric is computed once on this pooled set.
Averaged matrixcv_confusion_matrix, cv_sensitivity_avg, cv_specificity_avgThe confusion matrices of the folds are averaged cell by cell. Sensitivity and specificity from this matrix equal the pooled ones.

The card shows the mean over folds for accuracy, precision, recall, F1 and AUC, and the averaged-matrix values for sensitivity and specificity. The confusion matrix on the card is the pooled one (Cross-Validation Confusion Matrix (Aggregated)).

:::tip Small folds

With many small folds, for example the default leave-one-sample-out Group K-Fold, each test fold holds the replicates of a single sample, so it contains one class only. Then:

  • per-fold accuracy is the share of correct replicates of that sample, and the ± value is large;
  • per-fold AUC cannot be computed (it needs both classes), so CV AUC is empty or based on a few folds;
  • per-fold precision and recall are computed on one class only and are hard to interpret.

In this case, use the pooled values: CV AUC Agg and the aggregated confusion matrix.

:::

Consistency note​

For binary problems, the card may show a note like Note: Accuracy (0.850) differs from (sensitivity + specificity)/2 (0.800). This is not an error. Accuracy weighs every row equally, while (sensitivity + specificity)/2, the balanced accuracy, weighs both classes equally. They differ when the classes have different sizes.

PLS-DA: R² and Q²​

For binary PLS-DA, the classes are coded 0 and 1, the predicted classes are coded the same way (a sample the soft rule does not assign counts as 0.5), and

R2 or Q2=1−∑i(yi−y^i)2∑i(yi−yˉ)2.R^2 \text{ or } Q^2 = 1 - \frac{\sum_i (y_i - \hat{y}_i)^2}{\sum_i (y_i - \bar{y})^2} .

On the training data it is called R2R^2 (r2); with cross-validation, Q2Q^2 (cv_r2, card label CV R²). Because it is computed from class codes, Q2=1Q^2 = 1 means no CV errors, and each error lowers it by an amount that depends on the class balance. Q2Q^2 can be negative.

Regression metrics​

Linear regression reports the coefficient of determination

R2=1−∑i(yi−y^i)2∑i(yi−yˉ)2R^2 = 1 - \frac{\sum_i (y_i - \hat{y}_i)^2}{\sum_i (y_i - \bar{y})^2}

on every test fold (cv_scores, mean cv_mean ± cv_std) and on the training data (r2_score). R2=1R^2 = 1 is a perfect fit; R2=0R^2 = 0 means the model is no better than predicting the mean; R2<0R^2 < 0 means it is worse.

Source code​

How the per-fold metrics are collected (LDA; the other methods do the same):

analysis.py · run_lda_analysis() · lines 1067–1086
# Store fold results (unchanged)
cv_scores.append(accuracy_score(y_test, y_pred))
cv_predictions.extend(y_pred)
cv_true_labels.extend(y_test)

fold_report = classification_report(y_test, y_pred, output_dict=True, zero_division=0)
cv_class_reports.append(fold_report)

cm = confusion_matrix(y_test, y_pred, labels=overall_classes_list)
cv_confusion_matrices.append(cm)

if is_binary:
    tn, fp, fn, tp = cm.ravel()
    sensitivity = tp / (tp + fn) if (tp + fn) > 0 else 0
    specificity = tn / (tn + fp) if (tn + fp) > 0 else 0
    cv_sensitivity_scores.append(sensitivity)
    cv_specificity_scores.append(specificity)
else:
    cv_sensitivity_scores.append(None)
    cv_specificity_scores.append(None)

Pooled and averaged values after the CV loop:

analysis.py · run_lda_analysis() · lines 1088–1148
# --- After CV loop: compute aggregated AUC only for binary ---
auc_aggregated = None
if is_binary and cv_scores_all:
    # Try to compute aggregated AUC from all scores; if it fails, set to None
    try:
        auc_aggregated = roc_auc_score(cv_true_labels, cv_scores_all)
    except ValueError:
        auc_aggregated = None
    # Ensure no NaN
    if auc_aggregated is not None and np.isnan(auc_aggregated):
        auc_aggregated = None

# Compute mean/std of per-fold AUC (only binary)
if is_binary:
    valid_auc = [a for a in cv_auc_scores if a is not None]
    cv_auc_mean = np.mean(valid_auc) if valid_auc else None
    cv_auc_std = np.std(valid_auc) if valid_auc else None
else:
    cv_auc_mean = None
    cv_auc_std = None

# ---- Continue with existing CV metrics (unchanged) ----
cv_precision_scores = []
cv_recall_scores = []
cv_f1_scores = []

for report in cv_class_reports:
    if 'macro avg' in report:
        cv_precision_scores.append(report['macro avg']['precision'])
        cv_recall_scores.append(report['macro avg']['recall'])
        cv_f1_scores.append(report['macro avg']['f1-score'])

cv_confusion_matrix_aggregated = confusion_matrix(cv_true_labels, cv_predictions,
                                                  labels=overall_classes_list).tolist()

if is_binary:
    cm_agg = np.array(cv_confusion_matrix_aggregated)
    tn, fp, fn, tp = cm_agg.ravel()
    cv_sensitivity_aggregated = tp / (tp + fn) if (tp + fn) > 0 else 0
    cv_specificity_aggregated = tn / (tn + fp) if (tn + fp) > 0 else 0
else:
    cv_sensitivity_aggregated = None
    cv_specificity_aggregated = None

if cv_confusion_matrices:
    cv_confusion_matrix_avg, avg_matrix_classes = create_average_confusion_matrix(
        cv_confusion_matrices, [overall_classes_list] * len(cv_confusion_matrices)
    )
    if is_binary:
        cm_avg = np.array(cv_confusion_matrix_avg)
        tn, fp, fn, tp = cm_avg.ravel()
        cv_sensitivity_avg = tp / (tp + fn) if (tp + fn) > 0 else 0
        cv_specificity_avg = tn / (tn + fp) if (tn + fp) > 0 else 0
    else:
        cv_sensitivity_avg = None
        cv_specificity_avg = None
else:
    cv_confusion_matrix_avg = []
    avg_matrix_classes = []
    cv_sensitivity_avg = None
    cv_specificity_avg = None

The averaged confusion matrix:

analysis.py · create_average_confusion_matrix() · lines 159–197
def create_average_confusion_matrix(confusion_matrices, fold_classes_list):
    """Create average confusion matrix with consistent label ordering"""
    # Get all unique classes across all folds
    all_classes = set()
    for fold_classes in fold_classes_list:
        all_classes.update(fold_classes)
    all_classes = sorted(all_classes)  # Consistent ordering

    n_classes = len(all_classes)
    class_to_index = {cls: idx for idx, cls in enumerate(all_classes)}

    # Initialize sum matrix
    sum_matrix = np.zeros((n_classes, n_classes))
    fold_count = np.zeros((n_classes, n_classes))  # Track how many folds contributed to each cell

    for cm, fold_classes in zip(confusion_matrices, fold_classes_list):
        # Create mapping from fold class indices to global class indices
        fold_to_global = {}
        for i, cls in enumerate(fold_classes):
            fold_to_global[i] = class_to_index[cls]

        # Map the confusion matrix to global indices
        for i in range(cm.shape[0]):
            for j in range(cm.shape[1]):
                global_i = fold_to_global[i]
                global_j = fold_to_global[j]
                sum_matrix[global_i, global_j] += cm[i, j]
                fold_count[global_i, global_j] += 1

    # Calculate average (avoid division by zero)
    avg_matrix = np.zeros((n_classes, n_classes))
    for i in range(n_classes):
        for j in range(n_classes):
            if fold_count[i, j] > 0:
                avg_matrix[i, j] = sum_matrix[i, j] / fold_count[i, j]
            else:
                avg_matrix[i, j] = 0

    return avg_matrix.tolist(), all_classes

References​