SVM — Support Vector Machine

Task	Classification
Method key (analysis_type)	svm
Prediction on new data	Yes — see Prediction
Library	sklearn.svm.SVC

When to use

• Classes are separated by a curved boundary that linear methods (LDA, logistic regression) cannot follow.
• The data set is small or medium: SVM works well with tens to hundreds of samples.

How it works

A support vector machine looks for the boundary with the widest margin between two classes. Only the samples closest to the boundary, the support vectors, define it. With a kernel $K$ the boundary can be non-linear. The decision function for a sample $x$ is

$$f(x) = \sum_{i \in \text{SV}} \alpha_i y_i K(x_i, x) + b ,$$

and the class is given by the sign of $f(x)$. Chrometrica uses the RBF (Gaussian) kernel

$$K(x, x') = \exp\!\left(-\gamma \lVert x - x' \rVert^2\right), \qquad \gamma = \frac{1}{p \cdot \operatorname{Var}(X)} ,$$

where $p$ is the number of features and $\operatorname{Var}(X)$ the variance of all values of the training matrix (gamma='scale'). The penalty $C$ balances a wide margin against training errors.

For more than two classes, SVC trains one model for every pair of classes (one-vs-one) and takes a vote.

Probabilities. An SVM gives distances to the boundary, not probabilities. With probability=True, scikit-learn calibrates the distances into probabilities with Platt scaling, using an internal 5-fold cross-validation on the training data. The calibrated probability and the predicted class can occasionally disagree for samples close to the boundary.

Parameters

The model runs with random_state=42, probability=True and the scikit-learn defaults for everything else: kernel='rbf', C=1.0, gamma='scale'.

Parameter	UI label	Default	Notes
feature_selection	Enable Feature Selection	off	See Feature selection.
average_replicates	Average technical replicates	off	See Averaging replicates.

note

The dialog also shows Kernel (RBF / Linear / Polynomial / Sigmoid) and Regularization (C). In the current version these values are saved with the analysis but not passed to the model: it always uses the RBF kernel with C=1.0.

Preprocessing

No scaling is applied. The RBF kernel is based on Euclidean distances, so features with a large numeric range dominate the result. If your features have different ranges, bring them to a common scale with adjustments before the analysis.

Before the model: replicate averaging and feature selection, if they are on. See Order of steps.

Results and metrics

The common classification metrics are described in Metrics. For binary problems, AUC uses the calibrated probability of the positive class (the second class in alphabetical order).

SVM-specific fields:

Field	Meaning
prediction_probabilities	Calibrated class probabilities for every sample (final model)

Visualizations

• Confusion matrix and ROC curve
• Permutation plot
• Time series and heatmap of the input data

Source code

run_svm_analysis in chrometrica/analysis/analysis.py. The imports, the model in the CV loop and the final model:

analysis.py · run_svm_analysis() · lines 1738–1947

def run_svm_analysis(X, y, parameters, cv_method, cv_folds):
    """Run SVM analysis with optional feature selection"""
    from sklearn.svm import SVC
    from sklearn.model_selection import StratifiedKFold, GroupKFold, KFold
    from sklearn.metrics import classification_report, confusion_matrix, accuracy_score, roc_auc_score
    import numpy as np
    # ...
    for train_idx, test_idx in cv.split(X_processed, y, groups=groups if cv_method == 'group' else None):
        X_train, X_test = X_processed[train_idx], X_processed[test_idx]
        y_train, y_test = y[train_idx], y[test_idx]

        svm_cv = SVC(random_state=42, probability=True)
        svm_cv.fit(X_train, y_train)
        y_pred = svm_cv.predict(X_test)
    # ...
    # Fit final model
    svm = SVC(random_state=42, probability=True)
    svm.fit(X_processed, y)
    y_pred_full = svm.predict(X_processed)
    y_pred_proba = svm.predict_proba(X_processed)

Full source of run_svm_analysis()

analysis.py · run_svm_analysis() · lines 1738–2072

def run_svm_analysis(X, y, parameters, cv_method, cv_folds):
    """Run SVM analysis with optional feature selection"""
    from sklearn.svm import SVC
    from sklearn.model_selection import StratifiedKFold, GroupKFold, KFold
    from sklearn.metrics import classification_report, confusion_matrix, accuracy_score, roc_auc_score
    import numpy as np

    # ---------- Helper to sanitize JSON ----------
    def sanitize_for_json(obj):
        if isinstance(obj, dict):
            return {k: sanitize_for_json(v) for k, v in obj.items()}
        elif isinstance(obj, list):
            return [sanitize_for_json(v) for v in obj]
        elif isinstance(obj, float) and np.isnan(obj):
            return None
        elif isinstance(obj, (np.float32, np.float64)) and np.isnan(obj):
            return None
        elif isinstance(obj, np.ndarray):
            return sanitize_for_json(obj.tolist())
        else:
            return obj

    if parameters.get('average_replicates', False):
        groups = parameters.get('groups')
        if groups is None:
            raise ValueError("'groups' must be provided when averaging replicates is enabled.")
        X, y, new_groups = average_replicates(X, y, groups, method='mean')
        parameters['groups'] = new_groups

    # Get overall classes
    overall_classes = np.unique(y)
    overall_classes_list = overall_classes.tolist()
    n_overall_classes = len(overall_classes_list)
    is_binary = (n_overall_classes == 2)

    # Define positive class (the second class in sorted order, i.e. the "treatment" or "case" class)
    # If you need a different positive class, pass it via parameters['positive_class']
    positive_class = parameters.get('positive_class', overall_classes_list[1] if is_binary else None)

    # logger.info(f"Positive class for AUC: {positive_class}")
    # logger.info(f"Overall classes: {overall_classes_list}")

    # Check for feature selection
    feature_selection_params = get_feature_selection_params(parameters)
    if feature_selection_params:
        X_processed, selected_features, feature_scores = select_features(
            X, y, feature_selection_params, 'classification'
        )
    else:
        X_processed = X
        selected_features = list(range(X.shape[1]))
        feature_scores = None

    # --- AUC initialisation (only used if binary) ---
    cv_auc_scores = []
    cv_scores_all = []

    # Enhanced cross-validation with metrics
    cv_scores = []
    cv_predictions = []
    cv_true_labels = []
    cv_class_reports = []
    cv_confusion_matrices = []
    cv_sensitivity_scores = []
    cv_specificity_scores = []

    if cv_method == 'stratified':
        cv = StratifiedKFold(n_splits=cv_folds, shuffle=True, random_state=42)
    elif cv_method == 'group':
        groups = parameters.get('groups')
        if groups is None:
            raise ValueError("Groups parameter required for group cross-validation")
        cv = GroupKFold(n_splits=cv_folds)
    else:
        cv = KFold(n_splits=cv_folds, shuffle=True, random_state=42)

    for train_idx, test_idx in cv.split(X_processed, y, groups=groups if cv_method == 'group' else None):
        X_train, X_test = X_processed[train_idx], X_processed[test_idx]
        y_train, y_test = y[train_idx], y[test_idx]

        svm_cv = SVC(random_state=42, probability=True)
        svm_cv.fit(X_train, y_train)
        y_pred = svm_cv.predict(X_test)

        # ---------- AUC computation only for binary ----------
        if is_binary:
            # Use predict_proba to get the probability of the positive class
            if hasattr(svm_cv, "predict_proba"):
                y_prob = svm_cv.predict_proba(X_test)
                # Find the column index of the positive class in the model's classes
                try:
                    pos_idx = list(svm_cv.classes_).index(positive_class)
                except ValueError:
                    # If positive_class is not in the model's classes (shouldn't happen in binary),
                    # fallback to the second column (usually the positive class in binary)
                    pos_idx = 1
                y_score = y_prob[:, pos_idx]
            elif hasattr(svm_cv, "decision_function"):
                # If predict_proba is not available (but we set probability=True), fallback to decision_function
                y_score = svm_cv.decision_function(X_test)
                # We need to ensure that higher scores correspond to the positive class.
                # We can check the sign by comparing the mean decision score on the positive class in training.
                # Since we have probability, this branch is rarely used.
            else:
                y_score = None

            if y_score is not None:
                try:

                    y_test_binary = (y_test == positive_class).astype(int)
                    auc_fold = roc_auc_score(y_test_binary, y_score)
                except ValueError:
                    auc_fold = None
                cv_auc_scores.append(auc_fold)
                if isinstance(y_score, np.ndarray):
                    cv_scores_all.extend(y_score.tolist())
                else:
                    cv_scores_all.extend(y_score)
            else:
                cv_auc_scores.append(None)
        else:
            # Multiclass: skip AUC entirely
            cv_auc_scores.append(None)

        # Store fold results
        cv_scores.append(accuracy_score(y_test, y_pred))
        cv_predictions.extend(y_pred)
        cv_true_labels.extend(y_test)

        fold_report = classification_report(y_test, y_pred, output_dict=True, zero_division=0)
        cv_class_reports.append(fold_report)

        cm = confusion_matrix(y_test, y_pred, labels=overall_classes_list)
        cv_confusion_matrices.append(cm)

        if is_binary:
            tn, fp, fn, tp = cm.ravel()
            sensitivity = tp / (tp + fn) if (tp + fn) > 0 else 0
            specificity = tn / (tn + fp) if (tn + fp) > 0 else 0
            cv_sensitivity_scores.append(sensitivity)
            cv_specificity_scores.append(specificity)
        else:
            cv_sensitivity_scores.append(None)
            cv_specificity_scores.append(None)

    # --- After CV loop: aggregated AUC only for binary ---
    auc_aggregated = None
    if is_binary and cv_scores_all:
        try:
            cv_true_labels_binary = (np.array(cv_true_labels) == positive_class).astype(int)
            auc_aggregated = roc_auc_score(cv_true_labels_binary, cv_scores_all)
        except ValueError:
            auc_aggregated = None
        if auc_aggregated is not None and np.isnan(auc_aggregated):
            auc_aggregated = None

    if is_binary:
        valid_auc = [a for a in cv_auc_scores if a is not None]
        cv_auc_mean = np.mean(valid_auc) if valid_auc else None
        cv_auc_std = np.std(valid_auc) if valid_auc else None
    else:
        cv_auc_mean = None
        cv_auc_std = None

    # ---- Continue with existing CV metrics (unchanged) ----
    cv_precision_scores = []
    cv_recall_scores = []
    cv_f1_scores = []

    for report in cv_class_reports:
        if 'macro avg' in report:
            cv_precision_scores.append(report['macro avg']['precision'])
            cv_recall_scores.append(report['macro avg']['recall'])
            cv_f1_scores.append(report['macro avg']['f1-score'])

    cv_confusion_matrix_aggregated = confusion_matrix(cv_true_labels, cv_predictions,
                                                      labels=overall_classes_list).tolist()

    if is_binary:
        cm_agg = np.array(cv_confusion_matrix_aggregated)
        tn, fp, fn, tp = cm_agg.ravel()
        cv_sensitivity_aggregated = tp / (tp + fn) if (tp + fn) > 0 else 0
        cv_specificity_aggregated = tn / (tn + fp) if (tn + fp) > 0 else 0
    else:
        cv_sensitivity_aggregated = None
        cv_specificity_aggregated = None

    if cv_confusion_matrices:
        cv_confusion_matrix_avg, avg_matrix_classes = create_average_confusion_matrix(
            cv_confusion_matrices, [overall_classes_list] * len(cv_confusion_matrices)
        )
        if is_binary:
            cm_avg = np.array(cv_confusion_matrix_avg)
            tn, fp, fn, tp = cm_avg.ravel()
            cv_sensitivity_avg = tp / (tp + fn) if (tp + fn) > 0 else 0
            cv_specificity_avg = tn / (tn + fp) if (tn + fp) > 0 else 0
        else:
            cv_sensitivity_avg = None
            cv_specificity_avg = None
    else:
        cv_confusion_matrix_avg = []
        avg_matrix_classes = []
        cv_sensitivity_avg = None
        cv_specificity_avg = None

    # Fit final model
    svm = SVC(random_state=42, probability=True)
    svm.fit(X_processed, y)
    y_pred_full = svm.predict(X_processed)
    y_pred_proba = svm.predict_proba(X_processed)

    final_classes = svm.classes_.tolist()

    if cv_confusion_matrix_avg and avg_matrix_classes != final_classes:
        avg_to_final = {cls: idx for idx, cls in enumerate(avg_matrix_classes)}
        n_classes = len(final_classes)
        reordered_matrix = np.zeros((n_classes, n_classes))
        for i, true_cls in enumerate(final_classes):
            for j, pred_cls in enumerate(final_classes):
                if true_cls in avg_to_final and pred_cls in avg_to_final:
                    orig_i = avg_to_final[true_cls]
                    orig_j = avg_to_final[pred_cls]
                    if (orig_i < len(cv_confusion_matrix_avg) and
                            orig_j < len(cv_confusion_matrix_avg[0])):
                        reordered_matrix[i, j] = cv_confusion_matrix_avg[orig_i][orig_j]
        cv_confusion_matrix_avg = reordered_matrix.tolist()
        avg_matrix_classes = final_classes

    full_confusion_matrix = confusion_matrix(y, y_pred_full, labels=final_classes)
    if is_binary and full_confusion_matrix.shape == (2, 2):
        tn, fp, fn, tp = full_confusion_matrix.ravel()
        full_sensitivity = tp / (tp + fn) if (tp + fn) > 0 else 0
        full_specificity = tn / (tn + fp) if (tn + fp) > 0 else 0
    else:
        full_sensitivity = None
        full_specificity = None

    validation_message = None
    if is_binary and full_sensitivity is not None and full_specificity is not None:
        avg_sens_spec = (full_sensitivity + full_specificity) / 2
        accuracy = accuracy_score(y, y_pred_full)
        accuracy_diff = abs(accuracy - avg_sens_spec)
        if accuracy_diff > 0.01:
            validation_message = f"Note: Accuracy ({accuracy:.3f}) differs from (sensitivity + specificity)/2 ({avg_sens_spec:.3f}) by {accuracy_diff:.3f}"

    cv_sensitivity_scores_filtered = [s for s in cv_sensitivity_scores if s is not None]
    cv_sensitivity_mean = np.mean(cv_sensitivity_scores_filtered) if cv_sensitivity_scores_filtered else None
    cv_sensitivity_std = np.std(cv_sensitivity_scores_filtered) if cv_sensitivity_scores_filtered else None

    cv_specificity_scores_filtered = [s for s in cv_specificity_scores if s is not None]
    cv_specificity_mean = np.mean(cv_specificity_scores_filtered) if cv_specificity_scores_filtered else None
    cv_specificity_std = np.std(cv_specificity_scores_filtered) if cv_specificity_scores_filtered else None

    # --- Full-data (resubstitution) AUC and accuracy, field parity with run_lda_analysis ---
    full_auc = None
    if is_binary:
        try:
            pos_idx = final_classes.index(positive_class)
            y_true_binary = (y == positive_class).astype(int)
            full_auc = roc_auc_score(y_true_binary, y_pred_proba[:, pos_idx])
        except Exception:
            full_auc = None
    full_accuracy = accuracy_score(y, y_pred_full)

    results = {
        'method': 'SVM',
        'cv_scores': cv_scores,
        'cv_mean': np.mean(cv_scores) if cv_scores else 0.0,
        'cv_std': np.std(cv_scores) if cv_scores else 0.0,
        'cv_precision': np.mean(cv_precision_scores) if cv_precision_scores else 0.0,
        'cv_precision_std': np.std(cv_precision_scores) if cv_precision_scores else 0.0,
        'cv_recall': np.mean(cv_recall_scores) if cv_recall_scores else 0.0,
        'cv_recall_std': np.std(cv_recall_scores) if cv_recall_scores else 0.0,
        'cv_f1': np.mean(cv_f1_scores) if cv_f1_scores else 0.0,
        'cv_f1_std': np.std(cv_f1_scores) if cv_f1_scores else 0.0,
        # Sensitivity
        'cv_sensitivity': cv_sensitivity_mean,
        'cv_sensitivity_std': cv_sensitivity_std,
        'cv_sensitivity_aggregated': cv_sensitivity_aggregated,
        'cv_sensitivity_avg': cv_sensitivity_avg,
        'cv_sensitivity_scores': cv_sensitivity_scores,
        # Specificity
        'cv_specificity': cv_specificity_mean,
        'cv_specificity_std': cv_specificity_std,
        'cv_specificity_aggregated': cv_specificity_aggregated,
        'cv_specificity_avg': cv_specificity_avg,
        'cv_specificity_scores': cv_specificity_scores,
        # AUC (sanitized; only for binary)
        'cv_auc': sanitize_for_json(cv_auc_mean),
        'cv_auc_std': sanitize_for_json(cv_auc_std),
        'cv_auc_scores': sanitize_for_json(cv_auc_scores),
        'cv_auc_aggregated': sanitize_for_json(auc_aggregated),
        # Full-data AUC / accuracy (+ CV accuracy alias), field parity with LDA
        'auc': sanitize_for_json(full_auc),
        'accuracy': full_accuracy,
        'cv_accuracy': np.mean(cv_scores) if cv_scores else 0.0,
        # Validation
        'is_binary': is_binary,
        'validation_message': validation_message,
        # Confusion matrices
        'cv_confusion_matrix': cv_confusion_matrix_avg,
        'cv_confusion_matrices': [cm.tolist() for cm in cv_confusion_matrices],
        'cv_confusion_matrix_aggregated': cv_confusion_matrix_aggregated,
        'cv_class_reports': cv_class_reports,
        'classification_report': classification_report(y, y_pred_full, output_dict=True, zero_division=0),
        'confusion_matrix': full_confusion_matrix.tolist(),
        'sensitivity': full_sensitivity,
        'specificity': full_specificity,
        'labels': y.tolist(),
        'predictions': y_pred_full.tolist(),
        'prediction_probabilities': y_pred_proba.tolist(),
        'classes': final_classes,
    }

    if feature_selection_params:
        results['feature_selection'] = {
            'selected_features': selected_features,
            'feature_scores': feature_scores,
            'method': feature_selection_params.get('method', 'anova'),
            'k': feature_selection_params.get('k', 'all')
        }

    if is_binary and full_sensitivity is not None and full_specificity is not None:
        results['metric_consistency_check'] = {
            'accuracy': accuracy_score(y, y_pred_full),
            'sensitivity': full_sensitivity,
            'specificity': full_specificity,
            'avg_sens_spec': (full_sensitivity + full_specificity) / 2,
            'difference': abs(accuracy_score(y, y_pred_full) - (full_sensitivity + full_specificity) / 2)
        }

    results = sanitize_for_json(results)

    # Final fitted model for inference on unknown samples (popped in run_analysis)
    return attach_model_bundle(results, svm, selected_features, feature_selection_params)

Limitations and common pitfalls

• Scale. See Preprocessing. This is the most common reason for a poor SVM result.
• No tuning. C and gamma are fixed. An SVM with default hyper-parameters can be far from its best accuracy.
• Little interpretability. There are no coefficients or feature importances for the RBF kernel.
• Probabilities on small data. Platt scaling runs its own 5-fold CV inside each training set. With very few samples per class the probabilities, and the AUC, are unstable.

References

• Cortes C., Vapnik V. Support-vector networks. Machine Learning, 20, 273–297 (1995). doi:10.1007/BF00994018
• Platt J. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In: Advances in Large Margin Classifiers, MIT Press, 61–74 (1999).
• scikit-learn user guide: Support Vector Machines.