XGBoost — Extreme Gradient Boosting

Task	Classification
Method key (analysis_type)	xgboost
Prediction on new data	Yes — see Prediction
Library	xgboost.XGBClassifier (parameters)

When to use

• Larger data sets (hundreds of samples and more) with non-linear relations.
• You are ready to tune hyper-parameters: XGBoost has many, and the defaults are rarely the best choice for small chemometric data sets.

For small data sets, random forest or PLS-DA are usually more robust.

How it works

Gradient boosting builds decision trees one after another. Each new tree $f_m$ is fitted to correct the errors of the trees before it, and its prediction is added with a small weight, the learning rate $\eta$:

$$F_m(x) = F_{m-1}(x) + \eta\, f_m(x) .$$

XGBoost chooses each tree by minimising a regularised objective: the log-loss of the predictions plus a penalty on the tree complexity

$$\Omega(f) = \gamma\, T + \tfrac{1}{2} \lambda \lVert w \rVert^2 + \alpha \lVert w \rVert_1 ,$$

where $T$ is the number of leaves and $w$ the leaf values. For two classes the model outputs a probability through the logistic function; for more classes, one set of trees per class and a softmax.

Early stopping. If the score on an evaluation set has not improved for early_stopping_rounds trees, training stops and the best number of trees is kept.

Parameters

Defaults as set by the analysis dialog.

Parameter	UI label	Default	Notes
n_estimators	Number of Trees	100	Maximum number of trees
learning_rate	Learning Rate	0.1	$\eta$
max_depth	Max Depth	6	Depth of each tree
min_child_weight	Min Child Weight	1	Minimum sum of instance weights in a leaf
subsample	Subsample	1.0	Share of rows used for each tree
colsample_bytree	Column Sample	1.0	Share of features used for each tree
gamma	Gamma	0	$\gamma$: minimum loss reduction for a split
reg_alpha	Alpha (L1)	0	$\alpha$
reg_lambda	Lambda (L2)	1	$\lambda$
early_stopping_rounds	Early Stopping Rounds	10	0 turns early stopping off. See the warning below.
eval_metric	Evaluation Metric	mlogloss	Metric for early stopping. Converted automatically: for two classes mlogloss → logloss, merror → error.
objective	Objective Function	multi:softprob	For two classes the backend uses binary:logistic unless you set the objective explicitly. A binary objective with more than two classes is replaced by multi:softprob.
random_state	—	42
feature_selection	Enable Feature Selection	off	See Feature selection.
average_replicates	Average technical replicates	off	See Averaging replicates.

The model always runs with n_jobs=1.

note

The dialog also shows Tree Method and Booster. In the current version these values are saved with the analysis but not passed to the model: XGBoost uses its defaults (booster='gbtree', automatic tree method).

Preprocessing

• Class labels are encoded as integers 0, 1, … with LabelEncoder; results show the original labels.
• No scaling: trees do not need it.
• Before the model: replicate averaging and feature selection, if they are on. See Order of steps.

:::warning Early stopping and cross-validation

With early_stopping_rounds > 0 (the default is 10), each CV fold uses its own test fold as the early-stopping evaluation set. The number of trees is then chosen by looking at the test data, and the CV metrics are optimistic. To get an honest CV estimate, set Early Stopping Rounds to 0.

The final model holds out a random 10 % of the rows for early stopping (stratified by class only with Stratified K-Fold) and is trained on the other 90 %. This happens only when there are more than twice as many rows as folds.

:::

Results and metrics

The common classification metrics are described in Metrics. AUC is not computed for XGBoost.

XGBoost-specific fields:

Field	Meaning
feature_importances	Importance of each feature in the final model (feature_importances_, XGBoost default importance type)
cv_feature_importances	The same importances, averaged over the CV fold models
prediction_probabilities	Class probabilities for every sample (final model)
xgboost_parameters	All parameters actually passed to XGBClassifier
best_iteration, best_score	Best number of trees and its evaluation score, if early stopping was used

Visualizations

The visualization dialog offers no plots for XGBoost in the current version.

Source code

run_xgboost_analysis in chrometrica/analysis/analysis.py. The imports, the parameters, the model in the CV loop (with early stopping) and the final model:

analysis.py · run_xgboost_analysis() · lines 6259–6498

def run_xgboost_analysis(X, y, parameters, cv_method, cv_folds):
    """Run XGBoost analysis with cross-validation and optional feature selection"""
    import xgboost as xgb
    from sklearn.model_selection import StratifiedKFold, GroupKFold
    from sklearn.metrics import classification_report, confusion_matrix, accuracy_score
    from sklearn.preprocessing import LabelEncoder
    # ...
    xgb_params = {
        'objective': parameters.get('objective', 'multi:softprob'),
        'learning_rate': parameters.get('learning_rate', 0.1),
        'max_depth': parameters.get('max_depth', 6),
        'min_child_weight': parameters.get('min_child_weight', 1),
        'subsample': parameters.get('subsample', 1.0),
        'colsample_bytree': parameters.get('colsample_bytree', 1.0),
        'gamma': parameters.get('gamma', 0),
        'reg_alpha': parameters.get('reg_alpha', 0),
        'reg_lambda': parameters.get('reg_lambda', 1),
        'n_estimators': parameters.get('n_estimators', 100),
        'random_state': parameters.get('random_state', 42),
        'n_jobs': 1,  # Use 1 CPU core to avoid multiprocessing issues with Celery
        'eval_metric': parameters.get('eval_metric', 'mlogloss'),
        'num_class': len(class_names),
        'use_label_encoder': False,  # Disable label encoder warning
        'early_stopping_rounds': parameters.get('early_stopping_rounds', 10)
    }

    # Make objective, num_class and eval_metric consistent with the number of classes
    # (xgboost fails on 'binary:*' objective with num_class, and on binary metrics
    # 'error'/'logloss' with 'multi:*' objectives and vice versa)
    if len(class_names) == 2:
        xgb_params['objective'] = parameters.get('objective', 'binary:logistic')

    if len(class_names) > 2 and str(xgb_params['objective']).startswith('binary:'):
        logger.warning(f"XGBoost: objective '{xgb_params['objective']}' is binary, but there are "
                       f"{len(class_names)} classes - using 'multi:softprob'")
        xgb_params['objective'] = 'multi:softprob'

    if str(xgb_params['objective']).startswith('binary:'):
        # Binary objective: no num_class, binary metrics
        xgb_params.pop('num_class', None)
        metric_map = {'mlogloss': 'logloss', 'merror': 'error'}
    else:
        # Multi-class objective (also valid for 2 classes): multi-class metrics
        metric_map = {'logloss': 'mlogloss', 'error': 'merror'}
    xgb_params['eval_metric'] = metric_map.get(xgb_params['eval_metric'], xgb_params['eval_metric'])
    # ...
    for fold_idx, (train_idx, test_idx) in enumerate(cv.split(X_processed, y_encoded, groups=groups_data)):
        X_train, X_test = X_processed[train_idx], X_processed[test_idx]
        y_train, y_test = y_encoded[train_idx], y_encoded[test_idx]
        y_test_original = y[test_idx]

        # Train and predict
        xgb_cv = xgb.XGBClassifier(**xgb_params)

        # Use early stopping if validation set is available
        early_stopping_rounds = xgb_params.get('early_stopping_rounds', 10)
        if early_stopping_rounds > 0:
            xgb_cv.fit(
                X_train, y_train,
                eval_set=[(X_test, y_test)],
                verbose=False
            )
        else:
            xgb_cv.fit(X_train, y_train)
    # ...
    # Fit final model on all data
    xgb_final = xgb.XGBClassifier(**xgb_params)

    # Use a small validation split for early stopping if specified
    early_stopping_rounds = xgb_params.get('early_stopping_rounds', 10)
    if early_stopping_rounds > 0 and X_processed.shape[0] > cv_folds * 2:
        from sklearn.model_selection import train_test_split
        X_train_full, X_val, y_train_full, y_val = train_test_split(
            X_processed, y_encoded,
            test_size=0.1,
            stratify=y_encoded if cv_method == 'stratified' else None,
            random_state=xgb_params['random_state']
        )
        xgb_final.fit(
            X_train_full, y_train_full,
            eval_set=[(X_val, y_val)],
            verbose=False
        )
    else:
        xgb_final.fit(X_processed, y_encoded)

Full source of run_xgboost_analysis()

analysis.py · run_xgboost_analysis() · lines 6259–6660

def run_xgboost_analysis(X, y, parameters, cv_method, cv_folds):
    """Run XGBoost analysis with cross-validation and optional feature selection"""
    import xgboost as xgb
    from sklearn.model_selection import StratifiedKFold, GroupKFold
    from sklearn.metrics import classification_report, confusion_matrix, accuracy_score
    from sklearn.preprocessing import LabelEncoder
    import numpy as np

    if parameters.get('average_replicates', False):
        groups = parameters.get('groups')
        if groups is None:
            raise ValueError("'groups' must be provided when averaging replicates is enabled.")
        X, y, new_groups = average_replicates(X, y, groups, method='mean')
        # Update parameters with new groups (now each row is unique)
        parameters['groups'] = new_groups
        # If you used group CV, you might want to keep it; but with unique groups,
        # GroupKFold becomes equivalent to standard KFold. You can leave it as is.


    # Get overall classes for consistent confusion matrix dimensions
    overall_classes = np.unique(y)
    overall_classes_list = overall_classes.tolist()
    n_overall_classes = len(overall_classes_list)

    # Check for feature selection
    feature_selection_params = get_feature_selection_params(parameters)
    if feature_selection_params:
        X_processed, selected_features, feature_scores = select_features(
            X, y, feature_selection_params, 'classification'
        )
    else:
        X_processed = X
        selected_features = list(range(X.shape[1]))
        feature_scores = None

    # Encode string labels to integers for XGBoost
    le = LabelEncoder()
    y_encoded = le.fit_transform(y)
    class_names = le.classes_.tolist()

    # Get XGBoost parameters from analysis parameters with sensible defaults
    xgb_params = {
        'objective': parameters.get('objective', 'multi:softprob'),
        'learning_rate': parameters.get('learning_rate', 0.1),
        'max_depth': parameters.get('max_depth', 6),
        'min_child_weight': parameters.get('min_child_weight', 1),
        'subsample': parameters.get('subsample', 1.0),
        'colsample_bytree': parameters.get('colsample_bytree', 1.0),
        'gamma': parameters.get('gamma', 0),
        'reg_alpha': parameters.get('reg_alpha', 0),
        'reg_lambda': parameters.get('reg_lambda', 1),
        'n_estimators': parameters.get('n_estimators', 100),
        'random_state': parameters.get('random_state', 42),
        'n_jobs': 1,  # Use 1 CPU core to avoid multiprocessing issues with Celery
        'eval_metric': parameters.get('eval_metric', 'mlogloss'),
        'num_class': len(class_names),
        'use_label_encoder': False,  # Disable label encoder warning
        'early_stopping_rounds': parameters.get('early_stopping_rounds', 10)
    }

    # Make objective, num_class and eval_metric consistent with the number of classes
    # (xgboost fails on 'binary:*' objective with num_class, and on binary metrics
    # 'error'/'logloss' with 'multi:*' objectives and vice versa)
    if len(class_names) == 2:
        xgb_params['objective'] = parameters.get('objective', 'binary:logistic')

    if len(class_names) > 2 and str(xgb_params['objective']).startswith('binary:'):
        logger.warning(f"XGBoost: objective '{xgb_params['objective']}' is binary, but there are "
                       f"{len(class_names)} classes - using 'multi:softprob'")
        xgb_params['objective'] = 'multi:softprob'

    if str(xgb_params['objective']).startswith('binary:'):
        # Binary objective: no num_class, binary metrics
        xgb_params.pop('num_class', None)
        metric_map = {'mlogloss': 'logloss', 'merror': 'error'}
    else:
        # Multi-class objective (also valid for 2 classes): multi-class metrics
        metric_map = {'logloss': 'mlogloss', 'error': 'merror'}
    xgb_params['eval_metric'] = metric_map.get(xgb_params['eval_metric'], xgb_params['eval_metric'])

    # Enhanced cross-validation with metrics
    cv_scores = []
    cv_predictions = []  # Store predictions from each fold
    cv_true_labels = []  # Store true labels from each fold
    cv_class_reports = []  # Store classification reports from each fold
    cv_confusion_matrices = []  # Store confusion matrices for each fold (using overall classes)
    cv_feature_importances = []  # Store feature importances from each fold

    # Sensitivity and specificity tracking for binary classification
    cv_sensitivity_scores = []  # Store sensitivity from each fold (for binary)
    cv_specificity_scores = []  # Store specificity from each fold (for binary)

    if cv_method == 'stratified':
        cv = StratifiedKFold(n_splits=cv_folds, shuffle=True, random_state=xgb_params['random_state'])
    elif cv_method == 'group':
        groups = parameters.get('groups')
        if groups is None:
            raise ValueError("Groups parameter required for group cross-validation")
        cv = GroupKFold(n_splits=cv_folds)
    else:
        # Simple K-fold
        from sklearn.model_selection import KFold
        cv = KFold(n_splits=cv_folds, shuffle=True, random_state=xgb_params['random_state'])

    # For group CV, we need to handle groups in the loop
    groups_data = parameters.get('groups') if cv_method == 'group' else None

    for fold_idx, (train_idx, test_idx) in enumerate(cv.split(X_processed, y_encoded, groups=groups_data)):
        X_train, X_test = X_processed[train_idx], X_processed[test_idx]
        y_train, y_test = y_encoded[train_idx], y_encoded[test_idx]
        y_test_original = y[test_idx]

        # Train and predict
        xgb_cv = xgb.XGBClassifier(**xgb_params)

        # Use early stopping if validation set is available
        early_stopping_rounds = xgb_params.get('early_stopping_rounds', 10)
        if early_stopping_rounds > 0:
            xgb_cv.fit(
                X_train, y_train,
                eval_set=[(X_test, y_test)],
                verbose=False
            )
        else:
            xgb_cv.fit(X_train, y_train)

        # Predict probabilities and classes
        y_pred_encoded = xgb_cv.predict(X_test)

        # Ensure y_pred_encoded is 1D array
        if y_pred_encoded.ndim > 1:
            # If it's 2D, convert to 1D by taking argmax
            if y_pred_encoded.shape[1] > 1:
                y_pred_encoded = np.argmax(y_pred_encoded, axis=1)
            else:
                y_pred_encoded = y_pred_encoded.flatten()

        # Convert back to original class labels
        y_pred = le.inverse_transform(y_pred_encoded)

        # Store fold results
        fold_accuracy = accuracy_score(y_test_original, y_pred)
        cv_scores.append(fold_accuracy)
        cv_predictions.extend(y_pred)
        cv_true_labels.extend(y_test_original)

        # Store per-fold classification report
        fold_report = classification_report(y_test_original, y_pred, output_dict=True, zero_division=0)
        cv_class_reports.append(fold_report)

        # Create confusion matrix using overall classes to ensure consistent dimensions
        cm = confusion_matrix(y_test_original, y_pred, labels=overall_classes_list)
        cv_confusion_matrices.append(cm)

        # Calculate sensitivity and specificity for binary classification using the full 2x2 matrix
        if n_overall_classes == 2:
            tn, fp, fn, tp = cm.ravel()
            sensitivity = tp / (tp + fn) if (tp + fn) > 0 else 0
            specificity = tn / (tn + fp) if (tn + fp) > 0 else 0
            cv_sensitivity_scores.append(sensitivity)
            cv_specificity_scores.append(specificity)
        else:
            # For multiclass, we can't calculate single sensitivity/specificity values per fold
            cv_sensitivity_scores.append(None)
            cv_specificity_scores.append(None)

        # Store feature importances from this fold
        if hasattr(xgb_cv, 'feature_importances_'):
            cv_feature_importances.append(xgb_cv.feature_importances_)

    # Calculate CV-based metrics
    cv_precision_scores = []
    cv_recall_scores = []
    cv_f1_scores = []

    # Track if this is binary classification
    is_binary = n_overall_classes == 2

    for report in cv_class_reports:
        # Extract macro averages from each fold
        if 'macro avg' in report:
            cv_precision_scores.append(report['macro avg']['precision'])
            cv_recall_scores.append(report['macro avg']['recall'])
            cv_f1_scores.append(report['macro avg']['f1-score'])

    # Calculate overall CV confusion matrix (aggregated)
    cv_confusion_matrix_aggregated = confusion_matrix(cv_true_labels, cv_predictions,
                                                      labels=overall_classes_list).tolist()

    # Calculate overall CV sensitivity and specificity from aggregated confusion matrix
    if is_binary:
        cm_agg = np.array(cv_confusion_matrix_aggregated)
        tn, fp, fn, tp = cm_agg.ravel()
        cv_sensitivity_aggregated = tp / (tp + fn) if (tp + fn) > 0 else 0
        cv_specificity_aggregated = tn / (tn + fp) if (tn + fp) > 0 else 0
    else:
        cv_sensitivity_aggregated = None
        cv_specificity_aggregated = None

    # Calculate average confusion matrix using the outer function
    if cv_confusion_matrices:
        cv_confusion_matrix_avg, avg_matrix_classes = create_average_confusion_matrix(
            cv_confusion_matrices, [overall_classes_list] * len(cv_confusion_matrices)
        )

        # Calculate average sensitivity and specificity from averaged confusion matrix (binary only)
        if is_binary:
            cm_avg = np.array(cv_confusion_matrix_avg)
            tn, fp, fn, tp = cm_avg.ravel()
            cv_sensitivity_avg = tp / (tp + fn) if (tp + fn) > 0 else 0
            cv_specificity_avg = tn / (tn + fp) if (tn + fp) > 0 else 0
        else:
            cv_sensitivity_avg = None
            cv_specificity_avg = None
    else:
        cv_confusion_matrix_avg = []
        avg_matrix_classes = []
        cv_sensitivity_avg = None
        cv_specificity_avg = None

    # Fit final model on all data
    xgb_final = xgb.XGBClassifier(**xgb_params)

    # Use a small validation split for early stopping if specified
    early_stopping_rounds = xgb_params.get('early_stopping_rounds', 10)
    if early_stopping_rounds > 0 and X_processed.shape[0] > cv_folds * 2:
        from sklearn.model_selection import train_test_split
        X_train_full, X_val, y_train_full, y_val = train_test_split(
            X_processed, y_encoded,
            test_size=0.1,
            stratify=y_encoded if cv_method == 'stratified' else None,
            random_state=xgb_params['random_state']
        )
        xgb_final.fit(
            X_train_full, y_train_full,
            eval_set=[(X_val, y_val)],
            verbose=False
        )
    else:
        xgb_final.fit(X_processed, y_encoded)

    # Predict on all data
    y_pred_encoded_full = xgb_final.predict(X_processed)

    # Ensure y_pred_encoded_full is 1D array
    if y_pred_encoded_full.ndim > 1:
        # If it's 2D, convert to 1D by taking argmax
        if y_pred_encoded_full.shape[1] > 1:
            y_pred_encoded_full = np.argmax(y_pred_encoded_full, axis=1)
        else:
            y_pred_encoded_full = y_pred_encoded_full.flatten()

    y_pred_proba_full = xgb_final.predict_proba(X_processed)
    y_pred_full = le.inverse_transform(y_pred_encoded_full)

    # Get feature importances
    feature_importances = None
    if hasattr(xgb_final, 'feature_importances_'):
        feature_importances = xgb_final.feature_importances_.tolist()

        # Calculate average feature importances across CV folds
        if cv_feature_importances:
            avg_cv_feature_importances = np.mean(cv_feature_importances, axis=0).tolist()
        else:
            avg_cv_feature_importances = None

    # Get the final model's class order
    final_classes = class_names

    # Reorder the average confusion matrix to match final_classes if needed
    if cv_confusion_matrix_avg and avg_matrix_classes != final_classes:
        # Create mapping from avg_matrix_classes to final_classes
        avg_to_final = {cls: idx for idx, cls in enumerate(avg_matrix_classes)}

        # Create reordered confusion matrix
        n_classes = len(final_classes)
        reordered_matrix = np.zeros((n_classes, n_classes))

        for i, true_cls in enumerate(final_classes):
            for j, pred_cls in enumerate(final_classes):
                if true_cls in avg_to_final and pred_cls in avg_to_final:
                    orig_i = avg_to_final[true_cls]
                    orig_j = avg_to_final[pred_cls]
                    if (orig_i < len(cv_confusion_matrix_avg) and
                            orig_j < len(cv_confusion_matrix_avg[0])):
                        reordered_matrix[i, j] = cv_confusion_matrix_avg[orig_i][orig_j]

        cv_confusion_matrix_avg = reordered_matrix.tolist()
        avg_matrix_classes = final_classes

    # Calculate sensitivity and specificity for full model (self-test)
    full_confusion_matrix = confusion_matrix(y, y_pred_full, labels=final_classes)
    if is_binary and full_confusion_matrix.shape == (2, 2):
        tn, fp, fn, tp = full_confusion_matrix.ravel()
        full_sensitivity = tp / (tp + fn) if (tp + fn) > 0 else 0
        full_specificity = tn / (tn + fp) if (tn + fp) > 0 else 0
    else:
        full_sensitivity = None
        full_specificity = None

    # Validate consistency between metrics
    validation_message = None
    if is_binary and full_sensitivity is not None and full_specificity is not None:
        avg_sens_spec = (full_sensitivity + full_specificity) / 2
        accuracy = accuracy_score(y, y_pred_full)
        accuracy_diff = abs(accuracy - avg_sens_spec)
        if accuracy_diff > 0.01:  # Allow small floating point differences
            validation_message = f"Note: Accuracy ({accuracy:.3f}) differs from (sensitivity + specificity)/2 ({avg_sens_spec:.3f}) by {accuracy_diff:.3f}"

    # Calculate CV sensitivity mean and std (filtering out None values)
    cv_sensitivity_scores_filtered = [s for s in cv_sensitivity_scores if s is not None]
    if cv_sensitivity_scores_filtered:
        cv_sensitivity_mean = np.mean(cv_sensitivity_scores_filtered)
        cv_sensitivity_std = np.std(cv_sensitivity_scores_filtered)
    else:
        cv_sensitivity_mean = None
        cv_sensitivity_std = None

    # Calculate CV specificity mean and std (filtering out None values)
    cv_specificity_scores_filtered = [s for s in cv_specificity_scores if s is not None]
    if cv_specificity_scores_filtered:
        cv_specificity_mean = np.mean(cv_specificity_scores_filtered)
        cv_specificity_std = np.std(cv_specificity_scores_filtered)
    else:
        cv_specificity_mean = None
        cv_specificity_std = None

    cv_confusion_matrices_lists = [cm.tolist() for cm in cv_confusion_matrices]

    # Get additional XGBoost-specific information
    best_iteration = getattr(xgb_final, 'best_iteration', xgb_params.get('n_estimators', 100))
    best_score = getattr(xgb_final, 'best_score', None)

    results = {
        'method': 'XGBoost',
        'cv_scores': cv_scores,
        'cv_mean': np.mean(cv_scores) if cv_scores else 0.0,
        'cv_std': np.std(cv_scores) if cv_scores else 0.0,
        'cv_precision': np.mean(cv_precision_scores) if cv_precision_scores else 0.0,
        'cv_precision_std': np.std(cv_precision_scores) if cv_precision_scores else 0.0,
        'cv_recall': np.mean(cv_recall_scores) if cv_recall_scores else 0.0,
        'cv_recall_std': np.std(cv_recall_scores) if cv_recall_scores else 0.0,
        'cv_f1': np.mean(cv_f1_scores) if cv_f1_scores else 0.0,
        'cv_f1_std': np.std(cv_f1_scores) if cv_f1_scores else 0.0,
        # Sensitivity metrics
        'cv_sensitivity': cv_sensitivity_mean,
        'cv_sensitivity_std': cv_sensitivity_std,
        'cv_sensitivity_aggregated': cv_sensitivity_aggregated,
        'cv_sensitivity_avg': cv_sensitivity_avg,
        'cv_sensitivity_scores': cv_sensitivity_scores,  # Per-fold sensitivities
        # Specificity metrics
        'cv_specificity': cv_specificity_mean,
        'cv_specificity_std': cv_specificity_std,
        'cv_specificity_aggregated': cv_specificity_aggregated,
        'cv_specificity_avg': cv_specificity_avg,
        'cv_specificity_scores': cv_specificity_scores,  # Per-fold specificities
        # Validation flag
        'is_binary': is_binary,
        'validation_message': validation_message,
        # Full model metrics
        'sensitivity': full_sensitivity,
        'specificity': full_specificity,
        # Existing metrics
        'cv_confusion_matrix': cv_confusion_matrix_avg,
        'cv_confusion_matrices': cv_confusion_matrices_lists,
        'cv_confusion_matrix_aggregated': cv_confusion_matrix_aggregated,
        'cv_class_reports': cv_class_reports,
        'classification_report': classification_report(y, y_pred_full, output_dict=True, zero_division=0),
        'confusion_matrix': full_confusion_matrix.tolist(),
        'feature_importances': feature_importances,
        'cv_feature_importances': avg_cv_feature_importances if 'avg_cv_feature_importances' in locals() else None,
        'labels': y.tolist(),
        'predictions': y_pred_full.tolist(),
        'prediction_probabilities': y_pred_proba_full.tolist() if hasattr(xgb_final, 'predict_proba') else None,
        'classes': final_classes,
        'xgboost_parameters': {k: v for k, v in xgb_params.items() if k != 'early_stopping_rounds'},
        'early_stopping_rounds': xgb_params.get('early_stopping_rounds', 10),
        'best_iteration': best_iteration,
        'best_score': best_score if best_score is not None else None
    }

    # Add metric consistency check for binary classification
    if is_binary and full_sensitivity is not None and full_specificity is not None:
        results['metric_consistency_check'] = {
            'accuracy': accuracy_score(y, y_pred_full),
            'sensitivity': full_sensitivity,
            'specificity': full_specificity,
            'avg_sens_spec': (full_sensitivity + full_specificity) / 2,
            'difference': abs(accuracy_score(y, y_pred_full) - (full_sensitivity + full_specificity) / 2)
        }

    # Add feature selection info if used
    if feature_selection_params:
        results['feature_selection'] = {
            'selected_features': selected_features,
            'feature_scores': feature_scores,
            'method': feature_selection_params.get('method', 'anova'),
            'k': feature_selection_params.get('k', 'all')
        }

    # Final fitted model for inference on unknown samples (popped in run_analysis)
    return attach_model_bundle(results, xgb_final, selected_features, feature_selection_params, label_encoder=le)

Limitations and common pitfalls

• Early stopping on the test fold. See the warning in Preprocessing.
• Small data. With tens of samples, deep trees (max_depth = 6) and 100 rounds overfit easily. Lower Max Depth (2–3), raise Min Child Weight, or use fewer trees.
• No permutation test. The permutation plot is not available for XGBoost.

References

• Chen T., Guestrin C. XGBoost: a scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 785–794 (2016). doi:10.1145/2939672.2939785
• Friedman J. H. Greedy function approximation: a gradient boosting machine. Annals of Statistics, 29, 1189–1232 (2001). doi:10.1214/aos/1013203451
• XGBoost documentation: Introduction to boosted trees.