Logistic regression
| Task | Classification |
Method key (analysis_type) | logistic |
| Prediction on new data | Yes — see Prediction |
| Library | sklearn.linear_model.LogisticRegression, StandardScaler, Pipeline |
When to use
- You need class probabilities, not only labels.
- The classes can be separated by a straight boundary.
- You want coefficients that show how each feature moves the prediction.
Logistic regression makes fewer assumptions than LDA: it does not require normally distributed features or equal spreads in the classes.
How it works
For two classes, the model estimates the probability of the positive class as
The weights and the intercept minimise the log-loss plus an L2 penalty:
For more than two classes, scikit-learn fits a multinomial (softmax) model: one weight vector per class, and the probabilities of all classes add up to 1. A sample is assigned to the class with the highest probability.
Parameters
The model runs with random_state=42, max_iter=1000 and the scikit-learn
defaults for everything else: L2 penalty, C=1.0, solver lbfgs.
| Parameter | UI label | Default | Notes |
|---|---|---|---|
scale_data | — | true | Standardise features inside the model. Not shown in the dialog; always on. |
feature_selection | Enable Feature Selection | off | See Feature selection. |
average_replicates | Average technical replicates | off | See Averaging replicates. |
note
The dialog also shows Regularization (C) and Max Iterations. In the
current version these values are saved with the analysis but not passed to
the model: it always uses C=1.0 and max_iter=1000.
Preprocessing
- Class labels are encoded as integers with
LabelEncoder; results show the original labels. - Features are standardised (mean 0, standard deviation 1) by a
StandardScalerinside aPipeline. The scaler is fitted on the training rows of each fold only, so the test fold does not leak into the scaling. - Before the model: replicate averaging and feature selection, if they are on. See Order of steps.
Results and metrics
The common classification metrics are described in Metrics. For binary problems, AUC uses the predicted probability of the positive class.
Logistic-specific fields:
| Field | Meaning |
|---|---|
prediction_probabilities | Class probabilities for every sample (final model) |
coefficients | Weights of the final model, one row per class (one row for binary). They refer to standardised features: compare their sizes directly, a larger absolute value means a stronger effect. |
Visualizations
- Confusion matrix and ROC curve
- Permutation plot
- Time series and heatmap of the input data
Source code
run_logistic_analysis in chrometrica/analysis/analysis.py. The imports,
the model in the CV loop and the final model:
analysis.py · run_logistic_analysis() · lines 1375–1601
def run_logistic_analysis(X, y, parameters, cv_method, cv_folds):
"""Run Logistic Regression analysis with optional feature selection and scaling"""
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import LabelEncoder, StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.model_selection import StratifiedKFold, GroupKFold, KFold
from sklearn.metrics import classification_report, confusion_matrix, accuracy_score, roc_auc_score
import numpy as np
# ...
scale_data = parameters.get('scale_data', True)
# Encode labels for logistic regression
le = LabelEncoder()
y_encoded = le.fit_transform(y)
# ...
for train_idx, test_idx in cv.split(X_processed, y_encoded, groups=groups if cv_method == 'group' else None):
X_train, X_test = X_processed[train_idx], X_processed[test_idx]
y_train, y_test = y_encoded[train_idx], y_encoded[test_idx]
# Create and train model with optional scaling
if scale_data:
lr_cv = Pipeline([
('scaler', StandardScaler()),
('logistic', LogisticRegression(random_state=42, max_iter=1000))
])
else:
lr_cv = LogisticRegression(random_state=42, max_iter=1000)
lr_cv.fit(X_train, y_train)
y_pred = lr_cv.predict(X_test)
# ...
# Fit final model on all data
if scale_data:
lr = Pipeline([
('scaler', StandardScaler()),
('logistic', LogisticRegression(random_state=42, max_iter=1000))
])
else:
lr = LogisticRegression(random_state=42, max_iter=1000)
lr.fit(X_processed, y_encoded)
y_pred_full = lr.predict(X_processed)
y_pred_proba = lr.predict_proba(X_processed)
Full source of run_logistic_analysis()
analysis.py · run_logistic_analysis() · lines 1375–1735
def run_logistic_analysis(X, y, parameters, cv_method, cv_folds):
"""Run Logistic Regression analysis with optional feature selection and scaling"""
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import LabelEncoder, StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.model_selection import StratifiedKFold, GroupKFold, KFold
from sklearn.metrics import classification_report, confusion_matrix, accuracy_score, roc_auc_score
import numpy as np
# ---------- Helper to sanitize JSON ----------
def sanitize_for_json(obj):
if isinstance(obj, dict):
return {k: sanitize_for_json(v) for k, v in obj.items()}
elif isinstance(obj, list):
return [sanitize_for_json(v) for v in obj]
elif isinstance(obj, float) and np.isnan(obj):
return None
elif isinstance(obj, (np.float32, np.float64)) and np.isnan(obj):
return None
elif isinstance(obj, np.ndarray):
return sanitize_for_json(obj.tolist())
else:
return obj
if parameters.get('average_replicates', False):
groups = parameters.get('groups')
if groups is None:
raise ValueError("'groups' must be provided when averaging replicates is enabled.")
X, y, new_groups = average_replicates(X, y, groups, method='mean')
parameters['groups'] = new_groups
# Get parameters with defaults
scale_data = parameters.get('scale_data', True)
# Encode labels for logistic regression
le = LabelEncoder()
y_encoded = le.fit_transform(y)
# Get overall classes for consistent confusion matrix dimensions
overall_classes = np.unique(y)
overall_classes_list = overall_classes.tolist()
n_overall_classes = len(overall_classes_list)
is_binary = (n_overall_classes == 2)
# Check for feature selection
feature_selection_params = get_feature_selection_params(parameters)
if feature_selection_params:
X_processed, selected_features, feature_scores = select_features(
X, y_encoded, feature_selection_params, 'classification'
)
else:
X_processed = X
selected_features = list(range(X.shape[1]))
feature_scores = None
# --- AUC initialisation (only used if binary) ---
cv_auc_scores = [] # AUC per fold
cv_scores_all = [] # decision scores / probabilities for all test samples
# Enhanced cross-validation with metrics
cv_scores = []
cv_predictions = []
cv_true_labels = []
cv_class_reports = []
cv_confusion_matrices = []
cv_sensitivity_scores = []
cv_specificity_scores = []
if cv_method == 'stratified':
cv = StratifiedKFold(n_splits=cv_folds, shuffle=True, random_state=42)
elif cv_method == 'group':
groups = parameters.get('groups')
if groups is None:
raise ValueError("Groups parameter required for group cross-validation")
cv = GroupKFold(n_splits=cv_folds)
else:
cv = KFold(n_splits=cv_folds, shuffle=True, random_state=42)
for train_idx, test_idx in cv.split(X_processed, y_encoded, groups=groups if cv_method == 'group' else None):
X_train, X_test = X_processed[train_idx], X_processed[test_idx]
y_train, y_test = y_encoded[train_idx], y_encoded[test_idx]
# Create and train model with optional scaling
if scale_data:
lr_cv = Pipeline([
('scaler', StandardScaler()),
('logistic', LogisticRegression(random_state=42, max_iter=1000))
])
else:
lr_cv = LogisticRegression(random_state=42, max_iter=1000)
lr_cv.fit(X_train, y_train)
y_pred = lr_cv.predict(X_test)
# ---------- AUC computation only for binary ----------
if is_binary:
# For binary we use predict_proba (positive class probability) or decision_function
if hasattr(lr_cv, "predict_proba"):
y_prob = lr_cv.predict_proba(X_test)
y_score = y_prob[:, 1] # positive class probability
elif hasattr(lr_cv, "decision_function"):
y_score = lr_cv.decision_function(X_test)
else:
y_score = None
if y_score is not None:
try:
auc_fold = roc_auc_score(y_test, y_score)
except ValueError:
auc_fold = None
cv_auc_scores.append(auc_fold)
if isinstance(y_score, np.ndarray):
cv_scores_all.extend(y_score.tolist())
else:
cv_scores_all.extend(y_score)
else:
cv_auc_scores.append(None)
else:
# Multiclass: skip AUC entirely
cv_auc_scores.append(None)
# Store fold results
cv_scores.append(accuracy_score(y_test, y_pred))
cv_predictions.extend(y_pred)
cv_true_labels.extend(y_test)
# Store per-fold classification report with original class labels
fold_report = classification_report(y_test, y_pred, output_dict=True, zero_division=0)
original_fold_report = {}
for key, value in fold_report.items():
if key.isdigit():
class_idx = int(key)
if class_idx < len(le.classes_):
original_key = le.classes_[class_idx]
original_fold_report[original_key] = value
else:
original_fold_report[key] = value
cv_class_reports.append(original_fold_report)
cm = confusion_matrix(y_test, y_pred, labels=range(n_overall_classes))
cv_confusion_matrices.append(cm)
if is_binary:
tn, fp, fn, tp = cm.ravel()
sensitivity = tp / (tp + fn) if (tp + fn) > 0 else 0
specificity = tn / (tn + fp) if (tn + fp) > 0 else 0
cv_sensitivity_scores.append(sensitivity)
cv_specificity_scores.append(specificity)
else:
cv_sensitivity_scores.append(None)
cv_specificity_scores.append(None)
# --- After CV loop: compute aggregated AUC only for binary ---
auc_aggregated = None
if is_binary and cv_scores_all:
try:
auc_aggregated = roc_auc_score(cv_true_labels, cv_scores_all)
except ValueError:
auc_aggregated = None
if auc_aggregated is not None and np.isnan(auc_aggregated):
auc_aggregated = None
# Compute mean/std of per-fold AUC (only binary)
if is_binary:
valid_auc = [a for a in cv_auc_scores if a is not None]
cv_auc_mean = np.mean(valid_auc) if valid_auc else None
cv_auc_std = np.std(valid_auc) if valid_auc else None
else:
cv_auc_mean = None
cv_auc_std = None
# ---- Continue with existing CV metrics (unchanged) ----
cv_precision_scores = []
cv_recall_scores = []
cv_f1_scores = []
for report in cv_class_reports:
if 'macro avg' in report:
cv_precision_scores.append(report['macro avg']['precision'])
cv_recall_scores.append(report['macro avg']['recall'])
cv_f1_scores.append(report['macro avg']['f1-score'])
# Aggregated confusion matrix with original labels
cv_true_labels_original = le.inverse_transform(cv_true_labels)
cv_predictions_original = le.inverse_transform(cv_predictions)
cv_confusion_matrix_aggregated = confusion_matrix(cv_true_labels_original, cv_predictions_original,
labels=overall_classes_list).tolist()
if is_binary:
cm_agg = np.array(cv_confusion_matrix_aggregated)
tn, fp, fn, tp = cm_agg.ravel()
cv_sensitivity_aggregated = tp / (tp + fn) if (tp + fn) > 0 else 0
cv_specificity_aggregated = tn / (tn + fp) if (tn + fp) > 0 else 0
else:
cv_sensitivity_aggregated = None
cv_specificity_aggregated = None
if cv_confusion_matrices:
cv_confusion_matrix_avg, avg_matrix_classes = create_average_confusion_matrix(
cv_confusion_matrices, [overall_classes_list] * len(cv_confusion_matrices)
)
if is_binary:
cm_avg = np.array(cv_confusion_matrix_avg)
tn, fp, fn, tp = cm_avg.ravel()
cv_sensitivity_avg = tp / (tp + fn) if (tp + fn) > 0 else 0
cv_specificity_avg = tn / (tn + fp) if (tn + fp) > 0 else 0
else:
cv_sensitivity_avg = None
cv_specificity_avg = None
else:
cv_confusion_matrix_avg = []
avg_matrix_classes = []
cv_sensitivity_avg = None
cv_specificity_avg = None
# Fit final model on all data
if scale_data:
lr = Pipeline([
('scaler', StandardScaler()),
('logistic', LogisticRegression(random_state=42, max_iter=1000))
])
else:
lr = LogisticRegression(random_state=42, max_iter=1000)
lr.fit(X_processed, y_encoded)
y_pred_full = lr.predict(X_processed)
y_pred_proba = lr.predict_proba(X_processed)
if scale_data:
lr_model = lr.named_steps['logistic']
else:
lr_model = lr
final_classes = le.classes_.tolist()
# Reorder average confusion matrix if needed
if cv_confusion_matrix_avg and avg_matrix_classes != final_classes:
avg_to_final = {cls: idx for idx, cls in enumerate(avg_matrix_classes)}
n_classes = len(final_classes)
reordered_matrix = np.zeros((n_classes, n_classes))
for i, true_cls in enumerate(final_classes):
for j, pred_cls in enumerate(final_classes):
if true_cls in avg_to_final and pred_cls in avg_to_final:
orig_i = avg_to_final[true_cls]
orig_j = avg_to_final[pred_cls]
if (orig_i < len(cv_confusion_matrix_avg) and
orig_j < len(cv_confusion_matrix_avg[0])):
reordered_matrix[i, j] = cv_confusion_matrix_avg[orig_i][orig_j]
cv_confusion_matrix_avg = reordered_matrix.tolist()
avg_matrix_classes = final_classes
y_pred_full_original = le.inverse_transform(y_pred_full)
full_confusion_matrix = confusion_matrix(y, y_pred_full_original, labels=final_classes)
if is_binary and full_confusion_matrix.shape == (2, 2):
tn, fp, fn, tp = full_confusion_matrix.ravel()
full_sensitivity = tp / (tp + fn) if (tp + fn) > 0 else 0
full_specificity = tn / (tn + fp) if (tn + fp) > 0 else 0
else:
full_sensitivity = None
full_specificity = None
validation_message = None
if is_binary and full_sensitivity is not None and full_specificity is not None:
avg_sens_spec = (full_sensitivity + full_specificity) / 2
accuracy = accuracy_score(y, y_pred_full_original)
accuracy_diff = abs(accuracy - avg_sens_spec)
if accuracy_diff > 0.01:
validation_message = f"Note: Accuracy ({accuracy:.3f}) differs from (sensitivity + specificity)/2 ({avg_sens_spec:.3f}) by {accuracy_diff:.3f}"
cv_sensitivity_scores_filtered = [s for s in cv_sensitivity_scores if s is not None]
cv_sensitivity_mean = np.mean(cv_sensitivity_scores_filtered) if cv_sensitivity_scores_filtered else None
cv_sensitivity_std = np.std(cv_sensitivity_scores_filtered) if cv_sensitivity_scores_filtered else None
cv_specificity_scores_filtered = [s for s in cv_specificity_scores if s is not None]
cv_specificity_mean = np.mean(cv_specificity_scores_filtered) if cv_specificity_scores_filtered else None
cv_specificity_std = np.std(cv_specificity_scores_filtered) if cv_specificity_scores_filtered else None
# --- Full-data (resubstitution) AUC and accuracy, by analogy with run_lda_analysis ---
full_auc = None
if is_binary:
try:
pos_idx = final_classes.index(overall_classes_list[1])
full_auc = roc_auc_score(y, y_pred_proba[:, pos_idx])
except Exception:
full_auc = None
full_accuracy = accuracy_score(y, y_pred_full_original)
results = {
'method': 'Logistic Regression',
'cv_scores': cv_scores,
'cv_mean': np.mean(cv_scores) if cv_scores else 0.0,
'cv_std': np.std(cv_scores) if cv_scores else 0.0,
'cv_precision': np.mean(cv_precision_scores) if cv_precision_scores else 0.0,
'cv_precision_std': np.std(cv_precision_scores) if cv_precision_scores else 0.0,
'cv_recall': np.mean(cv_recall_scores) if cv_recall_scores else 0.0,
'cv_recall_std': np.std(cv_recall_scores) if cv_recall_scores else 0.0,
'cv_f1': np.mean(cv_f1_scores) if cv_f1_scores else 0.0,
'cv_f1_std': np.std(cv_f1_scores) if cv_f1_scores else 0.0,
# Sensitivity
'cv_sensitivity': cv_sensitivity_mean,
'cv_sensitivity_std': cv_sensitivity_std,
'cv_sensitivity_aggregated': cv_sensitivity_aggregated,
'cv_sensitivity_avg': cv_sensitivity_avg,
'cv_sensitivity_scores': cv_sensitivity_scores,
# Specificity
'cv_specificity': cv_specificity_mean,
'cv_specificity_std': cv_specificity_std,
'cv_specificity_aggregated': cv_specificity_aggregated,
'cv_specificity_avg': cv_specificity_avg,
'cv_specificity_scores': cv_specificity_scores,
# AUC (sanitized; only for binary)
'cv_auc': sanitize_for_json(cv_auc_mean),
'cv_auc_std': sanitize_for_json(cv_auc_std),
'cv_auc_scores': sanitize_for_json(cv_auc_scores),
'cv_auc_aggregated': sanitize_for_json(auc_aggregated),
# Full-data AUC / accuracy (+ CV accuracy alias), field parity with LDA
'auc': sanitize_for_json(full_auc),
'accuracy': full_accuracy,
'cv_accuracy': np.mean(cv_scores) if cv_scores else 0.0,
# Validation
'is_binary': is_binary,
'validation_message': validation_message,
# Confusion matrices
'cv_confusion_matrix': cv_confusion_matrix_avg,
'cv_confusion_matrices': [cm.tolist() for cm in cv_confusion_matrices],
'cv_confusion_matrix_aggregated': cv_confusion_matrix_aggregated,
'cv_class_reports': cv_class_reports,
'classification_report': classification_report(y, y_pred_full_original, output_dict=True, zero_division=0),
'confusion_matrix': full_confusion_matrix.tolist(),
'sensitivity': full_sensitivity,
'specificity': full_specificity,
'labels': y.tolist(),
'predictions': y_pred_full_original.tolist(),
'prediction_probabilities': y_pred_proba.tolist(),
'classes': final_classes,
'coefficients': lr_model.coef_.tolist() if hasattr(lr_model, 'coef_') else None,
'model_info': {'scale_data': scale_data}
}
if feature_selection_params:
results['feature_selection'] = {
'selected_features': selected_features,
'feature_scores': feature_scores,
'method': feature_selection_params.get('method', 'anova'),
'k': feature_selection_params.get('k', 'all')
}
if is_binary and full_sensitivity is not None and full_specificity is not None:
results['metric_consistency_check'] = {
'accuracy': accuracy_score(y, y_pred_full_original),
'sensitivity': full_sensitivity,
'specificity': full_specificity,
'avg_sens_spec': (full_sensitivity + full_specificity) / 2,
'difference': abs(accuracy_score(y, y_pred_full_original) - (full_sensitivity + full_specificity) / 2)
}
results = sanitize_for_json(results)
# Final fitted model for inference on unknown samples (popped in run_analysis)
return attach_model_bundle(results, lr, selected_features, feature_selection_params, label_encoder=le)
Limitations and common pitfalls
- Linear boundary. Like LDA, logistic regression separates classes with hyperplanes.
- Perfect separation. When the training classes are perfectly separable, the unpenalised weights would grow without limit. The L2 penalty keeps them finite, but the probabilities become very close to 0 and 1 and look more certain than they are.
- Correlated features. Colour channels are often strongly correlated. The predictions stay good, but the individual coefficients become unstable and hard to interpret.
References
- Cox D. R. The regression analysis of binary sequences. Journal of the Royal Statistical Society B, 20, 215–242 (1958).
- Hastie T., Tibshirani R., Friedman J. The Elements of Statistical Learning, 2nd ed., section 4.4. Springer (2009).
- scikit-learn user guide: Logistic regression.