Validation & Cross-Validation¶

Metrics¶

Regression¶

Regression metric utilities built on top of NumPy.

aurora.validation.metrics.regression.mean_squared_error(y_true, y_pred, *, sample_weight=None, squared=True)[source]¶

Compute the (weighted) mean squared error between true and predicted values.

Parameters:

y_true (array-like) – True target values. Supports NumPy arrays, PyTorch tensors, JAX arrays, and any array-like object convertible to NumPy.
y_pred (array-like) – Predicted values. Must have the same shape as y_true.
sample_weight (array-like, optional) – Sample weights. If provided, the weighted mean is computed. Must be broadcastable to the shape of y_true.
squared (bool, default=True) – If True, return the mean squared error (MSE). If False, return the root mean squared error (RMSE).

Returns:

The (weighted) mean squared error or its square root.

Return type:

float

See also

root_mean_squared_error: Convenience wrapper always returning RMSE.
mean_absolute_error: Mean absolute error (less sensitive to outliers).
r_squared: Coefficient of determination.

aurora.validation.metrics.regression.mean_absolute_error(y_true, y_pred, *, sample_weight=None)[source]¶

Compute the (weighted) mean absolute error between true and predicted values.

Parameters:

y_true (array-like) – True target values. Supports NumPy arrays, PyTorch tensors, JAX arrays, and any array-like object convertible to NumPy.
y_pred (array-like) – Predicted values. Must have the same shape as y_true.
sample_weight (array-like, optional) – Sample weights. If provided, the weighted mean is computed. Must be broadcastable to the shape of y_true.

Returns:

The (weighted) mean absolute error.

Return type:

float

See also

mean_squared_error: Mean squared error (more sensitive to outliers).
root_mean_squared_error: Root mean squared error.
r_squared: Coefficient of determination.

aurora.validation.metrics.regression.root_mean_squared_error(y_true, y_pred, *, sample_weight=None)[source]¶

Compute the (weighted) root mean squared error.

This is equivalent to mean_squared_error(..., squared=False).

Parameters:

y_true (array-like) – True target values.
y_pred (array-like) – Predicted values. Must have the same shape as y_true.
sample_weight (array-like, optional) – Sample weights. If provided, the weighted RMSE is computed.

Returns:

The (weighted) root mean squared error.

Return type:

float

See also

mean_squared_error: Mean squared error (optionally squared or not).
mean_absolute_error: Mean absolute error.
r_squared: Coefficient of determination.

aurora.validation.metrics.regression.r_squared(y_true, y_pred, *, sample_weight=None)[source]¶

Calculate R-squared (coefficient of determination).

Parameters:

y_true (array-like) – True target values.
y_pred (array-like) – Predicted values.
sample_weight (array-like, optional) – Sample weights.

Returns:

R-squared value. Best possible score is 1.0, can be negative (indicating the model performs worse than a horizontal line).

Return type:

float

Classification¶

Classification-oriented evaluation metrics.

This module provides standard metrics for evaluating binary and multi-class classification models, including accuracy, log-loss, Brier score, concordance index (AUC-ROC), precision, recall, F1, and confusion matrices.

All public functions accept NumPy arrays, PyTorch tensors, or any array-like input. Sample weights are supported where applicable.

Examples

>>> from aurora.validation.metrics.classification import accuracy_score, log_loss
>>> import numpy as np
>>> y_true = np.array([0, 1, 1, 0, 1])
>>> y_pred = np.array([0, 1, 0, 0, 1])
>>> accuracy_score(y_true, y_pred)
0.8

See also

aurora.validation.metrics: Top-level metrics interface.

aurora.validation.metrics.classification.accuracy_score(y_true, y_pred, *, sample_weight=None, normalize=True)[source]¶

Compute classification accuracy, optionally weighted.

Accuracy is the fraction of correct predictions among the total number of cases evaluated.

Parameters:

y_true (array-like of shape (n_samples,)) – True labels.
y_pred (array-like of shape (n_samples,)) – Predicted labels.
sample_weight (array-like of shape (n_samples,), optional) – Sample weights. If provided, the weighted fraction of correct predictions is returned when normalize=True.
normalize (bool, default=True) – If True, return the (weighted) fraction of correctly classified samples. If False, return the (weighted) count of correctly classified samples.

Returns:

accuracy – If normalize=True, a value in [0, 1]. Otherwise the (weighted) count of correct predictions.

Return type:

float

Raises:

ValueError – If y_true and y_pred have different shapes, or if sample_weight sums to zero or less.

Examples

>>> y_true = np.array([0, 1, 1, 0, 1])
>>> y_pred = np.array([0, 1, 0, 0, 1])
>>> accuracy_score(y_true, y_pred)
0.8
>>> accuracy_score(y_true, y_pred, normalize=False)
4.0

See also

confusion_matrix: Full confusion matrix for binary labels.
precision: Positive predictive value.
recall: True positive rate (sensitivity).

aurora.validation.metrics.classification.log_loss(y_true, y_prob, *, eps=1e-15, sample_weight=None)[source]¶

Compute the negative log-likelihood (log-loss) for probabilistic predictions.

Log-loss measures the divergence between predicted probabilities and the true labels. Lower values indicate better calibrated predictions.

Parameters:

y_true (array-like of shape (n_samples,)) – True labels. For binary classification, labels should be 0/1. For multi-class, integer labels in [0, n_classes).
y_prob (array-like) – Predicted probabilities. Shape must be either (n_samples,) for binary classification or (n_samples, n_classes) for multi-class classification.
eps (float, default=1e-15) – Clipping threshold for predicted probabilities to avoid log(0).
sample_weight (array-like of shape (n_samples,), optional) – Sample weights.

Returns:

loss – The (weighted) average log-loss.

Return type:

float

Raises:

ValueError – If y_true is not one-dimensional, y_prob has an unexpected shape, probability rows do not sum to 1, or labels are out of range.

Examples

>>> y_true = np.array([1, 0, 1])
>>> y_prob = np.array([0.9, 0.1, 0.8])
>>> log_loss(y_true, y_prob)
0.164252033486...

See also

brier_score_loss: Brier score for binary probabilistic predictions.
accuracy_score: Classification accuracy.

aurora.validation.metrics.classification.brier_score_loss(y_true, y_prob, *, sample_weight=None)[source]¶

Compute the Brier score for probabilistic binary predictions.

The Brier score is the mean squared difference between the predicted probabilities and the true binary labels.

\[\text{BS} = \frac{1}{n} \sum_{i=1}^{n} (y_i - p_i)^2\]

Lower values indicate better calibration and sharpness.

Parameters:

y_true (array-like of shape (n_samples,)) – True binary labels (must contain exactly the values 0 and 1).
y_prob (array-like of shape (n_samples,)) – Predicted probabilities for the positive class, in [0, 1].
sample_weight (array-like of shape (n_samples,), optional) – Sample weights.

Returns:

score – The (weighted) mean Brier score. Ranges from 0.0 (perfect) to 1.0 (worst).

Return type:

float

Raises:

ValueError – If y_prob is not one-dimensional or labels are not binary.

Examples

>>> y_true = np.array([1, 0, 1, 1])
>>> y_prob = np.array([0.9, 0.2, 0.8, 0.7])
>>> brier_score_loss(y_true, y_prob)
0.045000...

See also

log_loss: Negative log-likelihood loss.

aurora.validation.metrics.classification.concordance_index(y_true, y_score, *, sample_weight=None)[source]¶

Compute the concordance index (c-statistic) for binary outcomes.

The concordance index measures the probability that for a randomly selected pair of positive and negative samples, the positive sample has a higher predicted score than the negative sample.

Parameters:

y_true (array-like) – Binary labels (must contain exactly two unique values).
y_score (array-like) – Predicted scores or probabilities.
sample_weight (array-like, optional) – Sample weights. If provided, the concordance computation is weighted.

Returns:

The concordance index, ranging from 0.0 (perfect inverse discrimination) to 1.0 (perfect discrimination). A value of 0.5 indicates random predictions.

Return type:

float

Raises:

ValueError – If inputs have mismatched lengths, are empty, contain non-finite scores, or lack both positive and negative examples.

aurora.validation.metrics.classification.confusion_matrix(y_true, y_pred)[source]¶

Compute confusion matrix for binary classification.

Parameters:

y_true (array-like) – True binary labels (0 or 1).
y_pred (array-like) – Predicted binary labels (0 or 1).

Returns:

cm – Confusion matrix where cm[i, j] is the count of samples with true label i and predicted label j. [[TN, FP],

[FN, TP]]

Return type:

ndarray, shape (2, 2)

aurora.validation.metrics.classification.precision(y_true, y_pred, *, sample_weight=None)[source]¶

Compute precision (positive predictive value).

Precision = TP / (TP + FP)

Parameters:

y_true (array-like) – True binary labels.
y_pred (array-like) – Predicted binary labels.
sample_weight (array-like, optional) – Sample weights.

Returns:

precision – Precision score. Returns 0.0 if no positive predictions.

Return type:

float

aurora.validation.metrics.classification.recall(y_true, y_pred, *, sample_weight=None)[source]¶

Compute recall (sensitivity, true positive rate).

Recall = TP / (TP + FN)

Parameters:

y_true (array-like) – True binary labels.
y_pred (array-like) – Predicted binary labels.
sample_weight (array-like, optional) – Sample weights.

Returns:

recall – Recall score. Returns 0.0 if no positive samples.

Return type:

float

aurora.validation.metrics.classification.f1_score(y_true, y_pred, *, sample_weight=None)[source]¶

Compute F1 score (harmonic mean of precision and recall).

F1 = 2 * (precision * recall) / (precision + recall)

Parameters:

y_true (array-like) – True binary labels.
y_pred (array-like) – Predicted binary labels.
sample_weight (array-like, optional) – Sample weights.

Returns:

f1 – F1 score. Returns 0.0 if both precision and recall are 0.

Return type:

float

aurora.validation.metrics.classification.roc_auc(y_true, y_score, *, sample_weight=None)[source]¶

Compute Area Under the Receiver Operating Characteristic Curve (ROC-AUC).

This is equivalent to the concordance index for binary classification.

Parameters:

y_true (array-like) – True binary labels.
y_score (array-like) – Predicted scores or probabilities.
sample_weight (array-like, optional) – Sample weights.

Returns:

auc – ROC-AUC score, ranging from 0.0 to 1.0. A value of 0.5 indicates random predictions.

Return type:

float

GLM Metrics¶

GLM-specific validation metrics.

aurora.validation.metrics.glm.generalized_deviance(y_true, mu_pred, family)[source]¶

Compute the generalized deviance for predictions under a GLM family.

The deviance measures the goodness of fit of a generalized linear model, defined as twice the difference between the saturated log-likelihood and the fitted model log-likelihood:

D(y, mu) = 2 * [l(y; y) - l(y; mu)]

Parameters:

y_true (array-like) – Observed response values. Supports NumPy arrays, PyTorch tensors, and JAX arrays.
mu_pred (array-like) – Predicted mean values on the response scale. Must have the same shape as y_true.
family (Family) – Distribution family object (e.g., GaussianFamily(), PoissonFamily()). Must implement the deviance method.

Returns:

The generalized deviance.

Return type:

float

See also

aic: Akaike Information Criterion from deviance.
bic: Bayesian Information Criterion from deviance.

aurora.validation.metrics.glm.aic(deviance, n_params)[source]¶

Compute Akaike’s Information Criterion from deviance and parameter count.

AIC = deviance + 2 * n_params

Parameters:

deviance (float) – The model deviance (or -2 * log-likelihood up to a constant).
n_params (int) – Number of estimated parameters (including intercept).

Returns:

The AIC value. Lower values indicate a better trade-off between goodness of fit and model complexity.

Return type:

float

See also

bic: Bayesian Information Criterion (penalizes complexity more for n > 8).
generalized_deviance: Compute deviance from predictions.

aurora.validation.metrics.glm.bic(deviance, n_params, n_samples)[source]¶

Compute Bayesian Information Criterion from deviance and sample size.

BIC = deviance + log(n_samples) * n_params

Parameters:

deviance (float) – The model deviance (or -2 * log-likelihood up to a constant).
n_params (int) – Number of estimated parameters (including intercept).
n_samples (int) – Number of observations. Must be positive.

Returns:

The BIC value. Lower values indicate better models. BIC penalizes model complexity more strongly than AIC when n_samples > 8.

Return type:

float

Raises:

ValueError – If n_samples is not positive.

See also

aic: Akaike Information Criterion (less complexity penalty).
generalized_deviance: Compute deviance from predictions.

aurora.validation.metrics.glm.pseudo_r2(result, *, method='mcfadden')[source]¶

Compute pseudo \(R^2\) scores from a fitted GLM result.

Several pseudo R-squared measures are provided, each offering a different interpretation of model fit for non-Gaussian GLMs where the classical R-squared does not apply directly.

Parameters:

result (GLMResult) – A fitted GLM model result. Must have deviance_, null_deviance_, coef_, and intercept_ attributes.
method (str, default='mcfadden') –
Pseudo R-squared method. One of:
- 'mcfadden' (or 'deviance'): 1 - deviance / null_deviance.
- 'mcfadden_adj': Adjusted McFadden, penalising for the number of parameters.
- 'cox_snell': Cox & Snell’s measure, bounded below 1.
- 'nagelkerke' (or 'cragg_uhler'): Nagelkerke’s rescaled version, bounded to [0, 1].

Returns:

The pseudo R-squared value, clipped to [0, 1].

Return type:

float

Raises:

ValueError – If null_deviance_ is not positive for McFadden-style methods, or if n_samples is zero.
NotImplementedError – If an unsupported method name is provided.

See also

generalized_deviance: Compute the deviance used in pseudo R-squared.

Cross-Validation¶

Cross-validation utilities.

class aurora.validation.cross_val.KFold(n_splits=5, shuffle=False, random_state=None)[source]¶

Bases: object

K-fold cross-validation splitter yielding train and validation indices.

Splits the dataset into n_splits folds. For each split, one fold is used for validation and the remaining folds form the training set.

Parameters:

n_splits (int, default=5) – Number of folds. Must be at least 2.
shuffle (bool, default=False) – Whether to shuffle indices before splitting.
random_state (int or None, default=None) – Seed for the random number generator when shuffle=True.

Examples

>>> import numpy as np
>>> splitter = KFold(n_splits=3, shuffle=False)
>>> X = np.arange(9)
>>> for train, val in splitter.split(X):
...     print(train, val)
[3 4 5 6 7 8] [0 1 2]
[0 1 2 6 7 8] [3 4 5]
[0 1 2 3 4 5] [6 7 8]

n_splits: int = 5¶

shuffle: bool = False¶

random_state: int | None = None¶

split(X, y=None)[source]¶

Generate train/validation index pairs.

Parameters:

X (array-like, shape (n_samples, ...) or (n_samples,)) – Input data used only to determine the number of samples.
y (array-like, optional) – Ignored. Present for API compatibility with StratifiedKFold.

Yields:

train_indices (ndarray, shape (n_train,)) – Training set indices.
val_indices (ndarray, shape (n_val,)) – Validation set indices.

Raises:

ValueError – If n_splits is less than 2 or exceeds the number of samples.

Return type:

Iterator[tuple[ndarray, ndarray]]

class aurora.validation.cross_val.StratifiedKFold(n_splits=5, shuffle=False, random_state=None)[source]¶

Bases: object

Stratified K-fold splitter preserving label proportions in each fold.

Each fold maintains approximately the same class distribution as the full dataset, which is important for imbalanced classification problems.

Parameters:

n_splits (int, default=5) – Number of folds. Must be at least 2.
shuffle (bool, default=False) – Whether to shuffle within each class before splitting.
random_state (int or None, default=None) – Seed for the random number generator when shuffle=True.

Examples

>>> import numpy as np
>>> splitter = StratifiedKFold(n_splits=2, shuffle=False)
>>> X = np.arange(6)
>>> y = np.array([0, 0, 0, 1, 1, 1])
>>> for train, val in splitter.split(X, y):
...     print(train, val)
[1 2 4 5] [0 3]
[0 3 4 5] [1 2]

n_splits: int = 5¶

shuffle: bool = False¶

random_state: int | None = None¶

split(X, y)[source]¶

Generate stratified train/validation index pairs.

Parameters:

X (array-like, shape (n_samples, ...) or (n_samples,)) – Input data used to determine the number of samples.
y (array-like, shape (n_samples,)) – Target labels. Must be one-dimensional. Each class must have at least n_splits members.

Yields:

train_indices (ndarray, shape (n_train,)) – Training set indices.
val_indices (ndarray, shape (n_val,)) – Validation set indices.

Raises:

ValueError – If y is None, n_splits is less than 2, n_splits exceeds the number of samples, or any class has fewer than n_splits members.

Return type:

Iterator[tuple[ndarray, ndarray]]

class aurora.validation.cross_val.CrossValResult(scores, mean, std)[source]¶

Bases: object

Aggregate statistics for a cross-validation run.

Variables:

scores (ndarray, shape (n_splits,)) – Per-fold score values.
mean (float) – Mean of the per-fold scores.
std (float) – Standard deviation of the per-fold scores (ddof=1).

Parameters:

scores (ndarray)
mean (float)
std (float)

scores: ndarray¶

mean: float¶

std: float¶

aurora.validation.cross_val.cross_val_score(fit_func, score_func, X, y, *, n_splits=5, shuffle=False, random_state=None, splitter=None, fit_kwargs=None, score_kwargs=None, return_result=False)[source]¶

Evaluate a model using K-fold cross-validation.

fit_func must accept the training design matrix and response as its first two positional arguments and return a fitted model object. score_func must accept the fitted model, validation design matrix, and response, returning a scalar score where larger values indicate better performance.

Set return_result to True to obtain a CrossValResult with summary statistics in addition to the fold scores.

Parameters:

fit_func (callable) – Model fitting function with signature fit_func(X_train, y_train, **fit_kwargs) -> model.
score_func (callable) – Scoring function with signature score_func(model, X_test, y_test, **score_kwargs) -> float. Higher scores indicate better performance.
X (array-like, shape (n_samples, n_features)) – Design matrix.
y (array-like, shape (n_samples,)) – Response vector.
n_splits (int, default=5) – Number of cross-validation folds.
shuffle (bool, default=False) – Whether to shuffle data before splitting into folds.
random_state (int or None, default=None) – Random seed for reproducibility when shuffle=True.
splitter (str, object, or None, default=None) – Cross-validation strategy. Pass 'kfold' for standard K-fold, 'stratified' for stratified K-fold, or any object with a split(X, y) method. If None, uses KFold.
fit_kwargs (dict, optional) – Additional keyword arguments forwarded to fit_func.
score_kwargs (dict, optional) – Additional keyword arguments forwarded to score_func.
return_result (bool, default=False) – If True, return a CrossValResult with mean and std. If False, return only the array of per-fold scores.

Returns:

scores (ndarray, shape (n_splits,)) – Per-fold scores. Returned when return_result=False.
result (CrossValResult) – Aggregate statistics including mean and std. Returned when return_result=True.

Return type:

ndarray | CrossValResult

See also

KFold: Standard K-fold splitter.
StratifiedKFold: Stratified K-fold splitter.