Losses - Nixtla

The most important train signal is the forecast error, which is the difference between the observed value

y_{\tau}

and the prediction

\hat{y}_{\tau}

, at time

y_{\tau}

e_{\tau} = y_{\tau}-\hat{y}_{\tau} \qquad \qquad \tau \in \{t+1,\dots,t+H \}

The train loss summarizes the forecast errors in different evaluation metrics.

from utilsforecast.data import generate_series

from polars.testing import assert_frame_equal as pl_assert_frame_equal
models = ['model0', 'model1']
series = generate_series(1000, n_models=2, level=[80])
series_pl = generate_series(1000, n_models=2, level=[80], engine='polars')

1. Scale-dependent Errors

Mean Absolute Error (MAE)

\mathrm{MAE}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}_{\tau}) = \frac{1}{H} \sum^{t+H}_{\tau=t+1} |y_{\tau} - \hat{y}_{\tau}|

source

mae

 mae (df:~DFType, models:List[str], id_col:str='unique_id',
      target_col:str='y')

*Mean Absolute Error (MAE) MAE measures the relative prediction accuracy of a forecasting method by calculating the deviation of the prediction and the true value at a given time and averages these devations over the length of the series.*

	Type	Default	Details
df	DFType		Input dataframe with id, actual values and predictions.
models	List		Columns that identify the models predictions.
id_col	str	unique_id	Column that identifies each serie.
target_col	str	y	Column that contains the target.
Returns	DFType		dataframe with one row per id and one column per model.

def pd_vs_pl(pd_df, pl_df, models):
    pd.testing.assert_frame_equal(pd_df[models], 
                                  pl_df[models].to_pandas())

pd_vs_pl(
    mae(series, models),
    mae(series_pl, models),
    models=models
)

Mean Squared Error

\mathrm{MSE}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}_{\tau}) = \frac{1}{H} \sum^{t+H}_{\tau=t+1} (y_{\tau} - \hat{y}_{\tau})^{2}

source

mse

 mse (df:~DFType, models:List[str], id_col:str='unique_id',
      target_col:str='y')

*Mean Squared Error (MSE) MSE measures the relative prediction accuracy of a forecasting method by calculating the squared deviation of the prediction and the true value at a given time, and averages these devations over the length of the series.*

	Type	Default	Details
df	DFType		Input dataframe with id, actual values and predictions.
models	List		Columns that identify the models predictions.
id_col	str	unique_id	Column that identifies each serie.
target_col	str	y	Column that contains the target.
Returns	DFType		dataframe with one row per id and one column per model.

pd_vs_pl(
    mse(series, models),
    mse(series_pl, models),
    models,
)

Root Mean Squared Error

\mathrm{RMSE}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}_{\tau}) = \sqrt{\frac{1}{H} \sum^{t+H}_{\tau=t+1} (y_{\tau} - \hat{y}_{\tau})^{2}}

source

rmse

 rmse (df:~DFType, models:List[str], id_col:str='unique_id',
       target_col:str='y')

*Root Mean Squared Error (RMSE) RMSE measures the relative prediction accuracy of a forecasting method by calculating the squared deviation of the prediction and the observed value at a given time and averages these devations over the length of the series. Finally the RMSE will be in the same scale as the original time series so its comparison with other series is possible only if they share a common scale. RMSE has a direct connection to the L2 norm.*

	Type	Default	Details
df	DFType		Input dataframe with id, actual values and predictions.
models	List		Columns that identify the models predictions.
id_col	str	unique_id	Column that identifies each serie.
target_col	str	y	Column that contains the target.
Returns	DFType		dataframe with one row per id and one column per model.

pd_vs_pl(
    rmse(series, models),
    rmse(series_pl, models),
    models,
)

source

bias

 bias (df:~DFType, models:List[str], id_col:str='unique_id',
       target_col:str='y')

*Forecast estimator bias. Defined as prediction - actual*

	Type	Default	Details
df	DFType		Input dataframe with id, actual values and predictions.
models	List		Columns that identify the models predictions.
id_col	str	unique_id	Column that identifies each serie.
target_col	str	y	Column that contains the target.
Returns	DFType		dataframe with one row per id and one column per model.

pd_vs_pl(
    bias(series, models),
    bias(series_pl, models),
    models,
)

source

cfe

 cfe (df:~DFType, models:List[str], id_col:str='unique_id',
      target_col:str='y')

*Cumulative Forecast Error (CFE) Total signed forecast error per series. Positive values mean under forecast; negative mean over forecast.*

	Type	Default	Details
df	DFType		Input dataframe with id, actual values and predictions.
models	List		Columns that identify the models predictions.
id_col	str	unique_id	Column that identifies each serie.
target_col	str	y	Column that contains the target.
Returns	DFType		dataframe with one row per id and one column per model.

pd_vs_pl(
    cfe(series, models),
    cfe(series_pl, models),
    models,
)

# case for cfe
df = pd.DataFrame({
    "unique_id": ["X","X","Y","Y"],
    "y":         [5,  10, 3,  7],
    "y_hat":         [7,   7, 1, 10]
})
# errors:
#  X: (7 - 5) + (7 - 10) = 2 + (-3) = -1
#  Y: (1 - 3) + (10 - 7) = -2 + 3    = 1
expected = pd.DataFrame({
        "unique_id": ["X", "Y"],
        "y_hat": [-1, 1]
    })

# pandas 
out_pd = cfe(df, ["y_hat"])
pd.testing.assert_frame_equal(
    out_pd,
    expected
    )

df_pl = pl.from_pandas(df)
out_pl = cfe(df_pl, ["y_hat"])
pl_assert_frame_equal(
    out_pl,
    pl.from_pandas(expected)
)

source

pis

 pis (df:~DFType, models:List[str], id_col:str='unique_id',
      target_col:str='y')

*Compute the raw Absolute Periods In Stock (PIS) for one or multiple models. The PIS metric sums the absolute forecast errors per series without any scaling, yielding a scale-dependent measure of bias.*

	Type	Default	Details
df	DFType		Input dataframe with id, actual values and predictions.
models	List		Columns that identify the models predictions.
id_col	str	unique_id	Column that identifies each serie.
target_col	str	y	Column that contains the target.
Returns	DFType		dataframe with one row per id and one column per model.

pd_vs_pl(
    pis(series, models),
    pis(series_pl, models),
    models,
)

# case for pis
df = pd.DataFrame({
    "unique_id": ["A","A","B","B"],
    "y":         [10, 15,  5,  7],
    "y_hat":         [12, 14,  4, 10]
})
# errors:
#  A: |12−10| + |14−15| = 2 + 1 = 3
#  B: |4−5|  + |10−7| = 1 + 3 = 4
expected = pd.DataFrame({
        "unique_id": ["A", "B"],
        "y_hat": [3, 4]
    })

# pandas branch
out_pd = pis(df, ["y_hat"])
pd.testing.assert_frame_equal(
    out_pd,
    expected
)

df_pl = pl.from_pandas(df)
out_pl = pis(df_pl, ["y_hat"])
pl_assert_frame_equal(
    out_pl,
    pl.from_pandas(expected)
)

source

spis

 spis (df:~DFType, df_train:~DFType, models:List[str],
       id_col:str='unique_id', target_col:str='y')

*Compute the scaled Absolute Periods In Stock (sAPIS) for one or multiple models. The sPIS metric scales the sum of absolute forecast errors by the mean in-sample demand, yielding a scale-independent bias measure that can be aggregated across series.*

	Type	Default	Details
df	DFType		Input dataframe with id, actual values and predictions.
df_train	DFType
models	List		Columns that identify the models predictions.
id_col	str	unique_id	Column that identifies each serie.
target_col	str	y	Column that contains the target.
Returns	DFType		dataframe with one row per id and one column per model.

pd_vs_pl(
    spis(series, series, models),
    spis(series_pl, series_pl, models),
    models,
)

/tmp/ipykernel_12558/2968355870.py:19: FutureWarning: The default of observed=False is deprecated and will be changed to True in a future version of pandas. Pass observed=False to retain current behavior or observed=True to adopt the future default and silence this warning.
  .groupby(id_col)[target_col]

# case for scaled pis
df_train = pd.DataFrame({
    "unique_id": ["A","A","B","B"],
    "y": [1, 3,  2, 6]
})
# Forecast data
df = pd.DataFrame({
    "unique_id": ["A","A","B","B"],
    "y":          [3, 3,  2, 8],
    "y_hat":      [6, 2,  3, 5]
})
# For A: errors = |3−6|+|3−2| = 3+1 = 4  ÷ mean(1,3)=2 → 2
# For B: errors = |2−3|+|8−5| = 1+3 = 4  ÷ mean(2,6)=4 → 1
expected = pd.DataFrame({
        "unique_id": ["A", "B"],
        "y_hat": [2.0, 1.0]
    })

# pandas branch
out_pd = spis(
    df       = df,
    df_train = df_train,
    models   = ["y_hat"],
    id_col   = "unique_id",
    target_col   = "y",
)
pd.testing.assert_frame_equal(
    out_pd,
    expected
)

df_train_pl = pl.from_pandas(df_train)
df_pl = pl.from_pandas(df)
out_pl = spis(
    df       = df_pl,
    df_train = df_train_pl,
    models   = ["y_hat"],
    id_col   = "unique_id",
    target_col   = "y",
)
pl.testing.assert_frame_equal(
    out_pl,
    pl.from_pandas(expected)
)

2. Percentage Errors

Mean Absolute Percentage Error

\mathrm{MAPE}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}_{\tau}) = \frac{1}{H} \sum^{t+H}_{\tau=t+1} \frac{|y_{\tau}-\hat{y}_{\tau}|}{|y_{\tau}|}

source

mape

 mape (df:~DFType, models:List[str], id_col:str='unique_id',
       target_col:str='y')

*Mean Absolute Percentage Error (MAPE) MAPE measures the relative prediction accuracy of a forecasting method by calculating the percentual deviation of the prediction and the observed value at a given time and averages these devations over the length of the series. The closer to zero an observed value is, the higher penalty MAPE loss assigns to the corresponding error.*

	Type	Default	Details
df	DFType		Input dataframe with id, actual values and predictions.
models	List		Columns that identify the models predictions.
id_col	str	unique_id	Column that identifies each serie.
target_col	str	y	Column that contains the target.
Returns	DFType		dataframe with one row per id and one column per model.

pd_vs_pl(
    mape(series, models),
    mape(series_pl, models),
    models,
)

Symmetric Mean Absolute Percentage Error

\mathrm{SMAPE}_{2}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}_{\tau}) = \frac{1}{H} \sum^{t+H}_{\tau=t+1} \frac{|y_{\tau}-\hat{y}_{\tau}|}{|y_{\tau}|+|\hat{y}_{\tau}|}

source

smape

 smape (df:~DFType, models:List[str], id_col:str='unique_id',
        target_col:str='y')

*Symmetric Mean Absolute Percentage Error (SMAPE) SMAPE measures the relative prediction accuracy of a forecasting method by calculating the relative deviation of the prediction and the observed value scaled by the sum of the absolute values for the prediction and observed value at a given time, then averages these devations over the length of the series. This allows the SMAPE to have bounds between 0% and 100% which is desirable compared to normal MAPE that may be undetermined when the target is zero.*

	Type	Default	Details
df	DFType		Input dataframe with id, actual values and predictions.
models	List		Columns that identify the models predictions.
id_col	str	unique_id	Column that identifies each serie.
target_col	str	y	Column that contains the target.
Returns	DFType		dataframe with one row per id and one column per model.

pd_vs_pl(
    smape(series, models),
    smape(series_pl, models),
    models,
)

3. Scale-independent Errors

Mean Absolute Scaled Error

\mathrm{MASE}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}_{\tau}, \mathbf{\hat{y}}^{season}_{\tau}) = \frac{1}{H} \sum^{t+H}_{\tau=t+1} \frac{|y_{\tau}-\hat{y}_{\tau}|}{\mathrm{MAE}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}^{season}_{\tau})}

source

mase

 mase (df:~DFType, models:List[str], seasonality:int, train_df:~DFType,
       id_col:str='unique_id', target_col:str='y')

*Mean Absolute Scaled Error (MASE) MASE measures the relative prediction accuracy of a forecasting method by comparinng the mean absolute errors of the prediction and the observed value against the mean absolute errors of the seasonal naive model. The MASE partially composed the Overall Weighted Average (OWA), used in the M4 Competition.*

	Type	Default	Details
df	DFType		Input dataframe with id, actuals and predictions.
models	List		Columns that identify the models predictions.
seasonality	int		Main frequency of the time series; Hourly 24, Daily 7, Weekly 52, Monthly 12, Quarterly 4, Yearly 1.
train_df	DFType		Training dataframe with id and actual values. Must be sorted by time.
id_col	str	unique_id	Column that identifies each serie.
target_col	str	y	Column that contains the target.
Returns	DFType		dataframe with one row per id and one column per model.

pd_vs_pl(
    mase(series, models, 7, series),
    mase(series_pl, models, 7, series_pl),
    models,
)

Relative Mean Absolute Error

\mathrm{RMAE}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}_{\tau}, \mathbf{\hat{y}}^{base}_{\tau}) = \frac{1}{H} \sum^{t+H}_{\tau=t+1} \frac{|y_{\tau}-\hat{y}_{\tau}|}{\mathrm{MAE}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}^{base}_{\tau})}

source

rmae

 rmae (df:~DFType, models:List[str], baseline:str, id_col:str='unique_id',
       target_col:str='y')

*Relative Mean Absolute Error (RMAE) Calculates the RAME between two sets of forecasts (from two different forecasting methods). A number smaller than one implies that the forecast in the numerator is better than the forecast in the denominator.*

	Type	Default	Details
df	DFType		Input dataframe with id, times, actuals and predictions.
models	List		Columns that identify the models predictions.
baseline	str		Column that identifies the baseline model predictions.
id_col	str	unique_id	Column that identifies each serie.
target_col	str	y	Column that contains the target.
Returns	DFType		dataframe with one row per id and one column per model.

pd_vs_pl(
    rmae(series, models, models[0]),
    rmae(series_pl, models, models[0]),
    models,
)

Normalized Deviation

\mathrm{ND}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}_{\tau}) = \frac{\sum^{t+H}_{\tau=t+1} |y_{\tau} - \hat{y}_{\tau}|}{\sum^{t+H}_{\tau=t+1} | y_{\tau} |}

source

nd

 nd (df:~DFType, models:List[str], id_col:str='unique_id',
     target_col:str='y')

*Normalized Deviation (ND) ND measures the relative prediction accuracy of a forecasting method by calculating the sum of the absolute deviation of the prediction and the true value at a given time and dividing it by the sum of the absolute value of the ground truth.*

	Type	Default	Details
df	DFType		Input dataframe with id, times, actuals and predictions.
models	List		Columns that identify the models predictions.
id_col	str	unique_id	Column that identifies each serie.
target_col	str	y	Column that contains the target.
Returns	DFType		dataframe with one row per id and one column per model.

pd_vs_pl(
    nd(series, models),
    nd(series_pl, models),
    models,
)

Mean Squared Scaled Error

\mathrm{MSSE}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}_{\tau}, \mathbf{\hat{y}}^{season}_{\tau}) = \frac{1}{H} \sum^{t+H}_{\tau=t+1} \frac{(y_{\tau}-\hat{y}_{\tau})^2}{\mathrm{MSE}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}^{season}_{\tau})}

source

msse

 msse (df:~DFType, models:List[str], seasonality:int, train_df:~DFType,
       id_col:str='unique_id', target_col:str='y')

*Mean Squared Scaled Error (MSSE) MSSE measures the relative prediction accuracy of a forecasting method by comparinng the mean squared errors of the prediction and the observed value against the mean squared errors of the seasonal naive model.*

	Type	Default	Details
df	DFType		Input dataframe with id, actuals and predictions.
models	List		Columns that identify the models predictions.
seasonality	int		Main frequency of the time series; Hourly 24, Daily 7, Weekly 52, Monthly 12, Quarterly 4, Yearly 1.
train_df	DFType		Training dataframe with id and actual values. Must be sorted by time.
id_col	str	unique_id	Column that identifies each serie.
target_col	str	y	Column that contains the target.
Returns	DFType		dataframe with one row per id and one column per model.

pd_vs_pl(
    msse(series, models, 7, series),
    msse(series_pl, models, 7, series_pl),
    models,
)

Root Mean Squared Scaled Error

\mathrm{RMSSE}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}_{\tau}, \mathbf{\hat{y}}^{season}_{\tau}) = \sqrt{\frac{1}{H} \sum^{t+H}_{\tau=t+1} \frac{(y_{\tau}-\hat{y}_{\tau})^2}{\mathrm{MSE}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}^{season}_{\tau})}}

source

rmsse

 rmsse (df:~DFType, models:List[str], seasonality:int, train_df:~DFType,
        id_col:str='unique_id', target_col:str='y')

*Root Mean Squared Scaled Error (RMSSE) MSSE measures the relative prediction accuracy of a forecasting method by comparinng the mean squared errors of the prediction and the observed value against the mean squared errors of the seasonal naive model.*

	Type	Default	Details
df	DFType		Input dataframe with id, actuals and predictions.
models	List		Columns that identify the models predictions.
seasonality	int		Main frequency of the time series; Hourly 24, Daily 7, Weekly 52, Monthly 12, Quarterly 4, Yearly 1.
train_df	DFType		Training dataframe with id and actual values. Must be sorted by time.
id_col	str	unique_id	Column that identifies each serie.
target_col	str	y	Column that contains the target.
Returns	DFType		dataframe with one row per id and one column per model.

pd_vs_pl(
    rmsse(series, models, 7, series),
    rmsse(series_pl, models, 7, series_pl),
    models,
)

4. Probabilistic Errors

Quantile Loss

\mathrm{QL}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}^{(q)}_{\tau}) = \frac{1}{H} \sum^{t+H}_{\tau=t+1} \Big( (1-q)\,( \hat{y}^{(q)}_{\tau} - y_{\tau} )_{+} + q\,( y_{\tau} - \hat{y}^{(q)}_{\tau} )_{+} \Big)

source

quantile_loss

 quantile_loss (df:~DFType, models:Dict[str,str], q:float=0.5,
                id_col:str='unique_id', target_col:str='y')

*Quantile Loss (QL) QL measures the deviation of a quantile forecast. By weighting the absolute deviation in a non symmetric way, the loss pays more attention to under or over estimation.
A common value for q is 0.5 for the deviation from the median.*

	Type	Default	Details
df	DFType		Input dataframe with id, times, actuals and predictions.
models	Dict		Mapping from model name to the model predictions for the specified quantile.
q	float	0.5	Quantile for the predictions’ comparison.
id_col	str	unique_id	Column that identifies each serie.
target_col	str	y	Column that contains the target.
Returns	DFType		dataframe with one row per id and one column per model.

Scaled Quantile Loss

\mathrm{SQL}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}^{(q)}_{\tau}) = \frac{1}{H} \sum^{t+H}_{\tau=t+1} \frac{(1-q)\,( \hat{y}^{(q)}_{\tau} - y_{\tau} )_{+} + q\,( y_{\tau} - \hat{y}^{(q)}_{\tau} )_{+}}{\mathrm{MAE}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}^{season}_{\tau})}

source

scaled_quantile_loss

 scaled_quantile_loss (df:~DFType, models:Dict[str,str], seasonality:int,
                       train_df:~DFType, q:float=0.5,
                       id_col:str='unique_id', target_col:str='y')

*Scaled Quantile Loss (SQL) SQL measures the deviation of a quantile forecast scaled by the mean absolute errors of the seasonal naive model. By weighting the absolute deviation in a non symmetric way, the loss pays more attention to under or over estimation. A common value for q is 0.5 for the deviation from the median. This was the official measure used in the M5 Uncertainty competition with seasonality = 1.*

	Type	Default	Details
df	DFType		Input dataframe with id, times, actuals and predictions.
models	Dict		Mapping from model name to the model predictions for the specified quantile.
seasonality	int		Main frequency of the time series; Hourly 24, Daily 7, Weekly 52, Monthly 12, Quarterly 4, Yearly 1.
train_df	DFType		Training dataframe with id and actual values. Must be sorted by time.
q	float	0.5	Quantile for the predictions’ comparison.
id_col	str	unique_id	Column that identifies each serie.
target_col	str	y	Column that contains the target.
Returns	DFType		dataframe with one row per id and one column per model.

Multi-Quantile Loss

\mathrm{MQL}(\mathbf{y}_{\tau}, [\mathbf{\hat{y}}^{(q_{1})}_{\tau}, ... ,\hat{y}^{(q_{n})}_{\tau}]) = \frac{1}{n} \sum_{q_{i}} \mathrm{QL}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}^{(q_{i})}_{\tau})

source

mqloss

 mqloss (df:~DFType, models:Dict[str,List[str]], quantiles:numpy.ndarray,
         id_col:str='unique_id', target_col:str='y')

*Multi-Quantile loss (MQL) MQL calculates the average multi-quantile Loss for a given set of quantiles, based on the absolute difference between predicted quantiles and observed values. The limit behavior of MQL allows to measure the accuracy of a full predictive distribution with the continuous ranked probability score (CRPS). This can be achieved through a numerical integration technique, that discretizes the quantiles and treats the CRPS integral with a left Riemann approximation, averaging over uniformly distanced quantiles.*

	Type	Default	Details
df	DFType		Input dataframe with id, times, actuals and predictions.
models	Dict		Mapping from model name to the model predictions for each quantile.
quantiles	ndarray		Quantiles to compare against.
id_col	str	unique_id	Column that identifies each serie.
target_col	str	y	Column that contains the target.
Returns	DFType		dataframe with one row per id and one column per model.

pd_vs_pl(
    mqloss(series, mq_models, quantiles=quantiles),
    mqloss(series_pl, mq_models, quantiles=quantiles),
    models,
)

Scaled Multi-Quantile Loss

\mathrm{MQL}(\mathbf{y}_{\tau}, [\mathbf{\hat{y}}^{(q_{1})}_{\tau}, ... ,\hat{y}^{(q_{n})}_{\tau}]) = \frac{1}{n} \sum_{q_{i}} \frac{\mathrm{QL}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}^{(q_{i})}_{\tau})}{\mathrm{MAE}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}^{season}_{\tau})}

source

scaled_mqloss

 scaled_mqloss (df:~DFType, models:Dict[str,List[str]],
                quantiles:numpy.ndarray, seasonality:int,
                train_df:~DFType, id_col:str='unique_id',
                target_col:str='y')

*Scaled Multi-Quantile loss (SMQL) SMQL calculates the average multi-quantile Loss for a given set of quantiles, based on the absolute difference between predicted quantiles and observed values scaled by the mean absolute errors of the seasonal naive model. The limit behavior of MQL allows to measure the accuracy of a full predictive distribution with the continuous ranked probability score (CRPS). This can be achieved through a numerical integration technique, that discretizes the quantiles and treats the CRPS integral with a left Riemann approximation, averaging over uniformly distanced quantiles. This was the official measure used in the M5 Uncertainty competition with seasonality = 1.*

	Type	Default	Details
df	DFType		Input dataframe with id, times, actuals and predictions.
models	Dict		Mapping from model name to the model predictions for each quantile.
quantiles	ndarray		Quantiles to compare against.
seasonality	int		Main frequency of the time series; Hourly 24, Daily 7, Weekly 52, Monthly 12, Quarterly 4, Yearly 1.
train_df	DFType		Training dataframe with id and actual values. Must be sorted by time.
id_col	str	unique_id	Column that identifies each serie.
target_col	str	y	Column that contains the target.
Returns	DFType		dataframe with one row per id and one column per model.

pd_vs_pl(
    scaled_mqloss(series, mq_models, quantiles=quantiles, seasonality=1, train_df=series),
    scaled_mqloss(series_pl, mq_models, quantiles=quantiles, seasonality=1, train_df=series_pl),
    models,
)

Coverage

source

coverage

 coverage (df:~DFType, models:List[str], level:int,
           id_col:str='unique_id', target_col:str='y')

Coverage of y with y_hat_lo and y_hat_hi.

	Type	Default	Details
df	DFType		Input dataframe with id, times, actuals and predictions.
models	List		Columns that identify the models predictions.
level	int		Confidence level used for intervals.
id_col	str	unique_id	Column that identifies each serie.
target_col	str	y	Column that contains the target.
Returns	DFType		dataframe with one row per id and one column per model.

pd_vs_pl(
    coverage(series, models, 80),
    coverage(series_pl, models, 80),
    models,
)

Calibration

source

calibration

 calibration (df:~DFType, models:Dict[str,str], id_col:str='unique_id',
              target_col:str='y')

Fraction of y that is lower than the model’s predictions.

	Type	Default	Details
df	DFType		Input dataframe with id, times, actuals and predictions.
models	Dict		Mapping from model name to the model predictions.
id_col	str	unique_id	Column that identifies each serie.
target_col	str	y	Column that contains the target.
Returns	DFType		dataframe with one row per id and one column per model.

pd_vs_pl(
    calibration(series, q_models[0.1]),
    calibration(series_pl, q_models[0.1]),
    models,
)

CRPS

\mathrm{sCRPS}(\hat{F}_{\tau}, \mathbf{y}_{\tau}) = \frac{2}{N} \sum_{i} \int^{1}_{0} \frac{\mathrm{QL}(\hat{F}_{i,\tau}, y_{i,\tau})_{q}}{\sum_{i} | y_{i,\tau} |} dq

Where

\hat{F}_{\tau}

is the an estimated multivariate distribution, and

y_{i,\tau}

are its realizations.

source

scaled_crps

 scaled_crps (df:~DFType, models:Dict[str,List[str]],
              quantiles:numpy.ndarray, id_col:str='unique_id',
              target_col:str='y')

*Scaled Continues Ranked Probability Score Calculates a scaled variation of the CRPS, as proposed by Rangapuram (2021), to measure the accuracy of predicted quantiles y_hat compared to the observation y. This metric averages percentual weighted absolute deviations as defined by the quantile losses.*

	Type	Default	Details
df	DFType		Input dataframe with id, times, actuals and predictions.
models	Dict		Mapping from model name to the model predictions for each quantile.
quantiles	ndarray		Quantiles to compare against.
id_col	str	unique_id	Column that identifies each serie.
target_col	str	y	Column that contains the target.
Returns	DFType		dataframe with one row per id and one column per model.

pd_vs_pl(
    scaled_crps(series, mq_models, quantiles),
    scaled_crps(series_pl, mq_models, quantiles),
    models,
)

Tweedie Deviance

For a set of forecasts

\{\mu_i\}_{i=1}^N

and observations

\{y_i\}_{i=1}^N

, the mean Tweedie deviance with power

p

\mathrm{TD}_{p}(\boldsymbol{\mu}, \mathbf{y}) = \frac{1}{N} \sum_{i=1}^{N} d_{p}(y_i, \mu_i)

where the unit-scaled deviance for each pair

(y,\mu)

d_{p}(y,\mu) = 2 \begin{cases} \displaystyle \frac{y^{2-p}}{(1-p)(2-p)} \;-\; \frac{y\,\mu^{1-p}}{1-p} \;+\; \frac{\mu^{2-p}}{2-p}, & p \notin\{1,2\},\\[1em] \displaystyle y\,\ln\!\frac{y}{\mu}\;-\;(y-\mu), & p = 1\quad(\text{Poisson deviance}),\\[0.5em] \displaystyle -2\Bigl[\ln\!\frac{y}{\mu}\;-\;\frac{y-\mu}{\mu}\Bigr], & p = 2\quad(\text{Gamma deviance}). \end{cases}

$y_i$ are the true values, $\mu_i$ the predicted means.
$p$ controls the variance relationship $\mathrm{Var}(Y)\propto\mu^{p}$ .
When $1<p<2$ , this smoothly interpolates between Poisson ( $p=1$ ) and Gamma ( $p=2$ ) deviance.

source

tweedie_deviance

 tweedie_deviance (df:~DFType, models:List[str], power:float=1.5,
                   id_col:str='unique_id', target_col:str='y')

*Compute the Tweedie deviance loss for one or multiple models, grouped by an identifier. Each group’s deviance is calculated using the mean_tweedie_deviance function, which measures the deviation between actual and predicted values under the Tweedie distribution. The power parameter defines the specific compound distribution: - 1: Poisson - (1, 2): Compound Poisson-Gamma - 2: Gamma - >2: Inverse Gaussian*

	Type	Default	Details
df	DFType		Input dataframe with id, actuals and predictions.
models	List		Columns that identify the models predictions.
power	float	1.5	Tweedie power parameter. Determines the compound distribution.
id_col	str	unique_id	Column that identifies each serie.
target_col	str	y	Column that contains the target.
Returns	DFType		DataFrame with one row per id and one column per model, containing the mean Tweedie deviance.

# Normal test
for power in [0, 1, 1.5, 2, 3]:
    # Test Pandas vs Polars
    td_pd = tweedie_deviance(series,   models, target_col="y", power=power)
    td_pl = tweedie_deviance(series_pl, models, target_col="y", power=power)
    pd_vs_pl(
        td_pd,
        td_pl,
        models,
    )
    # Test for NaNs
    assert not td_pd[models].isna().any().any(), f"NaNs found in pd DataFrame for power {power}"
    assert not td_pl.select(pl.col(models).is_null().any()).sum_horizontal().item(), f"NaNs found in pl DataFrame for power {power}"
    # Test for infinites
    is_infinite = td_pd[models].isin([np.inf, -np.inf]).any().any()
    assert not is_infinite, f"Infinities found in pd DataFrame for power {power}"
    is_infinite_pl = td_pl.select(pl.col(models).is_infinite().any()).sum_horizontal().item()
    assert not is_infinite_pl, f"Infinities found in pl DataFrame for power {power}"

# Test zero handling (skip power >=2 since it requires all y > 0)
series.loc[0, 'y'] = 0.0  # Set a zero value to test the zero handling
series.loc[49, 'y'] = 0.0  # Set another zero value to test the zero handling
series_pl[0, 'y'] = 0.0  # Set a zero value to test the zero handling
series_pl[49, 'y'] = 0.0  # Set another zero value to test the zero handling
for power in [0, 1, 1.5]:
    # Test Pandas vs Polars
    td_pd = tweedie_deviance(series,   models, target_col="y", power=power)
    td_pl = tweedie_deviance(series_pl, models, target_col="y", power=power)
    pd_vs_pl(
        td_pd,
        td_pl,
        models,
    )
    # Test for NaNs
    assert not td_pd[models].isna().any().any(), f"NaNs found in pd DataFrame for power {power}"
    assert not td_pl.select(pl.col(models).is_null().any()).sum_horizontal().item(), f"NaNs found in pl DataFrame for power {power}"
    # Test for infinites
    is_infinite = td_pd[models].isin([np.inf, -np.inf]).any().any()
    assert not is_infinite, f"Infinities found in pd DataFrame for power {power}"
    is_infinite_pl = td_pl.select(pl.col(models).is_infinite().any()).sum_horizontal().item()
    assert not is_infinite_pl, f"Infinities found in pl DataFrame for power {power}"

​1. Scale-dependent Errors

​Mean Absolute Error (MAE)

​mae

​Mean Squared Error

​mse

​Root Mean Squared Error

​rmse

​bias

​cfe

​pis

​spis

​2. Percentage Errors

​Mean Absolute Percentage Error

​mape

​Symmetric Mean Absolute Percentage Error

​smape

​3. Scale-independent Errors

​Mean Absolute Scaled Error

​mase

​Relative Mean Absolute Error

​rmae

​Normalized Deviation

​nd

​Mean Squared Scaled Error

​msse

​Root Mean Squared Scaled Error

​rmsse

​4. Probabilistic Errors

​Quantile Loss

​quantile_loss

​Scaled Quantile Loss

​scaled_quantile_loss

​Multi-Quantile Loss

​mqloss

​Scaled Multi-Quantile Loss

​scaled_mqloss

​Coverage

​coverage

​Calibration

​calibration

​CRPS

​scaled_crps

​Tweedie Deviance

​tweedie_deviance

1. Scale-dependent Errors

Mean Absolute Error (MAE)

mae

Mean Squared Error

mse

Root Mean Squared Error

rmse

bias

cfe

pis

spis

2. Percentage Errors

Mean Absolute Percentage Error

mape

Symmetric Mean Absolute Percentage Error

smape

3. Scale-independent Errors

Mean Absolute Scaled Error

mase

Relative Mean Absolute Error

rmae

Normalized Deviation

nd

Mean Squared Scaled Error

msse

Root Mean Squared Scaled Error

rmsse

4. Probabilistic Errors

Quantile Loss

quantile_loss

Scaled Quantile Loss

scaled_quantile_loss

Multi-Quantile Loss

mqloss

Scaled Multi-Quantile Loss

scaled_mqloss

Coverage

coverage

Calibration

calibration

CRPS

scaled_crps

Tweedie Deviance

tweedie_deviance