> ## Documentation Index
> Fetch the complete documentation index at: https://nixtla-old-docs.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.
> Loss functions for model evaluation.
# Losses
The most important train signal is the forecast error, which is the
difference between the observed value $y_{\tau}$ and the prediction
$\hat{y}_{\tau}$, at time $y_{\tau}$:
$$
e_{\tau} = y_{\tau}-\hat{y}_{\tau} \qquad \qquad \tau \in \{t+1,\dots,t+H \}
$$
The train loss summarizes the forecast errors in different evaluation
metrics.
```python theme={null}
from utilsforecast.data import generate_series
```
```python theme={null}
from polars.testing import assert_frame_equal as pl_assert_frame_equal
models = ['model0', 'model1']
series = generate_series(1000, n_models=2, level=[80])
series_pl = generate_series(1000, n_models=2, level=[80], engine='polars')
```
## 1. Scale-dependent Errors
### Mean Absolute Error (MAE)
$$
\mathrm{MAE}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}_{\tau}) = \frac{1}{H} \sum^{t+H}_{\tau=t+1} |y_{\tau} - \hat{y}_{\tau}|
$$
***
source
#### mae
> ```text theme={null}
> mae (df:~DFType, models:List[str], id_col:str='unique_id',
> target_col:str='y')
> ```
\*Mean Absolute Error (MAE)
MAE measures the relative prediction accuracy of a forecasting method by
calculating the deviation of the prediction and the true value at a
given time and averages these devations over the length of the series.\*
| | **Type** | **Default** | **Details** |
| ----------- | ---------- | ----------- | ----------------------------------------------------------- |
| df | DFType | | Input dataframe with id, actual values and predictions. |
| models | List | | Columns that identify the models predictions. |
| id\_col | str | unique\_id | Column that identifies each serie. |
| target\_col | str | y | Column that contains the target. |
| **Returns** | **DFType** | | **dataframe with one row per id and one column per model.** |
```python theme={null}
def pd_vs_pl(pd_df, pl_df, models):
pd.testing.assert_frame_equal(pd_df[models],
pl_df[models].to_pandas())
```
```python theme={null}
pd_vs_pl(
mae(series, models),
mae(series_pl, models),
models=models
)
```
### Mean Squared Error
$$
\mathrm{MSE}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}_{\tau}) = \frac{1}{H} \sum^{t+H}_{\tau=t+1} (y_{\tau} - \hat{y}_{\tau})^{2}
$$
***
source
#### mse
> ```text theme={null}
> mse (df:~DFType, models:List[str], id_col:str='unique_id',
> target_col:str='y')
> ```
\*Mean Squared Error (MSE)
MSE measures the relative prediction accuracy of a forecasting method by
calculating the squared deviation of the prediction and the true value
at a given time, and averages these devations over the length of the
series.\*
| | **Type** | **Default** | **Details** |
| ----------- | ---------- | ----------- | ----------------------------------------------------------- |
| df | DFType | | Input dataframe with id, actual values and predictions. |
| models | List | | Columns that identify the models predictions. |
| id\_col | str | unique\_id | Column that identifies each serie. |
| target\_col | str | y | Column that contains the target. |
| **Returns** | **DFType** | | **dataframe with one row per id and one column per model.** |
```python theme={null}
pd_vs_pl(
mse(series, models),
mse(series_pl, models),
models,
)
```
### Root Mean Squared Error
$$
\mathrm{RMSE}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}_{\tau}) = \sqrt{\frac{1}{H} \sum^{t+H}_{\tau=t+1} (y_{\tau} - \hat{y}_{\tau})^{2}}
$$
***
source
#### rmse
> ```text theme={null}
> rmse (df:~DFType, models:List[str], id_col:str='unique_id',
> target_col:str='y')
> ```
\*Root Mean Squared Error (RMSE)
RMSE measures the relative prediction accuracy of a forecasting method
by calculating the squared deviation of the prediction and the observed
value at a given time and averages these devations over the length of
the series. Finally the RMSE will be in the same scale as the original
time series so its comparison with other series is possible only if they
share a common scale. RMSE has a direct connection to the L2 norm.\*
| | **Type** | **Default** | **Details** |
| ----------- | ---------- | ----------- | ----------------------------------------------------------- |
| df | DFType | | Input dataframe with id, actual values and predictions. |
| models | List | | Columns that identify the models predictions. |
| id\_col | str | unique\_id | Column that identifies each serie. |
| target\_col | str | y | Column that contains the target. |
| **Returns** | **DFType** | | **dataframe with one row per id and one column per model.** |
```python theme={null}
pd_vs_pl(
rmse(series, models),
rmse(series_pl, models),
models,
)
```
***
source
#### bias
> ```text theme={null}
> bias (df:~DFType, models:List[str], id_col:str='unique_id',
> target_col:str='y')
> ```
\*Forecast estimator bias.
Defined as prediction - actual\*
| | **Type** | **Default** | **Details** |
| ----------- | ---------- | ----------- | ----------------------------------------------------------- |
| df | DFType | | Input dataframe with id, actual values and predictions. |
| models | List | | Columns that identify the models predictions. |
| id\_col | str | unique\_id | Column that identifies each serie. |
| target\_col | str | y | Column that contains the target. |
| **Returns** | **DFType** | | **dataframe with one row per id and one column per model.** |
```python theme={null}
pd_vs_pl(
bias(series, models),
bias(series_pl, models),
models,
)
```
***
source
#### cfe
> ```text theme={null}
> cfe (df:~DFType, models:List[str], id_col:str='unique_id',
> target_col:str='y')
> ```
\*Cumulative Forecast Error (CFE)
Total signed forecast error per series. Positive values mean under
forecast; negative mean over forecast.\*
| | **Type** | **Default** | **Details** |
| ----------- | ---------- | ----------- | ----------------------------------------------------------- |
| df | DFType | | Input dataframe with id, actual values and predictions. |
| models | List | | Columns that identify the models predictions. |
| id\_col | str | unique\_id | Column that identifies each serie. |
| target\_col | str | y | Column that contains the target. |
| **Returns** | **DFType** | | **dataframe with one row per id and one column per model.** |
```python theme={null}
pd_vs_pl(
cfe(series, models),
cfe(series_pl, models),
models,
)
```
```python theme={null}
# case for cfe
df = pd.DataFrame({
"unique_id": ["X","X","Y","Y"],
"y": [5, 10, 3, 7],
"y_hat": [7, 7, 1, 10]
})
# errors:
# X: (7 - 5) + (7 - 10) = 2 + (-3) = -1
# Y: (1 - 3) + (10 - 7) = -2 + 3 = 1
expected = pd.DataFrame({
"unique_id": ["X", "Y"],
"y_hat": [-1, 1]
})
# pandas
out_pd = cfe(df, ["y_hat"])
pd.testing.assert_frame_equal(
out_pd,
expected
)
```
```python theme={null}
df_pl = pl.from_pandas(df)
out_pl = cfe(df_pl, ["y_hat"])
pl_assert_frame_equal(
out_pl,
pl.from_pandas(expected)
)
```
***
source
#### pis
> ```text theme={null}
> pis (df:~DFType, models:List[str], id_col:str='unique_id',
> target_col:str='y')
> ```
\*Compute the raw Absolute Periods In Stock (PIS) for one or multiple
models.
The PIS metric sums the absolute forecast errors per series without any
scaling, yielding a scale-dependent measure of bias.\*
| | **Type** | **Default** | **Details** |
| ----------- | ---------- | ----------- | ----------------------------------------------------------- |
| df | DFType | | Input dataframe with id, actual values and predictions. |
| models | List | | Columns that identify the models predictions. |
| id\_col | str | unique\_id | Column that identifies each serie. |
| target\_col | str | y | Column that contains the target. |
| **Returns** | **DFType** | | **dataframe with one row per id and one column per model.** |
```python theme={null}
pd_vs_pl(
pis(series, models),
pis(series_pl, models),
models,
)
```
```python theme={null}
# case for pis
df = pd.DataFrame({
"unique_id": ["A","A","B","B"],
"y": [10, 15, 5, 7],
"y_hat": [12, 14, 4, 10]
})
# errors:
# A: |12−10| + |14−15| = 2 + 1 = 3
# B: |4−5| + |10−7| = 1 + 3 = 4
expected = pd.DataFrame({
"unique_id": ["A", "B"],
"y_hat": [3, 4]
})
# pandas branch
out_pd = pis(df, ["y_hat"])
pd.testing.assert_frame_equal(
out_pd,
expected
)
```
```python theme={null}
df_pl = pl.from_pandas(df)
out_pl = pis(df_pl, ["y_hat"])
pl_assert_frame_equal(
out_pl,
pl.from_pandas(expected)
)
```
***
source
#### spis
> ```text theme={null}
> spis (df:~DFType, df_train:~DFType, models:List[str],
> id_col:str='unique_id', target_col:str='y')
> ```
\*Compute the scaled Absolute Periods In Stock (sAPIS) for one or
multiple models.
The sPIS metric scales the sum of absolute forecast errors by the mean
in-sample demand, yielding a scale-independent bias measure that can be
aggregated across series.\*
| | **Type** | **Default** | **Details** |
| ----------- | ---------- | ----------- | ----------------------------------------------------------- |
| df | DFType | | Input dataframe with id, actual values and predictions. |
| df\_train | DFType | | |
| models | List | | Columns that identify the models predictions. |
| id\_col | str | unique\_id | Column that identifies each serie. |
| target\_col | str | y | Column that contains the target. |
| **Returns** | **DFType** | | **dataframe with one row per id and one column per model.** |
```python theme={null}
pd_vs_pl(
spis(series, series, models),
spis(series_pl, series_pl, models),
models,
)
```
```text theme={null}
/tmp/ipykernel_12558/2968355870.py:19: FutureWarning: The default of observed=False is deprecated and will be changed to True in a future version of pandas. Pass observed=False to retain current behavior or observed=True to adopt the future default and silence this warning.
.groupby(id_col)[target_col]
```
```python theme={null}
# case for scaled pis
df_train = pd.DataFrame({
"unique_id": ["A","A","B","B"],
"y": [1, 3, 2, 6]
})
# Forecast data
df = pd.DataFrame({
"unique_id": ["A","A","B","B"],
"y": [3, 3, 2, 8],
"y_hat": [6, 2, 3, 5]
})
# For A: errors = |3−6|+|3−2| = 3+1 = 4 ÷ mean(1,3)=2 → 2
# For B: errors = |2−3|+|8−5| = 1+3 = 4 ÷ mean(2,6)=4 → 1
expected = pd.DataFrame({
"unique_id": ["A", "B"],
"y_hat": [2.0, 1.0]
})
# pandas branch
out_pd = spis(
df = df,
df_train = df_train,
models = ["y_hat"],
id_col = "unique_id",
target_col = "y",
)
pd.testing.assert_frame_equal(
out_pd,
expected
)
```
```python theme={null}
df_train_pl = pl.from_pandas(df_train)
df_pl = pl.from_pandas(df)
out_pl = spis(
df = df_pl,
df_train = df_train_pl,
models = ["y_hat"],
id_col = "unique_id",
target_col = "y",
)
pl.testing.assert_frame_equal(
out_pl,
pl.from_pandas(expected)
)
```
## 2. Percentage Errors
### Mean Absolute Percentage Error
$$
\mathrm{MAPE}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}_{\tau}) = \frac{1}{H} \sum^{t+H}_{\tau=t+1} \frac{|y_{\tau}-\hat{y}_{\tau}|}{|y_{\tau}|}
$$
***
source
#### mape
> ```text theme={null}
> mape (df:~DFType, models:List[str], id_col:str='unique_id',
> target_col:str='y')
> ```
\*Mean Absolute Percentage Error (MAPE)
MAPE measures the relative prediction accuracy of a forecasting method
by calculating the percentual deviation of the prediction and the
observed value at a given time and averages these devations over the
length of the series. The closer to zero an observed value is, the
higher penalty MAPE loss assigns to the corresponding error.\*
| | **Type** | **Default** | **Details** |
| ----------- | ---------- | ----------- | ----------------------------------------------------------- |
| df | DFType | | Input dataframe with id, actual values and predictions. |
| models | List | | Columns that identify the models predictions. |
| id\_col | str | unique\_id | Column that identifies each serie. |
| target\_col | str | y | Column that contains the target. |
| **Returns** | **DFType** | | **dataframe with one row per id and one column per model.** |
```python theme={null}
pd_vs_pl(
mape(series, models),
mape(series_pl, models),
models,
)
```
### Symmetric Mean Absolute Percentage Error
$$
\mathrm{SMAPE}_{2}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}_{\tau}) = \frac{1}{H} \sum^{t+H}_{\tau=t+1} \frac{|y_{\tau}-\hat{y}_{\tau}|}{|y_{\tau}|+|\hat{y}_{\tau}|}
$$
***
source
#### smape
> ```text theme={null}
> smape (df:~DFType, models:List[str], id_col:str='unique_id',
> target_col:str='y')
> ```
\*Symmetric Mean Absolute Percentage Error (SMAPE)
SMAPE measures the relative prediction accuracy of a forecasting method
by calculating the relative deviation of the prediction and the observed
value scaled by the sum of the absolute values for the prediction and
observed value at a given time, then averages these devations over the
length of the series. This allows the SMAPE to have bounds between 0%
and 100% which is desirable compared to normal MAPE that may be
undetermined when the target is zero.\*
| | **Type** | **Default** | **Details** |
| ----------- | ---------- | ----------- | ----------------------------------------------------------- |
| df | DFType | | Input dataframe with id, actual values and predictions. |
| models | List | | Columns that identify the models predictions. |
| id\_col | str | unique\_id | Column that identifies each serie. |
| target\_col | str | y | Column that contains the target. |
| **Returns** | **DFType** | | **dataframe with one row per id and one column per model.** |
```python theme={null}
pd_vs_pl(
smape(series, models),
smape(series_pl, models),
models,
)
```
## 3. Scale-independent Errors
### Mean Absolute Scaled Error
$$
\mathrm{MASE}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}_{\tau}, \mathbf{\hat{y}}^{season}_{\tau}) =
\frac{1}{H} \sum^{t+H}_{\tau=t+1} \frac{|y_{\tau}-\hat{y}_{\tau}|}{\mathrm{MAE}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}^{season}_{\tau})}
$$
***
source
#### mase
> ```text theme={null}
> mase (df:~DFType, models:List[str], seasonality:int, train_df:~DFType,
> id_col:str='unique_id', target_col:str='y')
> ```
\*Mean Absolute Scaled Error (MASE)
MASE measures the relative prediction accuracy of a forecasting method
by comparinng the mean absolute errors of the prediction and the
observed value against the mean absolute errors of the seasonal naive
model. The MASE partially composed the Overall Weighted Average (OWA),
used in the M4 Competition.\*
| | **Type** | **Default** | **Details** |
| ----------- | ---------- | ----------- | --------------------------------------------------------------------------------------------------------- |
| df | DFType | | Input dataframe with id, actuals and predictions. |
| models | List | | Columns that identify the models predictions. |
| seasonality | int | | Main frequency of the time series;
Hourly 24, Daily 7, Weekly 52, Monthly 12, Quarterly 4, Yearly 1. |
| train\_df | DFType | | Training dataframe with id and actual values. Must be sorted by time. |
| id\_col | str | unique\_id | Column that identifies each serie. |
| target\_col | str | y | Column that contains the target. |
| **Returns** | **DFType** | | **dataframe with one row per id and one column per model.** |
```python theme={null}
pd_vs_pl(
mase(series, models, 7, series),
mase(series_pl, models, 7, series_pl),
models,
)
```
### Relative Mean Absolute Error
$$
\mathrm{RMAE}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}_{\tau}, \mathbf{\hat{y}}^{base}_{\tau}) = \frac{1}{H} \sum^{t+H}_{\tau=t+1} \frac{|y_{\tau}-\hat{y}_{\tau}|}{\mathrm{MAE}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}^{base}_{\tau})}
$$
***
source
#### rmae
> ```text theme={null}
> rmae (df:~DFType, models:List[str], baseline:str, id_col:str='unique_id',
> target_col:str='y')
> ```
\*Relative Mean Absolute Error (RMAE)
Calculates the RAME between two sets of forecasts (from two different
forecasting methods). A number smaller than one implies that the
forecast in the numerator is better than the forecast in the
denominator.\*
| | **Type** | **Default** | **Details** |
| ----------- | ---------- | ----------- | ----------------------------------------------------------- |
| df | DFType | | Input dataframe with id, times, actuals and predictions. |
| models | List | | Columns that identify the models predictions. |
| baseline | str | | Column that identifies the baseline model predictions. |
| id\_col | str | unique\_id | Column that identifies each serie. |
| target\_col | str | y | Column that contains the target. |
| **Returns** | **DFType** | | **dataframe with one row per id and one column per model.** |
```python theme={null}
pd_vs_pl(
rmae(series, models, models[0]),
rmae(series_pl, models, models[0]),
models,
)
```
### Normalized Deviation
$$
\mathrm{ND}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}_{\tau}) = \frac{\sum^{t+H}_{\tau=t+1} |y_{\tau} - \hat{y}_{\tau}|}{\sum^{t+H}_{\tau=t+1} | y_{\tau} |}
$$
***
source
#### nd
> ```text theme={null}
> nd (df:~DFType, models:List[str], id_col:str='unique_id',
> target_col:str='y')
> ```
\*Normalized Deviation (ND)
ND measures the relative prediction accuracy of a forecasting method by
calculating the sum of the absolute deviation of the prediction and the
true value at a given time and dividing it by the sum of the absolute
value of the ground truth.\*
| | **Type** | **Default** | **Details** |
| ----------- | ---------- | ----------- | ----------------------------------------------------------- |
| df | DFType | | Input dataframe with id, times, actuals and predictions. |
| models | List | | Columns that identify the models predictions. |
| id\_col | str | unique\_id | Column that identifies each serie. |
| target\_col | str | y | Column that contains the target. |
| **Returns** | **DFType** | | **dataframe with one row per id and one column per model.** |
```python theme={null}
pd_vs_pl(
nd(series, models),
nd(series_pl, models),
models,
)
```
### Mean Squared Scaled Error
$$
\mathrm{MSSE}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}_{\tau}, \mathbf{\hat{y}}^{season}_{\tau}) =
\frac{1}{H} \sum^{t+H}_{\tau=t+1} \frac{(y_{\tau}-\hat{y}_{\tau})^2}{\mathrm{MSE}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}^{season}_{\tau})}
$$
***
source
#### msse
> ```text theme={null}
> msse (df:~DFType, models:List[str], seasonality:int, train_df:~DFType,
> id_col:str='unique_id', target_col:str='y')
> ```
\*Mean Squared Scaled Error (MSSE)
MSSE measures the relative prediction accuracy of a forecasting method
by comparinng the mean squared errors of the prediction and the observed
value against the mean squared errors of the seasonal naive model.\*
| | **Type** | **Default** | **Details** |
| ----------- | ---------- | ----------- | --------------------------------------------------------------------------------------------------------- |
| df | DFType | | Input dataframe with id, actuals and predictions. |
| models | List | | Columns that identify the models predictions. |
| seasonality | int | | Main frequency of the time series;
Hourly 24, Daily 7, Weekly 52, Monthly 12, Quarterly 4, Yearly 1. |
| train\_df | DFType | | Training dataframe with id and actual values. Must be sorted by time. |
| id\_col | str | unique\_id | Column that identifies each serie. |
| target\_col | str | y | Column that contains the target. |
| **Returns** | **DFType** | | **dataframe with one row per id and one column per model.** |
```python theme={null}
pd_vs_pl(
msse(series, models, 7, series),
msse(series_pl, models, 7, series_pl),
models,
)
```
### Root Mean Squared Scaled Error
$$
\mathrm{RMSSE}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}_{\tau}, \mathbf{\hat{y}}^{season}_{\tau}) =
\sqrt{\frac{1}{H} \sum^{t+H}_{\tau=t+1} \frac{(y_{\tau}-\hat{y}_{\tau})^2}{\mathrm{MSE}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}^{season}_{\tau})}}
$$
***
source
#### rmsse
> ```text theme={null}
> rmsse (df:~DFType, models:List[str], seasonality:int, train_df:~DFType,
> id_col:str='unique_id', target_col:str='y')
> ```
\*Root Mean Squared Scaled Error (RMSSE)
MSSE measures the relative prediction accuracy of a forecasting method
by comparinng the mean squared errors of the prediction and the observed
value against the mean squared errors of the seasonal naive model.\*
| | **Type** | **Default** | **Details** |
| ----------- | ---------- | ----------- | --------------------------------------------------------------------------------------------------------- |
| df | DFType | | Input dataframe with id, actuals and predictions. |
| models | List | | Columns that identify the models predictions. |
| seasonality | int | | Main frequency of the time series;
Hourly 24, Daily 7, Weekly 52, Monthly 12, Quarterly 4, Yearly 1. |
| train\_df | DFType | | Training dataframe with id and actual values. Must be sorted by time. |
| id\_col | str | unique\_id | Column that identifies each serie. |
| target\_col | str | y | Column that contains the target. |
| **Returns** | **DFType** | | **dataframe with one row per id and one column per model.** |
```python theme={null}
pd_vs_pl(
rmsse(series, models, 7, series),
rmsse(series_pl, models, 7, series_pl),
models,
)
```
## 4. Probabilistic Errors
### Quantile Loss
$$
\mathrm{QL}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}^{(q)}_{\tau}) =
\frac{1}{H} \sum^{t+H}_{\tau=t+1}
\Big( (1-q)\,( \hat{y}^{(q)}_{\tau} - y_{\tau} )_{+}
+ q\,( y_{\tau} - \hat{y}^{(q)}_{\tau} )_{+} \Big)
$$
***
source
#### quantile\_loss
> ```text theme={null}
> quantile_loss (df:~DFType, models:Dict[str,str], q:float=0.5,
> id_col:str='unique_id', target_col:str='y')
> ```
\*Quantile Loss (QL)
QL measures the deviation of a quantile forecast. By weighting the
absolute deviation in a non symmetric way, the loss pays more attention
to under or over estimation.\
A common value for q is 0.5 for the deviation from the median.\*
| | **Type** | **Default** | **Details** |
| ----------- | ---------- | ----------- | ---------------------------------------------------------------------------- |
| df | DFType | | Input dataframe with id, times, actuals and predictions. |
| models | Dict | | Mapping from model name to the model predictions for the specified quantile. |
| q | float | 0.5 | Quantile for the predictions’ comparison. |
| id\_col | str | unique\_id | Column that identifies each serie. |
| target\_col | str | y | Column that contains the target. |
| **Returns** | **DFType** | | **dataframe with one row per id and one column per model.** |
### Scaled Quantile Loss
$$
\mathrm{SQL}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}^{(q)}_{\tau}) =
\frac{1}{H} \sum^{t+H}_{\tau=t+1}
\frac{(1-q)\,( \hat{y}^{(q)}_{\tau} - y_{\tau} )_{+}
+ q\,( y_{\tau} - \hat{y}^{(q)}_{\tau} )_{+}}{\mathrm{MAE}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}^{season}_{\tau})}
$$
***
source
### scaled\_quantile\_loss
> ```text theme={null}
> scaled_quantile_loss (df:~DFType, models:Dict[str,str], seasonality:int,
> train_df:~DFType, q:float=0.5,
> id_col:str='unique_id', target_col:str='y')
> ```
\*Scaled Quantile Loss (SQL)
SQL measures the deviation of a quantile forecast scaled by the mean
absolute errors of the seasonal naive model. By weighting the absolute
deviation in a non symmetric way, the loss pays more attention to under
or over estimation. A common value for q is 0.5 for the deviation from
the median. This was the official measure used in the M5 Uncertainty
competition with seasonality = 1.\*
| | **Type** | **Default** | **Details** |
| ----------- | ---------- | ----------- | --------------------------------------------------------------------------------------------------------- |
| df | DFType | | Input dataframe with id, times, actuals and predictions. |
| models | Dict | | Mapping from model name to the model predictions for the specified quantile. |
| seasonality | int | | Main frequency of the time series;
Hourly 24, Daily 7, Weekly 52, Monthly 12, Quarterly 4, Yearly 1. |
| train\_df | DFType | | Training dataframe with id and actual values. Must be sorted by time. |
| q | float | 0.5 | Quantile for the predictions’ comparison. |
| id\_col | str | unique\_id | Column that identifies each serie. |
| target\_col | str | y | Column that contains the target. |
| **Returns** | **DFType** | | **dataframe with one row per id and one column per model.** |
### Multi-Quantile Loss
$$
\mathrm{MQL}(\mathbf{y}_{\tau},
[\mathbf{\hat{y}}^{(q_{1})}_{\tau}, ... ,\hat{y}^{(q_{n})}_{\tau}]) =
\frac{1}{n} \sum_{q_{i}} \mathrm{QL}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}^{(q_{i})}_{\tau})
$$
***
source
#### mqloss
> ```text theme={null}
> mqloss (df:~DFType, models:Dict[str,List[str]], quantiles:numpy.ndarray,
> id_col:str='unique_id', target_col:str='y')
> ```
\*Multi-Quantile loss (MQL)
MQL calculates the average multi-quantile Loss for a given set of
quantiles, based on the absolute difference between predicted quantiles
and observed values.
The limit behavior of MQL allows to measure the accuracy of a full
predictive distribution with the continuous ranked probability score
(CRPS). This can be achieved through a numerical integration technique,
that discretizes the quantiles and treats the CRPS integral with a left
Riemann approximation, averaging over uniformly distanced quantiles.\*
| | **Type** | **Default** | **Details** |
| ----------- | ---------- | ----------- | ------------------------------------------------------------------- |
| df | DFType | | Input dataframe with id, times, actuals and predictions. |
| models | Dict | | Mapping from model name to the model predictions for each quantile. |
| quantiles | ndarray | | Quantiles to compare against. |
| id\_col | str | unique\_id | Column that identifies each serie. |
| target\_col | str | y | Column that contains the target. |
| **Returns** | **DFType** | | **dataframe with one row per id and one column per model.** |
```python theme={null}
pd_vs_pl(
mqloss(series, mq_models, quantiles=quantiles),
mqloss(series_pl, mq_models, quantiles=quantiles),
models,
)
```
### Scaled Multi-Quantile Loss
$$
\mathrm{MQL}(\mathbf{y}_{\tau},
[\mathbf{\hat{y}}^{(q_{1})}_{\tau}, ... ,\hat{y}^{(q_{n})}_{\tau}]) =
\frac{1}{n} \sum_{q_{i}} \frac{\mathrm{QL}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}^{(q_{i})}_{\tau})}{\mathrm{MAE}(\mathbf{y}_{\tau}, \mathbf{\hat{y}}^{season}_{\tau})}
$$
***
source
### scaled\_mqloss
> ```text theme={null}
> scaled_mqloss (df:~DFType, models:Dict[str,List[str]],
> quantiles:numpy.ndarray, seasonality:int,
> train_df:~DFType, id_col:str='unique_id',
> target_col:str='y')
> ```
\*Scaled Multi-Quantile loss (SMQL)
SMQL calculates the average multi-quantile Loss for a given set of
quantiles, based on the absolute difference between predicted quantiles
and observed values scaled by the mean absolute errors of the seasonal
naive model. The limit behavior of MQL allows to measure the accuracy of
a full predictive distribution with the continuous ranked probability
score (CRPS). This can be achieved through a numerical integration
technique, that discretizes the quantiles and treats the CRPS integral
with a left Riemann approximation, averaging over uniformly distanced
quantiles. This was the official measure used in the M5 Uncertainty
competition with seasonality = 1.\*
| | **Type** | **Default** | **Details** |
| ----------- | ---------- | ----------- | --------------------------------------------------------------------------------------------------------- |
| df | DFType | | Input dataframe with id, times, actuals and predictions. |
| models | Dict | | Mapping from model name to the model predictions for each quantile. |
| quantiles | ndarray | | Quantiles to compare against. |
| seasonality | int | | Main frequency of the time series;
Hourly 24, Daily 7, Weekly 52, Monthly 12, Quarterly 4, Yearly 1. |
| train\_df | DFType | | Training dataframe with id and actual values. Must be sorted by time. |
| id\_col | str | unique\_id | Column that identifies each serie. |
| target\_col | str | y | Column that contains the target. |
| **Returns** | **DFType** | | **dataframe with one row per id and one column per model.** |
```python theme={null}
pd_vs_pl(
scaled_mqloss(series, mq_models, quantiles=quantiles, seasonality=1, train_df=series),
scaled_mqloss(series_pl, mq_models, quantiles=quantiles, seasonality=1, train_df=series_pl),
models,
)
```
### Coverage
***
source
#### coverage
> ```text theme={null}
> coverage (df:~DFType, models:List[str], level:int,
> id_col:str='unique_id', target_col:str='y')
> ```
*Coverage of y with y\_hat\_lo and y\_hat\_hi.*
| | **Type** | **Default** | **Details** |
| ----------- | ---------- | ----------- | ----------------------------------------------------------- |
| df | DFType | | Input dataframe with id, times, actuals and predictions. |
| models | List | | Columns that identify the models predictions. |
| level | int | | Confidence level used for intervals. |
| id\_col | str | unique\_id | Column that identifies each serie. |
| target\_col | str | y | Column that contains the target. |
| **Returns** | **DFType** | | **dataframe with one row per id and one column per model.** |
```python theme={null}
pd_vs_pl(
coverage(series, models, 80),
coverage(series_pl, models, 80),
models,
)
```
### Calibration
***
source
#### calibration
> ```text theme={null}
> calibration (df:~DFType, models:Dict[str,str], id_col:str='unique_id',
> target_col:str='y')
> ```
*Fraction of y that is lower than the model’s predictions.*
| | **Type** | **Default** | **Details** |
| ----------- | ---------- | ----------- | ----------------------------------------------------------- |
| df | DFType | | Input dataframe with id, times, actuals and predictions. |
| models | Dict | | Mapping from model name to the model predictions. |
| id\_col | str | unique\_id | Column that identifies each serie. |
| target\_col | str | y | Column that contains the target. |
| **Returns** | **DFType** | | **dataframe with one row per id and one column per model.** |
```python theme={null}
pd_vs_pl(
calibration(series, q_models[0.1]),
calibration(series_pl, q_models[0.1]),
models,
)
```
### CRPS
$$
\mathrm{sCRPS}(\hat{F}_{\tau}, \mathbf{y}_{\tau}) = \frac{2}{N} \sum_{i}
\int^{1}_{0} \frac{\mathrm{QL}(\hat{F}_{i,\tau}, y_{i,\tau})_{q}}{\sum_{i} | y_{i,\tau} |} dq
$$
Where $\hat{F}_{\tau}$ is the an estimated multivariate distribution,
and $y_{i,\tau}$ are its realizations.
***
source
#### scaled\_crps
> ```text theme={null}
> scaled_crps (df:~DFType, models:Dict[str,List[str]],
> quantiles:numpy.ndarray, id_col:str='unique_id',
> target_col:str='y')
> ```
\*Scaled Continues Ranked Probability Score
Calculates a scaled variation of the CRPS, as proposed by Rangapuram
(2021), to measure the accuracy of predicted quantiles `y_hat` compared
to the observation `y`. This metric averages percentual weighted
absolute deviations as defined by the quantile losses.\*
| | **Type** | **Default** | **Details** |
| ----------- | ---------- | ----------- | ------------------------------------------------------------------- |
| df | DFType | | Input dataframe with id, times, actuals and predictions. |
| models | Dict | | Mapping from model name to the model predictions for each quantile. |
| quantiles | ndarray | | Quantiles to compare against. |
| id\_col | str | unique\_id | Column that identifies each serie. |
| target\_col | str | y | Column that contains the target. |
| **Returns** | **DFType** | | **dataframe with one row per id and one column per model.** |
```python theme={null}
pd_vs_pl(
scaled_crps(series, mq_models, quantiles),
scaled_crps(series_pl, mq_models, quantiles),
models,
)
```
### Tweedie Deviance
For a set of forecasts $\{\mu_i\}_{i=1}^N$ and observations
$\{y_i\}_{i=1}^N$, the mean Tweedie deviance with power $p$ is
$$
\mathrm{TD}_{p}(\boldsymbol{\mu}, \mathbf{y})
= \frac{1}{N} \sum_{i=1}^{N} d_{p}(y_i, \mu_i)
$$
where the unit-scaled deviance for each pair $(y,\mu)$ is
$$
d_{p}(y,\mu)
=
2
\begin{cases}
\displaystyle
\frac{y^{2-p}}{(1-p)(2-p)}
\;-\;
\frac{y\,\mu^{1-p}}{1-p}
\;+\;
\frac{\mu^{2-p}}{2-p},
& p \notin\{1,2\},\\[1em]
\displaystyle
y\,\ln\!\frac{y}{\mu}\;-\;(y-\mu),
& p = 1\quad(\text{Poisson deviance}),\\[0.5em]
\displaystyle
-2\Bigl[\ln\!\frac{y}{\mu}\;-\;\frac{y-\mu}{\mu}\Bigr],
& p = 2\quad(\text{Gamma deviance}).
\end{cases}
$$
* $y_i$ are the true values, $\mu_i$ the predicted means.
* $p$ controls the variance relationship
$\mathrm{Var}(Y)\propto\mu^{p}$.
* When $1source
#### tweedie\_deviance
> ```text theme={null}
> tweedie_deviance (df:~DFType, models:List[str], power:float=1.5,
> id_col:str='unique_id', target_col:str='y')
> ```
\*Compute the Tweedie deviance loss for one or multiple models, grouped
by an identifier.
Each group’s deviance is calculated using the mean\_tweedie\_deviance
function, which measures the deviation between actual and predicted
values under the Tweedie distribution.
The `power` parameter defines the specific compound distribution: - 1:
Poisson - (1, 2): Compound Poisson-Gamma - 2: Gamma - >2: Inverse
Gaussian\*
| | **Type** | **Default** | **Details** |
| ----------- | ---------- | ----------- | ------------------------------------------------------------------------------------------------------ |
| df | DFType | | Input dataframe with id, actuals and predictions. |
| models | List | | Columns that identify the models predictions. |
| power | float | 1.5 | Tweedie power parameter. Determines the compound distribution. |
| id\_col | str | unique\_id | Column that identifies each serie. |
| target\_col | str | y | Column that contains the target. |
| **Returns** | **DFType** | | \*\*DataFrame with one row per id and one column per model, containing the mean Tweedie deviance. \*\* |
```python theme={null}
# Normal test
for power in [0, 1, 1.5, 2, 3]:
# Test Pandas vs Polars
td_pd = tweedie_deviance(series, models, target_col="y", power=power)
td_pl = tweedie_deviance(series_pl, models, target_col="y", power=power)
pd_vs_pl(
td_pd,
td_pl,
models,
)
# Test for NaNs
assert not td_pd[models].isna().any().any(), f"NaNs found in pd DataFrame for power {power}"
assert not td_pl.select(pl.col(models).is_null().any()).sum_horizontal().item(), f"NaNs found in pl DataFrame for power {power}"
# Test for infinites
is_infinite = td_pd[models].isin([np.inf, -np.inf]).any().any()
assert not is_infinite, f"Infinities found in pd DataFrame for power {power}"
is_infinite_pl = td_pl.select(pl.col(models).is_infinite().any()).sum_horizontal().item()
assert not is_infinite_pl, f"Infinities found in pl DataFrame for power {power}"
# Test zero handling (skip power >=2 since it requires all y > 0)
series.loc[0, 'y'] = 0.0 # Set a zero value to test the zero handling
series.loc[49, 'y'] = 0.0 # Set another zero value to test the zero handling
series_pl[0, 'y'] = 0.0 # Set a zero value to test the zero handling
series_pl[49, 'y'] = 0.0 # Set another zero value to test the zero handling
for power in [0, 1, 1.5]:
# Test Pandas vs Polars
td_pd = tweedie_deviance(series, models, target_col="y", power=power)
td_pl = tweedie_deviance(series_pl, models, target_col="y", power=power)
pd_vs_pl(
td_pd,
td_pl,
models,
)
# Test for NaNs
assert not td_pd[models].isna().any().any(), f"NaNs found in pd DataFrame for power {power}"
assert not td_pl.select(pl.col(models).is_null().any()).sum_horizontal().item(), f"NaNs found in pl DataFrame for power {power}"
# Test for infinites
is_infinite = td_pd[models].isin([np.inf, -np.inf]).any().any()
assert not is_infinite, f"Infinities found in pd DataFrame for power {power}"
is_infinite_pl = td_pl.select(pl.col(models).is_infinite().any()).sum_horizontal().item()
assert not is_infinite_pl, f"Infinities found in pl DataFrame for power {power}"
```