> ## Documentation Index
> Fetch the complete documentation index at: https://nixtla-old-docs.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

> Model performance evaluation

# Evaluation

***

<a href="https://github.com/Nixtla/utilsforecast/blob/main/utilsforecast/evaluation.py#L156" target="_blank" style={{ float: "right", fontSize: "smaller" }}>source</a>

### evaluate

> ```text theme={null}
>  evaluate (df:~AnyDFType, metrics:List[Callable],
>            models:Optional[List[str]]=None,
>            train_df:Optional[~AnyDFType]=None,
>            level:Optional[List[int]]=None, id_col:str='unique_id',
>            time_col:str='ds', target_col:str='y',
>            agg_fn:Optional[str]=None)
> ```

*Evaluate forecast using different metrics.*

|             | **Type**      | **Default** | **Details**                                                                                                                                       |
| ----------- | ------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| df          | AnyDFType     |             | Forecasts to evaluate.<br />Must have `id_col`, `time_col`, `target_col` and models’ predictions.                                                 |
| metrics     | List          |             | Functions with arguments `df`, `models`, `id_col`, `target_col` and optionally `train_df`.                                                        |
| models      | Optional      | None        | Names of the models to evaluate.<br />If `None` will use every column in the dataframe after removing id, time and target.                        |
| train\_df   | Optional      | None        | Training set. Used to evaluate metrics such as [`mase`](https://Nixtla.github.io/utilsforecast/losses.html#mase).                                 |
| level       | Optional      | None        | Prediction interval levels. Used to compute losses that rely on quantiles.                                                                        |
| id\_col     | str           | unique\_id  | Column that identifies each serie.                                                                                                                |
| time\_col   | str           | ds          | Column that identifies each timestep, its values can be timestamps or integers.                                                                   |
| target\_col | str           | y           | Column that contains the target.                                                                                                                  |
| agg\_fn     | Optional      | None        | Statistic to compute on the scores by id to reduce them to a single number.                                                                       |
| **Returns** | **AnyDFType** |             | **Metrics with one row per (id, metric) combination and one column per model.<br />If `agg_fn` is not `None`, there is only one row per metric.** |

```python theme={null}
from functools import partial

import numpy as np
import pandas as pd

from utilsforecast.losses import *
from utilsforecast.data import generate_series
```

```python theme={null}
series = generate_series(10, n_models=2, level=[80, 95])
```

```python theme={null}
series['unique_id'] = series['unique_id'].astype('int')
```

```python theme={null}
models = ['model0', 'model1']
metrics = [
    mae,
    mse,
    rmse,
    mape,
    smape,
    partial(mase, seasonality=7),
    quantile_loss,
    mqloss,
    coverage,
    calibration,
    scaled_crps,
]
```

```python theme={null}
evaluation = evaluate(
    series,
    metrics=metrics,
    models=models,
    train_df=series,
    level=[80, 95],
)
evaluation
```

|     | unique\_id | metric       | model0   | model1   |
| --- | ---------- | ------------ | -------- | -------- |
| 0   | 0          | mae          | 0.158108 | 0.163246 |
| 1   | 1          | mae          | 0.160109 | 0.143805 |
| 2   | 2          | mae          | 0.159815 | 0.170510 |
| 3   | 3          | mae          | 0.168537 | 0.161595 |
| 4   | 4          | mae          | 0.170182 | 0.163329 |
| ... | ...        | ...          | ...      | ...      |
| 175 | 5          | scaled\_crps | 0.034202 | 0.035472 |
| 176 | 6          | scaled\_crps | 0.034880 | 0.033610 |
| 177 | 7          | scaled\_crps | 0.034337 | 0.034745 |
| 178 | 8          | scaled\_crps | 0.033336 | 0.032459 |
| 179 | 9          | scaled\_crps | 0.034766 | 0.035243 |

```python theme={null}
summary = evaluation.drop(columns='unique_id').groupby('metric').mean().reset_index()
summary
```

|    | metric                 | model0   | model1   |
| -- | ---------------------- | -------- | -------- |
| 0  | calibration\_q0.025    | 0.000000 | 0.000000 |
| 1  | calibration\_q0.1      | 0.000000 | 0.000000 |
| 2  | calibration\_q0.9      | 0.833993 | 0.815833 |
| 3  | calibration\_q0.975    | 0.853991 | 0.836949 |
| 4  | coverage\_level80      | 0.833993 | 0.815833 |
| 5  | coverage\_level95      | 0.853991 | 0.836949 |
| 6  | mae                    | 0.161286 | 0.162281 |
| 7  | mape                   | 0.048894 | 0.049624 |
| 8  | mase                   | 0.966846 | 0.975354 |
| 9  | mqloss                 | 0.056904 | 0.056216 |
| 10 | mse                    | 0.048653 | 0.049198 |
| 11 | quantile\_loss\_q0.025 | 0.019990 | 0.019474 |
| 12 | quantile\_loss\_q0.1   | 0.067315 | 0.065781 |
| 13 | quantile\_loss\_q0.9   | 0.095510 | 0.093841 |
| 14 | quantile\_loss\_q0.975 | 0.044803 | 0.045767 |
| 15 | rmse                   | 0.220357 | 0.221543 |
| 16 | scaled\_crps           | 0.035003 | 0.034576 |
| 17 | smape                  | 0.024475 | 0.024902 |