> ## Documentation Index
> Fetch the complete documentation index at: https://nixtla-old-docs.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.
> Utilities for processing data before training/analysis
# Preprocessing
***
source
### id\_time\_grid
> ```text theme={null}
> id_time_grid (df:~DFType, freq:Union[str,int],
> start:Union[str,int,datetime.date,datetime.datetime]='per_s
> erie', end:Union[str,int,datetime.date,datetime.datetime]='
> global', id_col:str='unique_id', time_col:str='ds')
> ```
*Generate all expected combiations of ids and times.*
| | **Type** | **Default** | **Details** |
| ----------- | ---------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| df | DFType | | Input data |
| freq | Union | | Series’ frequency |
| start | Union | per\_serie | Initial timestamp for the series.
\* ‘per\_serie’ uses each serie’s first timestamp
\* ‘global’ uses the first timestamp seen in the data
\* Can also be a specific timestamp or integer, e.g. ‘2000-01-01’, 2000 or datetime(2000, 1, 1) |
| end | Union | global | Initial timestamp for the series.
\* ‘per\_serie’ uses each serie’s last timestamp
\* ‘global’ uses the last timestamp seen in the data
\* Can also be a specific timestamp or integer, e.g. ‘2000-01-01’, 2000 or datetime(2000, 1, 1) |
| id\_col | str | unique\_id | Column that identifies each serie. |
| time\_col | str | ds | Column that identifies each timestamp. |
| **Returns** | **DFType** | | **Dataframe with expected ids and times.** |
***
source
### fill\_gaps
> ```text theme={null}
> fill_gaps (df:~DFType, freq:Union[str,int],
> start:Union[str,int,datetime.date,datetime.datetime]='per_seri
> e',
> end:Union[str,int,datetime.date,datetime.datetime]='global',
> id_col:str='unique_id', time_col:str='ds')
> ```
*Enforce start and end datetimes for dataframe.*
| | **Type** | **Default** | **Details** |
| ----------- | ---------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| df | DFType | | Input data |
| freq | Union | | Series’ frequency |
| start | Union | per\_serie | Initial timestamp for the series.
\* ‘per\_serie’ uses each serie’s first timestamp
\* ‘global’ uses the first timestamp seen in the data
\* Can also be a specific timestamp or integer, e.g. ‘2000-01-01’, 2000 or datetime(2000, 1, 1) |
| end | Union | global | Initial timestamp for the series.
\* ‘per\_serie’ uses each serie’s last timestamp
\* ‘global’ uses the last timestamp seen in the data
\* Can also be a specific timestamp or integer, e.g. ‘2000-01-01’, 2000 or datetime(2000, 1, 1) |
| id\_col | str | unique\_id | Column that identifies each serie. |
| time\_col | str | ds | Column that identifies each timestamp. |
| **Returns** | **DFType** | | **Dataframe with gaps filled.** |
```python theme={null}
df = pd.DataFrame(
{
'unique_id': [0, 0, 0, 1, 1],
'ds': pd.to_datetime(['2020', '2021', '2023', '2021', '2022']),
'y': np.arange(5),
}
)
df
```
| | unique\_id | ds | y |
| - | ---------- | ---------- | - |
| 0 | 0 | 2020-01-01 | 0 |
| 1 | 0 | 2021-01-01 | 1 |
| 2 | 0 | 2023-01-01 | 2 |
| 3 | 1 | 2021-01-01 | 3 |
| 4 | 1 | 2022-01-01 | 4 |
The default functionality is taking the current starts and only
extending the end date to be the same for all series.
```python theme={null}
fill_gaps(
df,
freq='YS',
)
```
| | unique\_id | ds | y |
| - | ---------- | ---------- | --- |
| 0 | 0 | 2020-01-01 | 0.0 |
| 1 | 0 | 2021-01-01 | 1.0 |
| 2 | 0 | 2022-01-01 | NaN |
| 3 | 0 | 2023-01-01 | 2.0 |
| 4 | 1 | 2021-01-01 | 3.0 |
| 5 | 1 | 2022-01-01 | 4.0 |
| 6 | 1 | 2023-01-01 | NaN |
We can also specify `end='per_serie'` to only fill possible gaps within
each serie.
```python theme={null}
fill_gaps(
df,
freq='YS',
end='per_serie',
)
```
| | unique\_id | ds | y |
| - | ---------- | ---------- | --- |
| 0 | 0 | 2020-01-01 | 0.0 |
| 1 | 0 | 2021-01-01 | 1.0 |
| 2 | 0 | 2022-01-01 | NaN |
| 3 | 0 | 2023-01-01 | 2.0 |
| 4 | 1 | 2021-01-01 | 3.0 |
| 5 | 1 | 2022-01-01 | 4.0 |
We can also specify an end date in the future.
```python theme={null}
fill_gaps(
df,
freq='YS',
end='2024',
)
```
| | unique\_id | ds | y |
| - | ---------- | ---------- | --- |
| 0 | 0 | 2020-01-01 | 0.0 |
| 1 | 0 | 2021-01-01 | 1.0 |
| 2 | 0 | 2022-01-01 | NaN |
| 3 | 0 | 2023-01-01 | 2.0 |
| 4 | 0 | 2024-01-01 | NaN |
| 5 | 1 | 2021-01-01 | 3.0 |
| 6 | 1 | 2022-01-01 | 4.0 |
| 7 | 1 | 2023-01-01 | NaN |
| 8 | 1 | 2024-01-01 | NaN |
We can set all series to start at the same time.
```python theme={null}
fill_gaps(
df,
freq='YS',
start='global'
)
```
| | unique\_id | ds | y |
| - | ---------- | ---------- | --- |
| 0 | 0 | 2020-01-01 | 0.0 |
| 1 | 0 | 2021-01-01 | 1.0 |
| 2 | 0 | 2022-01-01 | NaN |
| 3 | 0 | 2023-01-01 | 2.0 |
| 4 | 1 | 2020-01-01 | NaN |
| 5 | 1 | 2021-01-01 | 3.0 |
| 6 | 1 | 2022-01-01 | 4.0 |
| 7 | 1 | 2023-01-01 | NaN |
We can also set a common start date for all series (which can be earlier
than their current starts).
```python theme={null}
fill_gaps(
df,
freq='YS',
start='2019',
)
```
| | unique\_id | ds | y |
| - | ---------- | ---------- | --- |
| 0 | 0 | 2019-01-01 | NaN |
| 1 | 0 | 2020-01-01 | 0.0 |
| 2 | 0 | 2021-01-01 | 1.0 |
| 3 | 0 | 2022-01-01 | NaN |
| 4 | 0 | 2023-01-01 | 2.0 |
| 5 | 1 | 2019-01-01 | NaN |
| 6 | 1 | 2020-01-01 | NaN |
| 7 | 1 | 2021-01-01 | 3.0 |
| 8 | 1 | 2022-01-01 | 4.0 |
| 9 | 1 | 2023-01-01 | NaN |
In case the times are integers the frequency, start and end must also be
integers.
```python theme={null}
df = pd.DataFrame(
{
'unique_id': [0, 0, 0, 1, 1],
'ds': [2020, 2021, 2023, 2021, 2022],
'y': np.arange(5),
}
)
df
```
| | unique\_id | ds | y |
| - | ---------- | ---- | - |
| 0 | 0 | 2020 | 0 |
| 1 | 0 | 2021 | 1 |
| 2 | 0 | 2023 | 2 |
| 3 | 1 | 2021 | 3 |
| 4 | 1 | 2022 | 4 |
```python theme={null}
fill_gaps(
df,
freq=1,
start=2019,
end=2024,
)
```
| | unique\_id | ds | y |
| -- | ---------- | ---- | --- |
| 0 | 0 | 2019 | NaN |
| 1 | 0 | 2020 | 0.0 |
| 2 | 0 | 2021 | 1.0 |
| 3 | 0 | 2022 | NaN |
| 4 | 0 | 2023 | 2.0 |
| 5 | 0 | 2024 | NaN |
| 6 | 1 | 2019 | NaN |
| 7 | 1 | 2020 | NaN |
| 8 | 1 | 2021 | 3.0 |
| 9 | 1 | 2022 | 4.0 |
| 10 | 1 | 2023 | NaN |
| 11 | 1 | 2024 | NaN |
The function also accepts polars dataframes
```python theme={null}
df = pl.DataFrame(
{
'unique_id': [0, 0, 0, 1, 1],
'ds': [
datetime(2020, 1, 1), datetime(2022, 1, 1), datetime(2023, 1, 1),
datetime(2021, 1, 1), datetime(2022, 1, 1)],
'y': np.arange(5),
}
)
df
```
| unique\_id | ds | y |
| ---------- | ------------------- | --- |
| i64 | datetime\[μs] | i64 |
| 0 | 2020-01-01 00:00:00 | 0 |
| 0 | 2022-01-01 00:00:00 | 1 |
| 0 | 2023-01-01 00:00:00 | 2 |
| 1 | 2021-01-01 00:00:00 | 3 |
| 1 | 2022-01-01 00:00:00 | 4 |
```python theme={null}
polars_ms = fill_gaps(
df.with_columns(pl.col('ds').cast(pl.Datetime(time_unit='ms'))),
freq='1y',
start=datetime(2019, 1, 1),
end=datetime(2024, 1, 1),
)
assert polars_ms.schema['ds'].time_unit == 'ms'
polars_ms
```
| unique\_id | ds | y |
| ---------- | ------------------- | ---- |
| i64 | datetime\[ms] | i64 |
| 0 | 2019-01-01 00:00:00 | null |
| 0 | 2020-01-01 00:00:00 | 0 |
| 0 | 2021-01-01 00:00:00 | null |
| 0 | 2022-01-01 00:00:00 | 1 |
| 0 | 2023-01-01 00:00:00 | 2 |
| … | … | … |
| 1 | 2020-01-01 00:00:00 | null |
| 1 | 2021-01-01 00:00:00 | 3 |
| 1 | 2022-01-01 00:00:00 | 4 |
| 1 | 2023-01-01 00:00:00 | null |
| 1 | 2024-01-01 00:00:00 | null |
```python theme={null}
df = pl.DataFrame(
{
'unique_id': [0, 0, 0, 1, 1],
'ds': [
date(2020, 1, 1), date(2022, 1, 1), date(2023, 1, 1),
date(2021, 1, 1), date(2022, 1, 1)],
'y': np.arange(5),
}
)
df
```
| unique\_id | ds | y |
| ---------- | ---------- | --- |
| i64 | date | i64 |
| 0 | 2020-01-01 | 0 |
| 0 | 2022-01-01 | 1 |
| 0 | 2023-01-01 | 2 |
| 1 | 2021-01-01 | 3 |
| 1 | 2022-01-01 | 4 |
```python theme={null}
fill_gaps(
df,
freq='1y',
start=date(2020, 1, 1),
end=date(2024, 1, 1),
)
```
| unique\_id | ds | y |
| ---------- | ---------- | ---- |
| i64 | date | i64 |
| 0 | 2020-01-01 | 0 |
| 0 | 2021-01-01 | null |
| 0 | 2022-01-01 | 1 |
| 0 | 2023-01-01 | 2 |
| 0 | 2024-01-01 | null |
| 1 | 2020-01-01 | null |
| 1 | 2021-01-01 | 3 |
| 1 | 2022-01-01 | 4 |
| 1 | 2023-01-01 | null |
| 1 | 2024-01-01 | null |
```python theme={null}
df = pl.DataFrame(
{
'unique_id': [0, 0, 0, 1, 1],
'ds': [2020, 2021, 2023, 2021, 2022],
'y': np.arange(5),
}
)
df
```
| unique\_id | ds | y |
| ---------- | ---- | --- |
| i64 | i64 | i64 |
| 0 | 2020 | 0 |
| 0 | 2021 | 1 |
| 0 | 2023 | 2 |
| 1 | 2021 | 3 |
| 1 | 2022 | 4 |
```python theme={null}
fill_gaps(
df,
freq=1,
start=2019,
end=2024,
)
```
| unique\_id | ds | y |
| ---------- | ---- | ---- |
| i64 | i64 | i64 |
| 0 | 2019 | null |
| 0 | 2020 | 0 |
| 0 | 2021 | 1 |
| 0 | 2022 | null |
| 0 | 2023 | 2 |
| … | … | … |
| 1 | 2020 | null |
| 1 | 2021 | 3 |
| 1 | 2022 | 4 |
| 1 | 2023 | null |
| 1 | 2024 | null |