> ## Documentation Index
> Fetch the complete documentation index at: https://nixtla-old-docs.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.
> Machine Learning forecasting methods are defined by many hyperparameters that control their behavior, with effects ranging from their speed and memory requirements to their predictive performance. For a long time, manual hyperparameter tuning prevailed. This approach is time-consuming, **automated hyperparameter optimization** methods have been introduced, proving more efficient than manual tuning, grid search, and random search.
The `BaseAuto` class offers shared API connections to hyperparameter optimization algorithms like [Optuna](https://docs.ray.io/en/latest/tune/examples/bayesopt_example.html), [HyperOpt](https://docs.ray.io/en/latest/tune/examples/hyperopt_example.html), [Dragonfly](https://docs.ray.io/en/releases-2.7.0/tune/examples/dragonfly_example.html) among others through `ray`, which gives you access to grid search, bayesian optimization and other state-of-the-art tools like hyperband.
Comprehending the impacts of hyperparameters is still a precious skill, as it can help guide the design of informed hyperparameter spaces that are faster to explore automatically.
# Hyperparameter Optimization
Figure 1. Example of dataset split
(left), validation (yellow) and test (orange). The hyperparameter
optimization guiding signal is obtained from the validation
set.
***
### BaseAuto
> ```text theme={null}
> BaseAuto (cls_model, h, loss, valid_loss, config,
> search_alg= object at 0x7f820028a2f0>, num_samples=10, cpus=4, gpus=0,
> refit_with_val=False, verbose=False, alias=None, backend='ray',
> callbacks=None)
> ```
\*Class for Automatic Hyperparameter Optimization, it builds on top of
`ray` to give access to a wide variety of hyperparameter optimization
tools ranging from classic grid search, to Bayesian optimization and
HyperBand algorithm.
The validation loss to be optimized is defined by the `config['loss']`
dictionary value, the config also contains the rest of the
hyperparameter search space.
It is important to note that the success of this hyperparameter
optimization heavily relies on a strong correlation between the
validation and test periods.\*
| | **Type** | **Default** | **Details** |
| ---------------- | ------------------------------ | -------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| cls\_model | PyTorch/PyTorchLightning model | | See `neuralforecast.models` [collection here](https://nixtla.github.io/neuralforecast/models.html). |
| h | int | | Forecast horizon |
| loss | PyTorch module | | Instantiated train loss class from [losses collection](https://nixtla.github.io/neuralforecast/losses.pytorch.html). |
| valid\_loss | PyTorch module | | Instantiated valid loss class from [losses collection](https://nixtla.github.io/neuralforecast/losses.pytorch.html). |
| config | dict or callable | | Dictionary with ray.tune defined search space or function that takes an optuna trial and returns a configuration dict. |
| search\_alg | BasicVariantGenerator | \ | For ray see [https://docs.ray.io/en/latest/tune/api\_docs/suggestion.html](https://docs.ray.io/en/latest/tune/api_docs/suggestion.html) For optuna see [https://optuna.readthedocs.io/en/stable/reference/samplers/index.html](https://optuna.readthedocs.io/en/stable/reference/samplers/index.html). |
| num\_samples | int | 10 | Number of hyperparameter optimization steps/samples. |
| cpus | int | 4 | Number of cpus to use during optimization. Only used with ray tune. |
| gpus | int | 0 | Number of gpus to use during optimization, default all available. Only used with ray tune. |
| refit\_with\_val | bool | False | Refit of best model should preserve val\_size. |
| verbose | bool | False | Track progress. |
| alias | NoneType | None | Custom name of the model. |
| backend | str | ray | Backend to use for searching the hyperparameter space, can be either ‘ray’ or ‘optuna’. |
| callbacks | NoneType | None | List of functions to call during the optimization process. ray reference: [https://docs.ray.io/en/latest/tune/tutorials/tune-metrics.html](https://docs.ray.io/en/latest/tune/tutorials/tune-metrics.html) optuna reference: [https://optuna.readthedocs.io/en/stable/tutorial/20\_recipes/007\_optuna\_callback.html](https://optuna.readthedocs.io/en/stable/tutorial/20_recipes/007_optuna_callback.html) |
***
### BaseAuto.fit
> ```text theme={null}
> BaseAuto.fit (dataset, val_size=0, test_size=0, random_seed=None,
> distributed_config=None)
> ```
\*BaseAuto.fit
Perform the hyperparameter optimization as specified by the BaseAuto
configuration dictionary `config`.
The optimization is performed on the
[`TimeSeriesDataset`](https://nixtlaverse.nixtla.io/neuralforecast/tsdataset.html#timeseriesdataset)
using temporal cross validation with the validation set that
sequentially precedes the test set.
**Parameters:** `dataset`: NeuralForecast’s
[`TimeSeriesDataset`](https://nixtlaverse.nixtla.io/neuralforecast/tsdataset.html#timeseriesdataset)
see details
[here](https://nixtla.github.io/neuralforecast/tsdataset.html)
`val_size`: int, size of temporal validation set (needs to be bigger
than 0). `test_size`: int, size of temporal test set (default
0\). `random_seed`: int=None, random\_seed for hyperparameter
exploration algorithms, not yet implemented. **Returns:**
`self`: fitted instance of `BaseAuto` with best hyperparameters and
results .\*
***
### BaseAuto.predict
> ```text theme={null}
> BaseAuto.predict (dataset, step_size=1, **data_kwargs)
> ```
\*BaseAuto.predict
Predictions of the best performing model on validation.
**Parameters:** `dataset`: NeuralForecast’s
[`TimeSeriesDataset`](https://nixtlaverse.nixtla.io/neuralforecast/tsdataset.html#timeseriesdataset)
see details
[here](https://nixtla.github.io/neuralforecast/tsdataset.html)
`step_size`: int, steps between sequential predictions, (default 1).
`**data_kwarg`: additional parameters for the dataset module.
`random_seed`: int=None, random\_seed for hyperparameter exploration
algorithms (not implemented). **Returns:** `y_hat`: numpy
predictions of the
[`NeuralForecast`](https://nixtlaverse.nixtla.io/neuralforecast/core.html#neuralforecast)
model. \*
```python theme={null}
class RayLogLossesCallback(tune.Callback):
def on_trial_complete(self, iteration, trials, trial, **info):
result = trial.last_result
print(40 * '-' + 'Trial finished' + 40 * '-')
print(f'Train loss: {result["train_loss"]:.2f}. Valid loss: {result["loss"]:.2f}')
print(80 * '-')
```
```python theme={null}
config = {
"hidden_size": tune.choice([512]),
"num_layers": tune.choice([3, 4]),
"input_size": 12,
"max_steps": 10,
"val_check_steps": 5
}
auto = BaseAuto(h=12, loss=MAE(), valid_loss=MSE(), cls_model=MLP, config=config, num_samples=2, cpus=1, gpus=0, callbacks=[RayLogLossesCallback()])
auto.fit(dataset=dataset)
y_hat = auto.predict(dataset=dataset)
assert mae(Y_test_df['y'].values, y_hat[:, 0]) < 200
```
```python theme={null}
def config_f(trial):
return {
"hidden_size": trial.suggest_categorical('hidden_size', [512]),
"num_layers": trial.suggest_categorical('num_layers', [3, 4]),
"input_size": 12,
"max_steps": 10,
"val_check_steps": 5
}
class OptunaLogLossesCallback:
def __call__(self, study, trial):
metrics = trial.user_attrs['METRICS']
print(40 * '-' + 'Trial finished' + 40 * '-')
print(f'Train loss: {metrics["train_loss"]:.2f}. Valid loss: {metrics["loss"]:.2f}')
print(80 * '-')
```
```python theme={null}
auto2 = BaseAuto(h=12, loss=MAE(), valid_loss=MSE(), cls_model=MLP, config=config_f, search_alg=optuna.samplers.RandomSampler(), num_samples=2, backend='optuna', callbacks=[OptunaLogLossesCallback()])
auto2.fit(dataset=dataset)
assert isinstance(auto2.results, optuna.Study)
y_hat2 = auto2.predict(dataset=dataset)
assert mae(Y_test_df['y'].values, y_hat2[:, 0]) < 200
```
### References
* [James Bergstra, Remi Bardenet, Yoshua Bengio, and Balazs Kegl
(2011). “Algorithms for Hyper-Parameter Optimization”. In: Advances
in Neural Information Processing Systems. url:
https://proceedings.neurips.cc/paper/2011/file/86e8f7ab32cfd12577bc2619bc635690-Paper.pdf](https://proceedings.neurips.cc/paper/2011/file/86e8f7ab32cfd12577bc2619bc635690-Paper.pdf)
* [Kirthevasan Kandasamy, Karun Raju Vysyaraju, Willie Neiswanger,
Biswajit Paria, Christopher R. Collins, Jeff Schneider, Barnabas
Poczos, Eric P. Xing (2019). “Tuning Hyperparameters without Grad
Students: Scalable and Robust Bayesian Optimisation with Dragonfly”.
Journal of Machine Learning Research. url:
https://arxiv.org/abs/1903.06694](https://arxiv.org/abs/1903.06694)
* [Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh,
Ameet Talwalkar (2016). “Hyperband: A Novel Bandit-Based Approach to
Hyperparameter Optimization”. Journal of Machine Learning Research.
url:
https://arxiv.org/abs/1603.06560](https://arxiv.org/abs/1603.06560)
