> ## Documentation Index > Fetch the complete documentation index at: https://nixtla-old-docs.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. > Temporal normalization has proven to be essential in neural forecasting tasks, as it enables network's non-linearities to express themselves. Forecasting scaling methods take particular interest in the temporal dimension where most of the variance dwells, contrary to other deep learning techniques like `BatchNorm` that normalizes across batch and temporal dimensions, and `LayerNorm` that normalizes across the feature dimension. Currently we support the following techniques: `std`, `median`, `norm`, `norm1`, `invariant`, `revin`. # TemporalNorm ## References * [Kin G. Olivares, David Luo, Cristian Challu, Stefania La Vattiata, Max Mergenthaler, Artur Dubrawski (2023). “HINT: Hierarchical Mixture Networks For Coherent Probabilistic Forecasting”. Neural Information Processing Systems, submitted. Working Paper version available at arxiv.](https://arxiv.org/abs/2305.07089) * [Taesung Kim and Jinhee Kim and Yunwon Tae and Cheonbok Park and Jang-Ho Choi and Jaegul Choo. “Reversible Instance Normalization for Accurate Time-Series Forecasting against Distribution Shift”. ICLR 2022.](https://openreview.net/pdf?id=cGDAkQo1C0p) * [David Salinas, Valentin Flunkert, Jan Gasthaus, Tim Januschowski (2020). “DeepAR: Probabilistic forecasting with autoregressive recurrent networks”. International Journal of Forecasting.](https://www.sciencedirect.com/science/article/pii/S0169207019301888)
Figure 1. Illustration of temporal normalization (left), layer normalization (center) and batch normalization (right). The entries in green show the components used to compute the normalizing statistics.
# 1. Auxiliary Functions *** ### masked\_median > ```text theme={null} > masked_median (x, mask, dim=-1, keepdim=True) > ``` \*Masked Median Compute the median of tensor `x` along dim, ignoring values where `mask` is False. `x` and `mask` need to be broadcastable. **Parameters:**
`x`: torch.Tensor to compute median of along `dim` dimension.
`mask`: torch Tensor bool with same shape as `x`, where `x` is valid and False where `x` should be masked. Mask should not be all False in any column of dimension dim to avoid NaNs from zero division.
`dim` (int, optional): Dimension to take median of. Defaults to -1.
`keepdim` (bool, optional): Keep dimension of `x` or not. Defaults to True.
**Returns:**
`x_median`: torch.Tensor with normalized values.\* *** ### masked\_mean > ```text theme={null} > masked_mean (x, mask, dim=-1, keepdim=True) > ``` \*Masked Mean Compute the mean of tensor `x` along dimension, ignoring values where `mask` is False. `x` and `mask` need to be broadcastable. **Parameters:**
`x`: torch.Tensor to compute mean of along `dim` dimension.
`mask`: torch Tensor bool with same shape as `x`, where `x` is valid and False where `x` should be masked. Mask should not be all False in any column of dimension dim to avoid NaNs from zero division.
`dim` (int, optional): Dimension to take mean of. Defaults to -1.
`keepdim` (bool, optional): Keep dimension of `x` or not. Defaults to True.
**Returns:**
`x_mean`: torch.Tensor with normalized values.\* # 2. Scalers *** ### minmax\_statistics > ```text theme={null} > minmax_statistics (x, mask, eps=1e-06, dim=-1) > ``` \*MinMax Scaler Standardizes temporal features by ensuring its range dweels between \[0,1] range. This transformation is often used as an alternative to the standard scaler. The scaled features are obtained as: $$ \mathbf{z} = (\mathbf{x}_{[B,T,C]}-\mathrm{min}({\mathbf{x}})_{[B,1,C]})/ (\mathrm{max}({\mathbf{x}})_{[B,1,C]}- \mathrm{min}({\mathbf{x}})_{[B,1,C]}) $$ **Parameters:**
`x`: torch.Tensor input tensor.
`mask`: torch Tensor bool, same dimension as `x`, indicates where `x` is valid and False where `x` should be masked. Mask should not be all False in any column of dimension dim to avoid NaNs from zero division.
`eps` (float, optional): Small value to avoid division by zero. Defaults to 1e-6.
`dim` (int, optional): Dimension over to compute min and max. Defaults to -1.
**Returns:**
`z`: torch.Tensor same shape as `x`, except scaled.\* *** ### minmax1\_statistics > ```text theme={null} > minmax1_statistics (x, mask, eps=1e-06, dim=-1) > ``` \*MinMax1 Scaler Standardizes temporal features by ensuring its range dweels between \[-1,1] range. This transformation is often used as an alternative to the standard scaler or classic Min Max Scaler. The scaled features are obtained as: $\mathbf{z} = 2 (\mathbf{x}_{[B,T,C]}-\mathrm{min}({\mathbf{x}})_{[B,1,C]})/ (\mathrm{max}({\mathbf{x}})_{[B,1,C]}- \mathrm{min}({\mathbf{x}})_{[B,1,C]})-1$ **Parameters:**
`x`: torch.Tensor input tensor.
`mask`: torch Tensor bool, same dimension as `x`, indicates where `x` is valid and False where `x` should be masked. Mask should not be all False in any column of dimension dim to avoid NaNs from zero division.
`eps` (float, optional): Small value to avoid division by zero. Defaults to 1e-6.
`dim` (int, optional): Dimension over to compute min and max. Defaults to -1.
**Returns:**
`z`: torch.Tensor same shape as `x`, except scaled.\* *** ### std\_statistics > ```text theme={null} > std_statistics (x, mask, dim=-1, eps=1e-06) > ``` \*Standard Scaler Standardizes features by removing the mean and scaling to unit variance along the `dim` dimension. For example, for `base_windows` models, the scaled features are obtained as (with dim=1): $\mathbf{z} = (\mathbf{x}_{[B,T,C]}-\bar{\mathbf{x}}_{[B,1,C]})/\hat{\sigma}_{[B,1,C]}$ **Parameters:**
`x`: torch.Tensor.
`mask`: torch Tensor bool, same dimension as `x`, indicates where `x` is valid and False where `x` should be masked. Mask should not be all False in any column of dimension dim to avoid NaNs from zero division.
`eps` (float, optional): Small value to avoid division by zero. Defaults to 1e-6.
`dim` (int, optional): Dimension over to compute mean and std. Defaults to -1.
**Returns:**
`z`: torch.Tensor same shape as `x`, except scaled.\* *** ### robust\_statistics > ```text theme={null} > robust_statistics (x, mask, dim=-1, eps=1e-06) > ``` \*Robust Median Scaler Standardizes features by removing the median and scaling with the mean absolute deviation (mad) a robust estimator of variance. This scaler is particularly useful with noisy data where outliers can heavily influence the sample mean / variance in a negative way. In these scenarios the median and amd give better results. For example, for `base_windows` models, the scaled features are obtained as (with dim=1): $\mathbf{z} = (\mathbf{x}_{[B,T,C]}-\textrm{median}(\mathbf{x})_{[B,1,C]})/\textrm{mad}(\mathbf{x})_{[B,1,C]}$ $\textrm{mad}(\mathbf{x}) = \frac{1}{N} \sum_{}|\mathbf{x} - \mathrm{median}(x)|$ **Parameters:**
`x`: torch.Tensor input tensor.
`mask`: torch Tensor bool, same dimension as `x`, indicates where `x` is valid and False where `x` should be masked. Mask should not be all False in any column of dimension dim to avoid NaNs from zero division.
`eps` (float, optional): Small value to avoid division by zero. Defaults to 1e-6.
`dim` (int, optional): Dimension over to compute median and mad. Defaults to -1.
**Returns:**
`z`: torch.Tensor same shape as `x`, except scaled.\* *** ### invariant\_statistics > ```text theme={null} > invariant_statistics (x, mask, dim=-1, eps=1e-06) > ``` \*Invariant Median Scaler Standardizes features by removing the median and scaling with the mean absolute deviation (mad) a robust estimator of variance. Aditionally it complements the transformation with the arcsinh transformation. For example, for `base_windows` models, the scaled features are obtained as (with dim=1): $\mathbf{z} = (\mathbf{x}_{[B,T,C]}-\textrm{median}(\mathbf{x})_{[B,1,C]})/\textrm{mad}(\mathbf{x})_{[B,1,C]}$ $\mathbf{z} = \textrm{arcsinh}(\mathbf{z})$ **Parameters:**
`x`: torch.Tensor input tensor.
`mask`: torch Tensor bool, same dimension as `x`, indicates where `x` is valid and False where `x` should be masked. Mask should not be all False in any column of dimension dim to avoid NaNs from zero division.
`eps` (float, optional): Small value to avoid division by zero. Defaults to 1e-6.
`dim` (int, optional): Dimension over to compute median and mad. Defaults to -1.
**Returns:**
`z`: torch.Tensor same shape as `x`, except scaled.\* *** ### identity\_statistics > ```text theme={null} > identity_statistics (x, mask, dim=-1, eps=1e-06) > ``` \*Identity Scaler A placeholder identity scaler, that is argument insensitive. **Parameters:**
`x`: torch.Tensor input tensor.
`mask`: torch Tensor bool, same dimension as `x`, indicates where `x` is valid and False where `x` should be masked. Mask should not be all False in any column of dimension dim to avoid NaNs from zero division.
`eps` (float, optional): Small value to avoid division by zero. Defaults to 1e-6.
`dim` (int, optional): Dimension over to compute median and mad. Defaults to -1.
**Returns:**
`x`: original torch.Tensor `x`.\* # 3. TemporalNorm Module *** ### TemporalNorm > ```text theme={null} > TemporalNorm (scaler_type='robust', dim=-1, eps=1e-06, num_features=None) > ``` \*Temporal Normalization Standardization of the features is a common requirement for many machine learning estimators, and it is commonly achieved by removing the level and scaling its variance. The `TemporalNorm` module applies temporal normalization over the batch of inputs as defined by the type of scaler. $\mathbf{z}_{[B,T,C]} = \textrm{Scaler}(\mathbf{x}_{[B,T,C]})$ If `scaler_type` is `revin` learnable normalization parameters are added on top of the usual normalization technique, the parameters are learned through scale decouple global skip connections. The technique is available for point and probabilistic outputs. $\mathbf{\hat{z}}_{[B,T,C]} = \boldsymbol{\hat{\gamma}}_{[1,1,C]} \mathbf{z}_{[B,T,C]} +\boldsymbol{\hat{\beta}}_{[1,1,C]}$ **Parameters:**
`scaler_type`: str, defines the type of scaler used by TemporalNorm. Available \[`identity`, `standard`, `robust`, `minmax`, `minmax1`, `invariant`, `revin`].
`dim` (int, optional): Dimension over to compute scale and shift. Defaults to -1.
`eps` (float, optional): Small value to avoid division by zero. Defaults to 1e-6.
`num_features`: int=None, for RevIN-like learnable affine parameters initialization.
**References**
- [Kin G. Olivares, David Luo, Cristian Challu, Stefania La Vattiata, Max Mergenthaler, Artur Dubrawski (2023). “HINT: Hierarchical Mixture Networks For Coherent Probabilistic Forecasting”. Neural Information Processing Systems, submitted. Working Paper version available at arxiv.](https://arxiv.org/abs/2305.07089)
\* *** ### TemporalNorm.transform > ```text theme={null} > TemporalNorm.transform (x, mask) > ``` \*Center and scale the data. **Parameters:**
`x`: torch.Tensor shape \[batch, time, channels].
`mask`: torch Tensor bool, shape \[batch, time] where `x` is valid and False where `x` should be masked. Mask should not be all False in any column of dimension dim to avoid NaNs from zero division.
**Returns:**
`z`: torch.Tensor same shape as `x`, except scaled.\* *** ### TemporalNorm.inverse\_transform > ```text theme={null} > TemporalNorm.inverse_transform (z, x_shift=None, x_scale=None) > ``` \*Scale back the data to the original representation. **Parameters:**
`z`: torch.Tensor shape \[batch, time, channels], scaled.
**Returns:**
`x`: torch.Tensor original data.\* # Example ```python theme={null} import numpy as np ``` ```python theme={null} # Declare synthetic batch to normalize x1 = 10**0 * np.arange(36)[:, None] x2 = 10**1 * np.arange(36)[:, None] np_x = np.concatenate([x1, x2], axis=1) np_x = np.repeat(np_x[None, :,:], repeats=2, axis=0) np_x[0,:,:] = np_x[0,:,:] + 100 np_mask = np.ones(np_x.shape) np_mask[:, -12:, :] = 0 print(f'x.shape [batch, time, features]={np_x.shape}') print(f'mask.shape [batch, time, features]={np_mask.shape}') ``` ```python theme={null} # Validate scalers x = 1.0*torch.tensor(np_x) mask = torch.tensor(np_mask) scaler = TemporalNorm(scaler_type='standard', dim=1) x_scaled = scaler.transform(x=x, mask=mask) x_recovered = scaler.inverse_transform(x_scaled) plt.plot(x[0,:,0], label='x1', color='#78ACA8') plt.plot(x[0,:,1], label='x2', color='#E3A39A') plt.title('Before TemporalNorm') plt.xlabel('Time') plt.legend() plt.show() plt.plot(x_scaled[0,:,0], label='x1', color='#78ACA8') plt.plot(x_scaled[0,:,1]+0.1, label='x2+0.1', color='#E3A39A') plt.title(f'TemporalNorm \'{scaler.scaler_type}\' ') plt.xlabel('Time') plt.legend() plt.show() plt.plot(x_recovered[0,:,0], label='x1', color='#78ACA8') plt.plot(x_recovered[0,:,1], label='x2', color='#E3A39A') plt.title('Recovered') plt.xlabel('Time') plt.legend() plt.show() ```