> ## Documentation Index > Fetch the complete documentation index at: https://nixtla-old-docs.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # TFT In summary Temporal Fusion Transformer (TFT) combines gating layers, an LSTM recurrent encoder, with multi-head attention layers for a multi-step forecasting strategy decoder.
TFT’s inputs are static exogenous $\mathbf{x}^{(s)}$, historic exogenous $\mathbf{x}^{(h)}_{[:t]}$, exogenous available at the time of the prediction $\mathbf{x}^{(f)}_{[:t+H]}$ and autorregresive features $\mathbf{y}_{[:t]}$, each of these inputs is further decomposed into categorical and continuous. The network uses a multi-quantile regression to model the following conditional probability:$\mathbb{P}(\mathbf{y}_{[t+1:t+H]}|\;\mathbf{y}_{[:t]},\; \mathbf{x}^{(h)}_{[:t]},\; \mathbf{x}^{(f)}_{[:t+H]},\; \mathbf{x}^{(s)})$ **References**
- [Jan Golda, Krzysztof Kudrynski. “NVIDIA, Deep Learning Forecasting Examples”](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Forecasting/TFT)
- [Bryan Lim, Sercan O. Arik, Nicolas Loeff, Tomas Pfister, “Temporal Fusion Transformers for interpretable multi-horizon time series forecasting”](https://www.sciencedirect.com/science/article/pii/S0169207021000637)
Figure 1. Temporal Fusion Transformer Architecture.
## 1. Auxiliary Functions ### 1.1 Gating Mechanisms The Gated Residual Network (GRN) provides adaptive depth and network complexity capable of accommodating different size datasets. As residual connections allow for the network to skip the non-linear transformation of input $\mathbf{a}$ and context $\mathbf{c}$. The Gated Linear Unit (GLU) provides the flexibility of supressing unnecesary parts of the GRN. Consider GRN’s output $\gamma$ then GLU transformation is defined by: $\mathrm{GLU}(\gamma) = \sigma(\mathbf{W}_{4}\gamma +b_{4}) \odot (\mathbf{W}_{5}\gamma +b_{5})$
Figure 2. Gated Residual Network.
### 1.2 Variable Selection Networks TFT includes automated variable selection capabilities, through its variable selection network (VSN) components. The VSN takes the original input $\{\mathbf{x}^{(s)}, \mathbf{x}^{(h)}_{[:t]}, \mathbf{x}^{(f)}_{[:t]}\}$ and transforms it through embeddings or linear transformations into a high dimensional space $\{\mathbf{E}^{(s)}, \mathbf{E}^{(h)}_{[:t]}, \mathbf{E}^{(f)}_{[:t+H]}\}$. For the observed historic data, the embedding matrix $\mathbf{E}^{(h)}_{t}$ at time $t$ is a concatenation of $j$ variable $e^{(h)}_{t,j}$ embeddings: The variable selection weights are given by: $s^{(h)}_{t}=\mathrm{SoftMax}(\mathrm{GRN}(\mathbf{E}^{(h)}_{t},\mathbf{E}^{(s)}))$ The VSN processed features are then: $\tilde{\mathbf{E}}^{(h)}_{t}= \sum_{j} s^{(h)}_{j} \tilde{e}^{(h)}_{t,j}$
Figure 3. Variable Selection Network.
### 1.3. Multi-Head Attention To avoid information bottlenecks from the classic Seq2Seq architecture, TFT incorporates a decoder-encoder attention mechanism inherited transformer architectures ([Li et. al 2019](https://arxiv.org/abs/1907.00235), [Vaswani et. al 2017](https://arxiv.org/abs/1706.03762)). It transform the the outputs of the LSTM encoded temporal features, and helps the decoder better capture long-term relationships. The original multihead attention for each component $H_{m}$ and its query, key, and value representations are denoted by $Q_{m}, K_{m}, V_{m}$, its transformation is given by: TFT modifies the original multihead attention to improve its interpretability. To do it it uses shared values $\tilde{V}$ across heads and employs additive aggregation, $\mathrm{InterpretableMultiHead}(Q,K,V) = \tilde{H} W_{M}$. The mechanism has a great resemblence to a single attention layer, but it allows for $M$ multiple attention weights, and can be therefore be interpreted as the average ensemble of $M$ single attention layers. ## 2. TFT Architecture The first TFT’s step is embed the original input $\{\mathbf{x}^{(s)}, \mathbf{x}^{(h)}, \mathbf{x}^{(f)}\}$ into a high dimensional space $\{\mathbf{E}^{(s)}, \mathbf{E}^{(h)}, \mathbf{E}^{(f)}\}$, after which each embedding is gated by a variable selection network (VSN). The static embedding $\mathbf{E}^{(s)}$ is used as context for variable selection and as initial condition to the LSTM. Finally the encoded variables are fed into the multi-head attention decoder. ### 2.1 Static Covariate Encoder The static embedding $\mathbf{E}^{(s)}$ is transformed by the StaticCovariateEncoder into contexts $c_{s}, c_{e}, c_{h}, c_{c}$. Where $c_{s}$ are temporal variable selection contexts, $c_{e}$ are TemporalFusionDecoder enriching contexts, and $c_{h}, c_{c}$ are LSTM’s hidden/contexts for the TemporalCovariateEncoder. ### 2.2 Temporal Covariate Encoder TemporalCovariateEncoder encodes the embeddings $\mathbf{E}^{(h)}, \mathbf{E}^{(f)}$ and contexts $(c_{h}, c_{c})$ with an LSTM. An analogous process is repeated for the future data, with the main difference that $\mathbf{E}^{(f)}$ contains the future available information. ### 2.3 Temporal Fusion Decoder The TemporalFusionDecoder enriches the LSTM’s outputs with $c_{e}$ and then uses an attention layer, and multi-step adapter. *** source ### TFT > ```text theme={null} > TFT (h, input_size, tgt_size:int=1, stat_exog_list=None, > hist_exog_list=None, futr_exog_list=None, hidden_size:int=128, > n_head:int=4, attn_dropout:float=0.0, grn_activation:str='ELU', > n_rnn_layers:int=1, rnn_type:str='lstm', > one_rnn_initial_state:bool=False, dropout:float=0.1, loss=MAE(), > valid_loss=None, max_steps:int=1000, learning_rate:float=0.001, > num_lr_decays:int=-1, early_stop_patience_steps:int=-1, > val_check_steps:int=100, batch_size:int=32, > valid_batch_size:Optional[int]=None, windows_batch_size:int=1024, > inference_windows_batch_size:int=1024, start_padding_enabled=False, > step_size:int=1, scaler_type:str='robust', random_seed:int=1, > drop_last_loader=False, alias:Optional[str]=None, optimizer=None, > optimizer_kwargs=None, lr_scheduler=None, lr_scheduler_kwargs=None, > dataloader_kwargs=None, **trainer_kwargs) > ``` \*TFT The Temporal Fusion Transformer architecture (TFT) is an Sequence-to-Sequence model that combines static, historic and future available data to predict an univariate target. The method combines gating layers, an LSTM recurrent encoder, with and interpretable multi-head attention layer and a multi-step forecasting strategy decoder. **Parameters:**
`h`: int, Forecast horizon.
`input_size`: int, autorregresive inputs size, y=\[1,2,3,4] input\_size=2 -> y\_\[t-2:t]=\[1,2].
`tgt_size`: int=1, target size.
`stat_exog_list`: str list, static continuous columns.
`hist_exog_list`: str list, historic continuous columns.
`futr_exog_list`: str list, future continuous columns.
`hidden_size`: int, units of embeddings and encoders.
`n_head`: int=4, number of attention heads in temporal fusion decoder.
`attn_dropout`: float (0, 1), dropout of fusion decoder’s attention layer.
`grn_activation`: str, activation for the GRN module from \[‘ReLU’, ‘Softplus’, ‘Tanh’, ‘SELU’, ‘LeakyReLU’, ‘Sigmoid’, ‘ELU’, ‘GLU’].
`n_rnn_layers`: int=1, number of RNN layers.
`rnn_type`: str=“lstm”, recurrent neural network (RNN) layer type from \[“lstm”,“gru”].
`one_rnn_initial_state`:str=False, Initialize all rnn layers with the same initial states computed from static covariates.
`dropout`: float (0, 1), dropout of inputs VSNs.
`loss`: PyTorch module, instantiated train loss class from [losses collection](https://nixtla.github.io/neuralforecast/losses.pytorch.html).
`valid_loss`: PyTorch module=`loss`, instantiated valid loss class from [losses collection](https://nixtla.github.io/neuralforecast/losses.pytorch.html).
`max_steps`: int=1000, maximum number of training steps.
`learning_rate`: float=1e-3, Learning rate between (0, 1).
`num_lr_decays`: int=-1, Number of learning rate decays, evenly distributed across max\_steps.
`early_stop_patience_steps`: int=-1, Number of validation iterations before early stopping.
`val_check_steps`: int=100, Number of training steps between every validation loss check.
`batch_size`: int, number of different series in each batch.
`valid_batch_size`: int=None, number of different series in each validation and test batch.
`windows_batch_size`: int=None, windows sampled from rolled data, default uses all.
`inference_windows_batch_size`: int=-1, number of windows to sample in each inference batch, -1 uses all.
`start_padding_enabled`: bool=False, if True, the model will pad the time series with zeros at the beginning, by input size.
`step_size`: int=1, step size between each window of temporal data.
`scaler_type`: str=‘robust’, type of scaler for temporal inputs normalization see [temporal scalers](https://nixtla.github.io/neuralforecast/common.scalers.html).
`random_seed`: int, random seed initialization for replicability.
`drop_last_loader`: bool=False, if True `TimeSeriesDataLoader` drops last non-full batch.
`alias`: str, optional, Custom name of the model.
`optimizer`: Subclass of ‘torch.optim.Optimizer’, optional, user specified optimizer instead of the default choice (Adam).
`optimizer_kwargs`: dict, optional, list of parameters used by the user specified `optimizer`.
`lr_scheduler`: Subclass of ‘torch.optim.lr\_scheduler.LRScheduler’, optional, user specified lr\_scheduler instead of the default choice (StepLR).
`lr_scheduler_kwargs`: dict, optional, list of parameters used by the user specified `lr_scheduler`.
`dataloader_kwargs`: dict, optional, list of parameters passed into the PyTorch Lightning dataloader by the `TimeSeriesDataLoader`.
`**trainer_kwargs`: int, keyword trainer arguments inherited from [PyTorch Lighning’s trainer](https://pytorch-lightning.readthedocs.io/en/stable/api/pytorch_lightning.trainer.trainer.Trainer.html?highlight=trainer).
**References:**
- [Bryan Lim, Sercan O. Arik, Nicolas Loeff, Tomas Pfister, “Temporal Fusion Transformers for interpretable multi-horizon time series forecasting”](https://www.sciencedirect.com/science/article/pii/S0169207021000637)\* ## 3. TFT methods *** ### TFT.fit > ```text theme={null} > TFT.fit (dataset, val_size=0, test_size=0, random_seed=None, > distributed_config=None) > ``` \*Fit. The `fit` method, optimizes the neural network’s weights using the initialization parameters (`learning_rate`, `windows_batch_size`, …) and the `loss` function as defined during the initialization. Within `fit` we use a PyTorch Lightning `Trainer` that inherits the initialization’s `self.trainer_kwargs`, to customize its inputs, see [PL’s trainer arguments](https://pytorch-lightning.readthedocs.io/en/stable/api/pytorch_lightning.trainer.trainer.Trainer.html?highlight=trainer). The method is designed to be compatible with SKLearn-like classes and in particular to be compatible with the StatsForecast library. By default the `model` is not saving training checkpoints to protect disk memory, to get them change `enable_checkpointing=True` in `__init__`. **Parameters:**
`dataset`: NeuralForecast’s [`TimeSeriesDataset`](https://nixtlaverse.nixtla.io/neuralforecast/tsdataset.html#timeseriesdataset), see [documentation](https://nixtla.github.io/neuralforecast/tsdataset.html).
`val_size`: int, validation size for temporal cross-validation.
`random_seed`: int=None, random\_seed for pytorch initializer and numpy generators, overwrites model.\_\_init\_\_’s.
`test_size`: int, test size for temporal cross-validation.
\* *** ### TFT.predict > ```text theme={null} > TFT.predict (dataset, test_size=None, step_size=1, random_seed=None, > quantiles=None, **data_module_kwargs) > ``` \*Predict. Neural network prediction with PL’s `Trainer` execution of `predict_step`. **Parameters:**
`dataset`: NeuralForecast’s [`TimeSeriesDataset`](https://nixtlaverse.nixtla.io/neuralforecast/tsdataset.html#timeseriesdataset), see [documentation](https://nixtla.github.io/neuralforecast/tsdataset.html).
`test_size`: int=None, test size for temporal cross-validation.
`step_size`: int=1, Step size between each window.
`random_seed`: int=None, random\_seed for pytorch initializer and numpy generators, overwrites model.\_\_init\_\_’s.
`quantiles`: list of floats, optional (default=None), target quantiles to predict.
`**data_module_kwargs`: PL’s TimeSeriesDataModule args, see [documentation](https://pytorch-lightning.readthedocs.io/en/1.6.1/extensions/datamodules.html#using-a-datamodule).\* *** source ### TFT.feature\_importances, > ```text theme={null} > TFT.feature_importances, () > ``` \*Compute the feature importances for historical, future, and static features. Returns: dict: A dictionary containing the feature importances for each feature type. The keys are ‘hist\_vsn’, ‘future\_vsn’, and ‘static\_vsn’, and the values are pandas DataFrames with the corresponding feature importances.\* *** source ### TFT.attention\_weights > ```text theme={null} > TFT.attention_weights () > ``` \*Batch average attention weights Returns: np.ndarray: A 1D array containing the attention weights for each time step.\* *** source ### TFT.attention\_weights > ```text theme={null} > TFT.attention_weights () > ``` \*Batch average attention weights Returns: np.ndarray: A 1D array containing the attention weights for each time step.\* *** source ### TFT.feature\_importance\_correlations > ```text theme={null} > TFT.feature_importance_correlations () > ``` \*Compute the correlation between the past and future feature importances and the mean attention weights. Returns: pd.DataFrame: A DataFrame containing the correlation coefficients between the past feature importances and the mean attention weights.\* ## Usage Example ```python theme={null} import matplotlib.pyplot as plt import numpy as np import pandas as pd from neuralforecast import NeuralForecast # from neuralforecast.models import TFT from neuralforecast.losses.pytorch import DistributionLoss from neuralforecast.utils import AirPassengersPanel, AirPassengersStatic AirPassengersPanel["month"] = AirPassengersPanel.ds.dt.month Y_train_df = AirPassengersPanel[ AirPassengersPanel.ds < AirPassengersPanel["ds"].values[-12] ] # 132 train Y_test_df = AirPassengersPanel[ AirPassengersPanel.ds >= AirPassengersPanel["ds"].values[-12] ].reset_index(drop=True) # 12 test nf = NeuralForecast( models=[ TFT( h=12, input_size=48, hidden_size=20, grn_activation="ELU", rnn_type="lstm", n_rnn_layers=1, one_rnn_initial_state=False, loss=DistributionLoss(distribution="StudentT", level=[80, 90]), learning_rate=0.005, stat_exog_list=["airline1"], futr_exog_list=["y_[lag12]", "month"], hist_exog_list=["trend"], max_steps=300, val_check_steps=10, early_stop_patience_steps=10, scaler_type="robust", windows_batch_size=None, enable_progress_bar=True, ), ], freq="ME", ) nf.fit(df=Y_train_df, static_df=AirPassengersStatic, val_size=12) Y_hat_df = nf.predict(futr_df=Y_test_df) # Plot quantile predictions Y_hat_df = Y_hat_df.reset_index(drop=False).drop(columns=["unique_id", "ds"]) plot_df = pd.concat([Y_test_df, Y_hat_df], axis=1) plot_df = pd.concat([Y_train_df, plot_df]) plot_df = plot_df[plot_df.unique_id == "Airline1"].drop("unique_id", axis=1) plt.plot(plot_df["ds"], plot_df["y"], c="black", label="True") plt.plot(plot_df["ds"], plot_df["TFT"], c="purple", label="mean") plt.plot(plot_df["ds"], plot_df["TFT-median"], c="blue", label="median") plt.fill_between( x=plot_df["ds"][-12:], y1=plot_df["TFT-lo-90"][-12:].values, y2=plot_df["TFT-hi-90"][-12:].values, alpha=0.4, label="level 90", ) plt.legend() plt.grid() plt.plot() ``` # Interpretability ## 1. Attention Weights ```python theme={null} attention = nf.models[0].attention_weights() ``` ```python theme={null} def plot_attention( self, plot: str = "time", output: str = "plot", width: int = 800, height: int = 400 ): """ Plot the attention weights. Args: plot (str, optional): The type of plot to generate. Can be one of the following: - 'time': Display the mean attention weights over time. - 'all': Display the attention weights for each horizon. - 'heatmap': Display the attention weights as a heatmap. - An integer in the range [1, model.h) to display the attention weights for a specific horizon. output (str, optional): The type of output to generate. Can be one of the following: - 'plot': Display the plot directly. - 'figure': Return the plot as a figure object. width (int, optional): Width of the plot in pixels. Default is 800. height (int, optional): Height of the plot in pixels. Default is 400. Returns: matplotlib.figure.Figure: If `output` is 'figure', the function returns the plot as a figure object. """ attention = ( self.mean_on_batch(self.interpretability_params["attn_wts"]) .mean(dim=0) .cpu() .numpy() ) fig, ax = plt.subplots(figsize=(width / 100, height / 100)) if plot == "time": attention = attention[self.input_size :, :].mean(axis=0) ax.plot(np.arange(-self.input_size, self.h), attention) ax.axvline( x=0, color="black", linewidth=3, linestyle="--", label="prediction start" ) ax.set_title("Mean Attention") ax.set_xlabel("time") ax.set_ylabel("Attention") ax.legend() elif plot == "all": for i in range(self.input_size, attention.shape[0]): ax.plot( np.arange(-self.input_size, self.h), attention[i, :], label=f"horizon {i-self.input_size+1}", ) ax.axvline( x=0, color="black", linewidth=3, linestyle="--", label="prediction start" ) ax.set_title("Attention per horizon") ax.set_xlabel("time") ax.set_ylabel("Attention") ax.legend() elif plot == "heatmap": cax = ax.imshow( attention, aspect="auto", cmap="viridis", extent=[-self.input_size, self.h, -self.input_size, self.h], ) fig.colorbar(cax) ax.set_title("Attention Heatmap") ax.set_xlabel("Attention (current time step)") ax.set_ylabel("Attention (previous time step)") elif isinstance(plot, int) and (plot in np.arange(1, self.h + 1)): i = self.input_size + plot - 1 ax.plot( np.arange(-self.input_size, self.h), attention[i, :], label=f"horizon {plot}", ) ax.axvline( x=0, color="black", linewidth=3, linestyle="--", label="prediction start" ) ax.set_title(f"Attention weight for horizon {plot}") ax.set_xlabel("time") ax.set_ylabel("Attention") ax.legend() else: raise ValueError( 'plot has to be in ["time","all","heatmap"] or integer in range(1,model.h)' ) plt.tight_layout() if output == "plot": plt.show() elif output == "figure": return fig else: raise ValueError(f"Invalid output: {output}. Expected 'plot' or 'figure'.") ``` #### 1.1 Mean attention ```python theme={null} plot_attention(nf.models[0], plot="time") ``` #### 1.2 Attention of all future time steps ```python theme={null} plot_attention(nf.models[0], plot="all") ``` #### 1.3 Attention of a specific future time step ```python theme={null} plot_attention(nf.models[0], plot=8) ``` ## 2. Feature Importance ### 2.1 Global feature importance ```python theme={null} feature_importances = nf.models[0].feature_importances() feature_importances.keys() ``` #### Static variable importances ```python theme={null} feature_importances["Static covariates"].sort_values(by="importance").plot(kind="barh") ``` #### Past variable importances ```python theme={null} feature_importances["Past variable importance over time"].mean().sort_values().plot( kind="barh" ) ``` #### Future variable importances ```python theme={null} feature_importances["Future variable importance over time"].mean().sort_values().plot( kind="barh" ) ``` ### 2.2 Variable importances over time #### Future variable importance over time Importance of each future covariate at each future time step ```python theme={null} df = feature_importances["Future variable importance over time"] fig, ax = plt.subplots(figsize=(20, 10)) bottom = np.zeros(len(df.index)) for col in df.columns: p = ax.bar(np.arange(-len(df), 0), df[col].values, 0.6, label=col, bottom=bottom) bottom += df[col] ax.set_title("Future variable importance over time ponderated by attention") ax.set_ylabel("Importance") ax.set_xlabel("Time") ax.grid(True) ax.legend() plt.show() ``` 2.3 #### Past variable importance over time ```python theme={null} df = feature_importances["Past variable importance over time"] fig, ax = plt.subplots(figsize=(20, 10)) bottom = np.zeros(len(df.index)) for col in df.columns: p = ax.bar(np.arange(-len(df), 0), df[col].values, 0.6, label=col, bottom=bottom) bottom += df[col] ax.set_title("Past variable importance over time") ax.set_ylabel("Importance") ax.set_xlabel("Time") ax.legend() ax.grid(True) plt.show() ``` #### Past variable importance over time ponderated by attention Decomposition of the importance of each time step based on importance of each variable at that time step ```python theme={null} df = feature_importances["Past variable importance over time"] mean_attention = ( nf.models[0] .attention_weights()[nf.models[0].input_size :, :] .mean(axis=0)[: nf.models[0].input_size] ) df = df.multiply(mean_attention, axis=0) fig, ax = plt.subplots(figsize=(20, 10)) bottom = np.zeros(len(df.index)) for col in df.columns: p = ax.bar(np.arange(-len(df), 0), df[col].values, 0.6, label=col, bottom=bottom) bottom += df[col] ax.set_title("Past variable importance over time ponderated by attention") ax.set_ylabel("Importance") ax.set_xlabel("Time") ax.legend() ax.grid(True) plt.plot( np.arange(-len(df), 0), mean_attention, color="black", marker="o", linestyle="-", linewidth=2, label="mean_attention", ) plt.legend() plt.show() ``` ### 3. Variable importance correlations over time Variables which gain and lose importance at same moments ```python theme={null} nf.models[0].feature_importance_correlations() ```