NN Modules

1. MLP

Multi-Layer Perceptron

MLP

 MLP (in_features, out_features, activation, hidden_size, num_layers,
      dropout)

*Multi-Layer Perceptron Class Parameters:
in_features: int, dimension of input.
out_features: int, dimension of output.
activation: str, activation function to use.
hidden_size: int, dimension of hidden layers.
num_layers: int, number of hidden layers.
dropout: float, dropout rate.
*

2. Temporal Convolutions

For long time in deep learning, sequence modelling was synonymous with recurrent networks, yet several papers have shown that simple convolutional architectures can outperform canonical recurrent networks like LSTMs by demonstrating longer effective memory. References
-van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A. W., & Kavukcuoglu, K. (2016). Wavenet: A generative model for raw audio. Computing Research Repository, abs/1609.03499. URL: http://arxiv.org/abs/1609.03499. arXiv:1609.03499.
-Shaojie Bai, Zico Kolter, Vladlen Koltun. (2018). An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling. Computing Research Repository, abs/1803.01271. URL: https://arxiv.org/abs/1803.01271.

Chomp1d

 Chomp1d (horizon)

*Chomp1d Receives x input of dim [N,C,T], and trims it so that only ‘time available’ information is used. Used by one dimensional causal convolutions CausalConv1d. Parameters:
horizon: int, length of outsample values to skip.*

CausalConv1d

 CausalConv1d (in_channels, out_channels, kernel_size, padding, dilation,
               activation, stride:int=1)

*Causal Convolution 1d Receives x input of dim [N,C_in,T], and computes a causal convolution in the time dimension. Skipping the H steps of the forecast horizon, through its dilation. Consider a batch of one element, the dilated convolution operation on the

t

time step is defined:

\mathrm{Conv1D}(\mathbf{x},\mathbf{w})(t) = (\mathbf{x}_{[*d]} \mathbf{w})(t) = \sum^{K}_{k=1} w_{k} \mathbf{x}_{t-dk}

where

d

is the dilation factor,

K

is the kernel size,

t-dk

is the index of the considered past observation. The dilation effectively applies a filter with skip connections. If

d=1

one recovers a normal convolution. Parameters:
in_channels: int, dimension of x input’s initial channels.
out_channels: int, dimension of x outputs’s channels.
activation: str, identifying activations from PyTorch activations. select from ‘ReLU’,‘Softplus’,‘Tanh’,‘SELU’, ‘LeakyReLU’,‘PReLU’,‘Sigmoid’.
padding: int, number of zero padding used to the left.
kernel_size: int, convolution’s kernel size.
dilation: int, dilation skip connections.
Returns:
x: tensor, torch tensor of dim [N,C_out,T] activation(conv1d(inputs, kernel) + bias).
*

TemporalConvolutionEncoder

 TemporalConvolutionEncoder (in_channels, out_channels, kernel_size,
                             dilations, activation:str='ReLU')

*Temporal Convolution Encoder Receives x input of dim [N,T,C_in], permutes it to [N,C_in,T] applies a deep stack of exponentially dilated causal convolutions. The exponentially increasing dilations of the convolutions allow for the creation of weighted averages of exponentially large long-term memory. Parameters:
in_channels: int, dimension of x input’s initial channels.
out_channels: int, dimension of x outputs’s channels.
kernel_size: int, size of the convolving kernel.
dilations: int list, controls the temporal spacing between the kernel points.
activation: str, identifying activations from PyTorch activations. select from ‘ReLU’,‘Softplus’,‘Tanh’,‘SELU’, ‘LeakyReLU’,‘PReLU’,‘Sigmoid’.
Returns:
x: tensor, torch tensor of dim [N,T,C_out].
*

3. Transformers

References
- Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, Wancai Zhang. “Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting”
- Haixu Wu, Jiehui Xu, Jianmin Wang, Mingsheng Long.

TransEncoder

 TransEncoder (attn_layers, conv_layers=None, norm_layer=None)

*Base class for all neural network modules. Your models should also subclass this class. Modules can also contain other Modules, allowing them to be nested in a tree structure. You can assign the submodules as regular attributes::

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self) -> None:
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

Submodules assigned in this way will be registered, and will also have their parameters converted when you call :meth:to, etc. .. note:: As per the example above, an __init__() call to the parent class must be made before assignment on the child. :ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool*

TransEncoderLayer

 TransEncoderLayer (attention, hidden_size, conv_hidden_size=None,
                    dropout=0.1, activation='relu')

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self) -> None:
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

TransDecoder

 TransDecoder (layers, norm_layer=None, projection=None)

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self) -> None:
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

TransDecoderLayer

 TransDecoderLayer (self_attention, cross_attention, hidden_size,
                    conv_hidden_size=None, dropout=0.1, activation='relu')

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self) -> None:
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

AttentionLayer

 AttentionLayer (attention, hidden_size, n_heads, d_keys=None,
                 d_values=None)

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self) -> None:
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

FullAttention

 FullAttention (mask_flag=True, factor=5, scale=None,
                attention_dropout=0.1, output_attention=False)

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self) -> None:
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

TriangularCausalMask

 TriangularCausalMask (B, L, device='cpu')

TriangularCausalMask

source

DataEmbedding_inverted

 DataEmbedding_inverted (c_in, hidden_size, dropout=0.1)

DataEmbedding_inverted

DataEmbedding

 DataEmbedding (c_in, exog_input_size, hidden_size, pos_embedding=True,
                dropout=0.1)

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self) -> None:
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

TemporalEmbedding

 TemporalEmbedding (d_model, embed_type='fixed', freq='h')

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self) -> None:
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

FixedEmbedding

 FixedEmbedding (c_in, d_model)

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self) -> None:
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

TimeFeatureEmbedding

 TimeFeatureEmbedding (input_size, hidden_size)

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self) -> None:
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

source

TokenEmbedding

 TokenEmbedding (c_in, hidden_size)

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self) -> None:
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

PositionalEmbedding

 PositionalEmbedding (hidden_size, max_len=5000)

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self) -> None:
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

source

SeriesDecomp

 SeriesDecomp (kernel_size)

Series decomposition block

source

MovingAvg

 MovingAvg (kernel_size, stride)

Moving average block to highlight the trend of time series

RevIN

 RevIN (num_features:int, eps=1e-05, affine=False, subtract_last=False,
        non_norm=False)

RevIN (Reversible-Instance-Normalization)

RevINMultivariate

 RevINMultivariate (num_features:int, eps=1e-05, affine=False,
                    subtract_last=False, non_norm=False)

ReversibleInstanceNorm1d for Multivariate models

Getting Started

Capabilities

Tutorials

Use cases

API Reference

1. MLP

MLP

2. Temporal Convolutions

Chomp1d

CausalConv1d

TemporalConvolutionEncoder

3. Transformers

TransEncoder

TransEncoderLayer

TransDecoder

TransDecoderLayer

AttentionLayer

FullAttention

TriangularCausalMask

DataEmbedding_inverted

DataEmbedding

TemporalEmbedding

FixedEmbedding

TimeFeatureEmbedding

TokenEmbedding

PositionalEmbedding

SeriesDecomp

MovingAvg

RevIN

RevINMultivariate

​1. MLP

​MLP

​2. Temporal Convolutions

​Chomp1d

​CausalConv1d

​TemporalConvolutionEncoder

​3. Transformers

​TransEncoder

​TransEncoderLayer

​TransDecoder

​TransDecoderLayer

​AttentionLayer

​FullAttention

​TriangularCausalMask

​DataEmbedding_inverted

​DataEmbedding

​TemporalEmbedding

​FixedEmbedding

​TimeFeatureEmbedding

​TokenEmbedding

​PositionalEmbedding

​SeriesDecomp

​MovingAvg

​RevIN

​RevINMultivariate

1. MLP

MLP

2. Temporal Convolutions

Chomp1d

CausalConv1d

TemporalConvolutionEncoder

3. Transformers

TransEncoder

TransEncoderLayer

TransDecoder

TransDecoderLayer

AttentionLayer

FullAttention

TriangularCausalMask

DataEmbedding_inverted

DataEmbedding

TemporalEmbedding

FixedEmbedding

TimeFeatureEmbedding

TokenEmbedding

PositionalEmbedding

SeriesDecomp

MovingAvg

RevIN

RevINMultivariate