Snapshot of zhouhaoyi/Informer2020: 6.5k★, Python, AI Tools. The GitHub repository for the paper "Informer" accepted by AAAI 2021.
Snapshot summary built from the project's own GitHub metadata — there's no written TopGit review yet. The page will update automatically when a full review is published.
WHY NO REVIEW YET
TopGit writes full reviews for the most-starred, most-requested repositories. This page is a snapshot until then — see the READ ME tab for the original README in full.
Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting (AAAI'21 Best Paper)
This is the origin Pytorch implementation of Informer in the following paper:
Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting. Special thanks to Jieqi Peng@cookieminions for building this repo.
:triangular_flag_on_post:News(Mar 27, 2023): We will release Informer V2 soon.
:triangular_flag_on_post:News(Feb 28, 2023): The Informer's extension paper is online on AIJ.
:triangular_flag_on_post:News(Mar 25, 2021): We update all experiment results with hyperparameter settings.
:triangular_flag_on_post:News(Feb 22, 2021): We provide Colab Examples for friendly usage.
:triangular_flag_on_post:News(Feb 8, 2021): Our Informer paper has been awarded AAAI'21 Best Paper [Official][Beihang][Rutgers]! We will continue this line of research and update on this repo. Please star this repo and cite our paper if you find our work is helpful for you.
Figure 1. The architecture of Informer.
ProbSparse Attention
The self-attention scores form a long-tail distribution, where the "active" queries lie in the "head" scores and "lazy" queries lie in the "tail" area. We designed the ProbSparse Attention to select the "active" queries rather than the "lazy" queries. The ProbSparse Attention with Top-u queries forms a sparse Transformer by the probability distribution.
Why not use Top-u keys? The self-attention layer's output is the re-represent of input. It is formulated as a weighted combination of values w.r.t. the score of dot-product pairs. The top queries with full keys encourage a complete re-represent of leading components in the input, and it is equivalent to selecting the "head" scores among all the dot-product pairs. If we choose Top-u keys, the full keys just preserve the trivial sum of values within the "long tail" scores but wreck the leading components' re-represent.
Figure 2. The illustration of ProbSparse Attention.
Requirements
Python 3.6
matplotlib == 3.1.1
numpy == 1.19.4
pandas == 0.25.1
scikit_learn == 0.21.3
torch == 1.8.0
Dependencies can be installed using the following command:
pip install -r requirements.txt
Data
The ETT dataset used in the paper can be downloaded in the repo ETDataset.
The required data files should be put into data/ETT/ folder. A demo slice of the ETT data is illustrated in the following figure. Note that the input of each dataset is zero-mean normalized in this implementation.
Figure 3. An example of the ETT data.
The ECL data and Weather data can be downloaded here.
Google Drive
BaiduPan, password: 6gan
Reproducibility
To easily reproduce the results you can follow the next steps:
Initialize the docker image using: make init.
Download the datasets using: make dataset.
Run each script in scripts/ using make run_module module="bash ETTh1.sh" for each script.
Alternatively, run all the scripts at once:
for file in `ls scripts`; do make run_module module="bash scripts/$script"; done
Usage
Colab Examples: We provide google colabs to help reproduce and customize our repo, which includes experiments(train and test), prediction, visualization and custom data.
Commands for training and testing the model with ProbSparse self-attention on Dataset ETTh1, ETTh2 and ETTm1 respectively:
The detailed descriptions about the arguments are as following:
Parameter name
Description of parameter
model
The model of experiment. This can be set to informer, informerstack, informerlight(TBD)
data
The dataset name
root_path
The root path of the data file (defaults to ./data/ETT/)
data_path
The data file name (defaults to ETTh1.csv)
features
The forecasting task (defaults to M). This can be set to M,S,MS (M : multivariate predict multivariate, S : univariate predict univariate, MS : multivariate predict univariate)
target
Target feature in S or MS task (defaults to OT)
freq
Freq for time features encoding (defaults to h). This can be set to s,t,h,d,b,w,m (s:secondly, t:minutely, h:hourly, d:daily, b:business days, w:weekly, m:monthly).You can also use more detailed freq like 15min or 3h
checkpoints
Location of model checkpoints (defaults to ./checkpoints/)
seq_len
Input sequence length of Informer encoder (defaults to 96)
label_len
Start token length of Informer decoder (defaults to 48)
pred_len
Prediction sequence length (defaults to 24)
enc_in
Encoder input size (defaults to 7)
dec_in
Decoder input size (defaults to 7)
c_out
Output size (defaults to 7)
d_model
Dimension of model (defaults to 512)
n_heads
Num of heads (defaults to 8)
e_layers
Num of encoder layers (defaults to 2)
d_layers
Num of decoder layers (defaults to 1)
s_layers
Num of stack encoder layers (defaults to 3,2,1)
d_ff
Dimension of fcn (defaults to 2048)
factor
Probsparse attn factor (defaults to 5)
padding
Padding type(defaults to 0).
distil
Whether to use distilling in encoder, using this argument means not using distilling (defaults to True)
dropout
The probability of dropout (defaults to 0.05)
attn
Attention used in encoder (defaults to prob). This can be set to prob (informer), full (transformer)
embed
Time features encoding (defaults to timeF). This can be set to timeF, fixed, learned
activation
Activation function (defaults to gelu)
output_attention
Whether to output attention in encoder, using this argument means outputing attention (defaults to False)
do_predict
Whether to predict unseen future data, using this argument means making predictions (defaults to False)
mix
Whether to use mix attention in generative decoder, using this argument means not using mix attention (defaults to True)
cols
Certain cols from the data files as the input features
num_workers
The num_works of Data loader (defaults to 0)
itr
Experiments times (defaults to 2)
train_epochs
Train epochs (defaults to 6)
batch_size
The batch size of training input data (defaults to 32)
patience
Early stopping patience (defaults to 3)
learning_rate
Optimizer learning rate (defaults to 0.0001)
des
Experiment description (defaults to test)
loss
Loss function (defaults to mse)
lradj
Ways to adjust the learning rate (defaults to type1)
use_amp
Whether to use automatic mixed precision training, using this argument means using amp (defaults to False)
inverse
Whether to inverse output data, using this argument means inversing output data (defaults to False)
use_gpu
Whether to use gpu (defaults to True)
gpu
The gpu no, used for training and inference (defaults to 0)
use_multi_gpu
Whether to use multiple gpus, using this argument means using mulitple gpus (defaults to False)
devices
Device ids of multile gpus (defaults to 0,1,2,3)
Results
We have updated the experiment results of all methods due to the change in data scaling. We are lucky that Informer gets performance improvement. Thank you @lk1983823 for reminding the data scaling in issue 41.
Besides, the experiment parameters of each data set are formated in the .sh files in the directory ./scripts/. You can refer to these parameters for experiments, and you can also adjust the parameters to obtain better mse and mae results or draw better prediction figures.
Figure 4. Univariate forecasting results.
Figure 5. Multivariate forecasting results.
FAQ
If you run into a problem like RuntimeError: The size of tensor a (98) must match the size of tensor b (96) at non-singleton dimension 1, you can check torch version or modify code about Conv1d of TokenEmbedding in models/embed.py as the way of circular padding mode in Conv1d changed in different torch versions.
Citation
If you find this repository useful in your research, please consider citing the following papers:
@article{haoyietal-informerEx-2023,
author = {Haoyi Zhou and
Jianxin Li and
Shanghang Zhang and
Shuai Zhang and
Mengyi Yan and
Hui Xiong},
title = {Expanding the prediction capacity in long sequence time-series forecasting},
journal = {Artificial Intelligence},
volume = {318},
pages = {103886},
issn = {0004-3702},
year = {2023},
}
@inproceedings{haoyietal-informer-2021,
author = {Haoyi Zhou and
Shanghang Zhang and
Jieqi Peng and
Shuai Zhang and
Jianxin Li and
Hui Xiong and
Wancai Zhang},
title = {Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting},
booktitle = {The Thirty-Fifth {AAAI} Conference on Artificial Intelligence, {AAAI} 2021, Virtual Conference},
volume = {35},
number = {12},
pages = {11106--11115},
publisher = {{AAAI} Press},
year = {2021},
}
Contact
If you have any questions, feel free to contact Haoyi Zhou through Email ([email protected]) or Github issues. Pull requests are highly welcomed!
Acknowledgments
Thanks for the computing infrastructure provided by Beijing Advanced Innovation Center for Big Data and Brain Computing (BDBC).
At the same time, thank you all for your attention to this work!
How active is development on zhouhaoyi/Informer2020?
The most recent commit recorded on zhouhaoyi/Informer2020 was 1.4 years ago, based on the GitHub push timestamp. The repository has 1.3k forks — one of the better signals of community interest.
How many stars does zhouhaoyi/Informer2020 have?
zhouhaoyi/Informer2020 has 6.5k GitHub stars — refresh the page for the live number, or check github.com/zhouhaoyi/Informer2020. TopGit mirrors GitHub's count but does not claim minute-by-minute accuracy.
What else is in the AI Tools space?
zhouhaoyi/Informer2020 is tracked by TopGit under the AI Tools category, alongside 6 GitHub-tagged topics. Trending and Topics pages list peer repositories of comparable stars and language.
What language is zhouhaoyi/Informer2020 written in?
zhouhaoyi/Informer2020 is written primarily in Python. GitHub's language field is based on the largest share of bytes in the default branch.
What topics is zhouhaoyi/Informer2020 associated with?
GitHub's repository topics for zhouhaoyi/Informer2020: "deep-learning", "forecasting", "pytorch", "self-attention", "time-series", "transformer". TopGit's editorial category is AI Tools.
Where do I read more about zhouhaoyi/Informer2020?
This TopGit page is a snapshot — the READ ME tab shows the project's own README content (links stripped, images preserved). The GitHub repository at github.com/zhouhaoyi/Informer2020 is the definitive source.
Why is zhouhaoyi/Informer2020 categorized under AI Tools?
TopGit places zhouhaoyi/Informer2020 in the AI Tools category based on its GitHub topics and description (tagged: "deep-learning", "forecasting", "pytorch"). Categories are assigned from real repository metadata, not editorial guesswork.
Read full README in the tab above.
Want a second opinion on Informer2020?
Ask an AI that can read this page — one click and you get its take on Informer2020.