Skip to content

Commit

Permalink
Update README.md
Browse files Browse the repository at this point in the history
  • Loading branch information
taoleicn authored Feb 25, 2021
1 parent 4f691ec commit ca404e3
Showing 1 changed file with 17 additions and 4 deletions.
21 changes: 17 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@

## News
SRU++, a new SRU variant, is released. [[tech report](https://arxiv.org/pdf/2102.1245))] [[blog](https://www.asapp.com/blog/reducing-the-high-cost-and-carbon-footprint-of-training-nlp-models-with-sru/)]

The experimental code and SRU++ implementation are available on [the dev branch](https://github.com/asappresearch/sru/tree/3.0.0-dev/experiments/srupp_experiments) which will be merged into master later.

## About

**SRU** is a recurrent unit that can run over 10 times faster than cuDNN LSTM, without loss of accuracy tested on many tasks.
Expand All @@ -8,10 +13,8 @@
</p>
For example, the figure above presents the processing time of a single mini-batch of 32 samples. SRU achieves 10 to 16 times speed-up compared to LSTM, and operates as fast as (or faster than) word-level convolution using conv2d.

The [paper](https://arxiv.org/abs/1709.02755) has multiple versions, please check the latest one.

#### Reference:
Simple Recurrent Units for Highly Parallelizable Recurrence
Simple Recurrent Units for Highly Parallelizable Recurrence [[paper](https://arxiv.org/abs/1709.02755)]
```
@inproceedings{lei2018sru,
title={Simple Recurrent Units for Highly Parallelizable Recurrence},
Expand All @@ -20,10 +23,20 @@ Simple Recurrent Units for Highly Parallelizable Recurrence
year={2018}
}
```

When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute [[paper](https://arxiv.org/pdf/2102.12459)]
```
@article{lei2021srupp,
title={When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute},
author={Tao Lei},
journal={arXiv preprint arXiv:2102.12459},
year={2021}
}
```
<br>

## Requirements
- [PyTorch](http://pytorch.org/) >=1.3.1 recommended, [pytorch installation details](docs/pytorch_installation.md)
- [PyTorch](http://pytorch.org/) >=1.6 recommended
- [ninja](https://ninja-build.org/)

Install requirements via `pip install -r requirements.txt`.
Expand Down

0 comments on commit ca404e3

Please sign in to comment.