News
SRU++, a new SRU variant, is released. [tech report] [blog]
The experimental code and SRU++ implementation are available on the dev branch which will be merged into master later.
About
SRU is a recurrent unit that can run over 10 times faster than cuDNN LSTM, without loss of accuracy tested on many tasks.
Average processing time of LSTM, conv2d and SRU, tested on GTX 1070
Reference:
Simple Recurrent Units for Highly Parallelizable Recurrence [paper]
@inproceedings{lei2018sru,
title={Simple Recurrent Units for Highly Parallelizable Recurrence},
author={Tao Lei and Yu Zhang and Sida I. Wang and Hui Dai and Yoav Artzi},
booktitle={Empirical Methods in Natural Language Processing (EMNLP)},
year={2018}
}
When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute [paper]
@article{lei2021srupp,
title={When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute},
author={Tao Lei},
journal={arXiv preprint arXiv:2102.12459},
year={2021}
}
Requirements
Install requirements via pip install -r requirements.txt.
Installation
From source:
SRU can be installed as a regular package via python setup.py install or pip install ..
From PyPi:
pip install sru
Directly use the source without installation:
Make sure this repo and CUDA library can be found by the system, e.g.
export PYTHONPATH=path_to_repo/sru
export LD_LIBRARY_PATH=/usr/local/cuda/lib64
Examples
The usage of SRU is similar to nn.LSTM. SRU likely requires more stacking layers than LSTM. We recommend starting by 2 layers and use more if necessary (see our report for more experimental details).
import torch
from sru import SRU, SRUCell
# input has length 20, batch size 32 and dimension 128
x = torch.FloatTensor(20, 32, 128).cuda()
input_size, hidden_size = 128, 128
rnn = SRU(input_size, hidden_size,
num_layers = 2, # number of stacking RNN layers
dropout = 0.0, # dropout applied between RNN layers
bidirectional = False, # bidirectional RNN
layer_norm = False, # apply layer normalization on the output of each layer
highway_bias = -2, # initial bias of highway gate (<= 0)
)
rnn.cuda()
output_states, c_states = rnn(x) # forward pass
# output_states is (length, batch size, number of directions * hidden size)
# c_states is (layers, batch size, number of directions * hidden size)
Contributing
Please read and follow the guidelines.
Other Implementations
@musyoku had a very nice SRU implementaion in chainer.
@adrianbg implemented the first CPU version.
2 maintainers
Metadata
2 maintainers
Release files for sru 2.6.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| sru-2.6.0.tar.gz | 22.4 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| sru-2.6.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 46.6 kB
Release files / sru-2.6.0.tar.gz
| Download URL | sru-2.6.0.tar.gz |
|---|---|
| Size | 22.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
e472e32bb3492d10faa121a29f985660c81fb649989924da90c8d1fcafd1152c
|
|
BLAKE2b-256 checksum How to use checksums |
40ca7537e0ef8c3361402b1787474f0960521d4de82673ab45c1f11909e1c7a1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.1 importlib_metadata/4.0.1 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.6.13
|
Release files / sru-2.6.0-py3-none-any.whl
| Download URL | sru-2.6.0-py3-none-any.whl |
|---|---|
| Size | 24.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
bcb4d6c095145ffc564223067e60f5d8a6f363ba66de00a87de14faa9947058f
|
|
BLAKE2b-256 checksum How to use checksums |
905606bfae51fab82bae05567af7f43f22e0948fe06c76b06b77ae67846a6089
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.1 importlib_metadata/4.0.1 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.6.13
|
Security for sru 2.6.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
