This is a text generation method which returns a generator, streaming out each token in real-time during inference, based on Huggingface/Transformers.
Project description
transformers-stream-generator
Description
This is a text generation method which returns a generator, streaming out each token in real-time during inference, based on Huggingface/Transformers.
Web Demo
- original
- stream
Installation
pip install transformers-stream-generator
Usage
- just add two lines of code before your original code
from transformers_stream_generator import init_stream_support
init_stream_support()
- add
do_stream=True
inmodel.generate
function and keepdo_sample=True
, then you can get a generator
generator = model.generate(input_ids, do_stream=True, do_sample=True)
for token in generator:
word = tokenizer.decode(token)
print(word)
Example
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Close
Hashes for transformers-stream-generator-0.0.5.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 271deace0abf9c0f83b36db472c8ba61fdc7b04d1bf89d845644acac2795ed57 |
|
MD5 | 069ae3115525fa148d88af8f01772ee2 |
|
BLAKE2b-256 | 42c265f13aec253100e1916e9bd7965fe17bde796ebabeb1265f45191ab4ddc0 |