This is a text generation method which returns a generator, streaming out each token in real-time during inference, based on Huggingface/Transformers.
Project description
transformers-stream-generator
Description
This is a text generation method which returns a generator, streaming out each token in real-time during inference, based on Huggingface/Transformers.
Web Demo
- original
- stream
Installation
pip install transformers-stream-generator
Usage
- just add two lines of code before your original code
from transformers_stream_generator import init_stream_support
init_stream_support()
- add
do_stream=True
inmodel.generate
function and keepdo_sample=True
, then you can get a generator
generator = model.generate(input_ids, do_stream=True, do_sample=True)
for token in generator:
word = tokenizer.decode(token)
print(word)
Example
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Close
Hashes for transformers-stream-generator-0.0.4.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 165ec99798aca106e888c772c8b5f10ba95ab8bd103542bf304b93ed8c300b38 |
|
MD5 | 0056a03d8efb710dda5ee0d1e851400e |
|
BLAKE2b-256 | 36263492ab0e45d814533b34ca605f8a20fdc032736f937679c6f212d81a76a5 |