Unofficial python bindings for llm-rs. 🐍❤️🦀
Project description
llm-rs-python: Python Bindings for Rust's llm Library
Welcome to llm-rs
, an unofficial Python interface for the Rust-based llm library, made possible through PyO3. Our package combines the convenience of Python with the performance of Rust to offer an efficient tool for your machine learning projects. 🐍❤️🦀
With llm-rs
, you can operate a variety of Large Language Models (LLMs) including LLama and GPT-NeoX directly on your CPU.
For a detailed overview of all the supported architectures, visit the llm project page.
Installation
Simply install it via pip: pip install llm-rs
Usage
Running local GGML models:
Models can be loaded via the AutoModel
interface.
from llm_rs import AutoModel, KnownModels
#load the model
model = AutoModel.from_pretrained("path/to/model.bin",model_type=KnownModels.Llama)
#generate
print(model.generate("The meaning of life is"))
Streaming Text
Text can be yielded from a generator via the stream
function:
from llm_rs import AutoModel, KnownModels
#load the model
model = AutoModel.from_pretrained("path/to/model.bin",model_type=KnownModels.Llama)
#generate
for token in model.stream("The meaning of life is"):
print(token)
Running GGML models from the Hugging Face Hub
GGML converted models can be directly downloaded and run from the hub.
from llm_rs import AutoModel
model = AutoModel.from_pretrained("rustformers/mpt-7b-ggml",model_file="mpt-7b-q4_0-ggjt.bin")
If there are multiple models in a repo the model_file
has to be specified.
If you want to load repositories which were not created throught this library, you have to specify the model_type
parameter as the metadata files needed to infer the architecture are missing.
Running Pytorch Transfomer models from the Hugging Face Hub
llm-rs
supports automatic conversion of all supported transformer architectures on the Huggingface Hub.
To run covnersions additional dependencies are needed which can be installed via pip install llm-rs[convert]
.
The models can then be loaded and automatically converted via the from_pretrained
function.
from llm_rs import AutoModel
model = AutoModel.from_pretrained("mosaicml/mpt-7b")
Convert Huggingface Hub Models
The following example shows how a Pythia model can be covnverted, quantized and run.
from llm_rs.convert import AutoConverter
from llm_rs import AutoModel, AutoQuantizer
import sys
#define the model which should be converted and an output directory
export_directory = "path/to/directory"
base_model = "EleutherAI/pythia-410m"
#convert the model
converted_model = AutoConverter.convert(base_model, export_directory)
#quantize the model (this step is optional)
quantized_model = AutoQuantizer.quantize(converted_model)
#load the quantized model
model = AutoModel.load(quantized_model,verbose=True)
#generate text
def callback(text):
print(text,end="")
sys.stdout.flush()
model.generate("The meaning of life is",callback=callback)
Documentation
For in-depth information on customizing the loading and generation processes, refer to our detailed documentation.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for llm_rs-0.2.8-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 24494b2ac394dea1d6e9773213521296bbe6db5018a0032f5311dab7839c3586 |
|
MD5 | 6d461cc430676bac7e0731503b4bf989 |
|
BLAKE2b-256 | 799bef368f40bbe65511750513c518f504f031c10196077e38f6572238052b8e |
Hashes for llm_rs-0.2.8-pp39-pypy39_pp73-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0405e21c1c0b0658c4193076f56d2b0e45dac86774b02c5025148c8c6941c18f |
|
MD5 | 3b0b6d93d2ab84464d6a029862e77302 |
|
BLAKE2b-256 | 945973a8bde7e2b91f8fc7e08e9e865cb01f0d6c71cc2a4ba2e636c90816bf67 |
Hashes for llm_rs-0.2.8-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cd0079e90292430cdbb92ec9bc06a9686cc976a154d15a8d5051b0b443eae993 |
|
MD5 | ed4b0649902cc40b97b6e36bac52296b |
|
BLAKE2b-256 | 8f3ac014cf759a3b94df1ea6047e82843e32e584a3dd20634ba6480d5827f9be |
Hashes for llm_rs-0.2.8-pp38-pypy38_pp73-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c6e7e062ad23ab6b92fb5ee70d2a8c8005fdb18b3214369d23d23f993ee82793 |
|
MD5 | 384992fe6810e048af53eadfe3703ab8 |
|
BLAKE2b-256 | 18c1a528d43d87ad33d2fbf653e315ca45e985c3324dbca5a9c5ca22a84091ca |
Hashes for llm_rs-0.2.8-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 18b613afc1e3f3aeec5f900d1491636009cb0d8e7c009009702d6b2d9c19f00c |
|
MD5 | e96215f9a70c83968faf5914e05c8c4c |
|
BLAKE2b-256 | 5c48729c2c4883c5ccaae16b572f3e10881a779dff9fb368e2a9931d0a4f7789 |
Hashes for llm_rs-0.2.8-pp37-pypy37_pp73-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b016248907f0e5ef59d0f764541c6ed6f581997724093b6e36250ae4ec5f9283 |
|
MD5 | 597dd0a9db73dde71dfd2e9590a4af19 |
|
BLAKE2b-256 | 4f2240f797f66489a800a32263e8848cf7bb2d13c6dddba5ceee056c997c2668 |
Hashes for llm_rs-0.2.8-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f65a565ba4165395c739585cdea99673e44d87d5a39633f85d46a8e62779be7c |
|
MD5 | ac9301f2766de43c88b79da5d82dab3d |
|
BLAKE2b-256 | 422b077a9494dc5cf29dd3ff7bb6cdf95ddae3f3b61e1644a2aae94192c58073 |
Hashes for llm_rs-0.2.8-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5b8eae78a827035e596913389115b7c306a86d97a0c852cb5af6ae68bcac92c9 |
|
MD5 | 8a3e2cc88fd8480c567c3b42e16bbfca |
|
BLAKE2b-256 | 0bef04adca66481df45618d3ac8f752e78e402e9442301a9a899a16ab77c5264 |
Hashes for llm_rs-0.2.8-cp311-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ae2eb09e3959d97a7b74f86ec70b0418441b72829f6fba820b5c05aae590b0a1 |
|
MD5 | 848e2d13a39c4981ca6e3b4cda851b21 |
|
BLAKE2b-256 | 84ddd5a6e37c81f4bd3d12ae1d32d63c648e5004ab9e273b5ef919d34b1a0efb |
Hashes for llm_rs-0.2.8-cp311-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e9cb552dd7a7b8eddd256dcd12e980f60d1a04b85cae9a50c75e6b58096e9829 |
|
MD5 | 4e1140e874c61dfa202b2367f525e4fd |
|
BLAKE2b-256 | d687b4dc9176179adef95889a20c875cea05557614c25034d407d8f0ee24676e |
Hashes for llm_rs-0.2.8-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f74cff29e18815ff435d19f33e1a0e88394e1113ed2c8aed50f48fa59665ecc9 |
|
MD5 | aa38d9b93690757955786f71f01aba58 |
|
BLAKE2b-256 | 3f260cc7b6559485ba879099e2ef65aa158fb05fa452580539eefbd2097103cb |
Hashes for llm_rs-0.2.8-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 42294c8d7ecba79593986c53bc54482e5abb48745665e6edb30438a2d11b286c |
|
MD5 | 13d4c87a2f18e8e591effbb5ec01a02a |
|
BLAKE2b-256 | 1bb26b7249be77e7ee4010f796c5d76a959134e73dc2f03459e73848323dc768 |
Hashes for llm_rs-0.2.8-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5bf97d7414eddaf23fa532d14f45db83fc29bea679249e6c107ceb5940022979 |
|
MD5 | 28e0dd32b25711b4bd045cd3632ee3f5 |
|
BLAKE2b-256 | 4c1005a9b076ff2f67ffc7f51e3feecf0263163a71608e3816213ef41691a23e |
Hashes for llm_rs-0.2.8-cp311-cp311-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b069f33e1394f4b5ab532bd182871aa2984779ac8d83bdaef2db960e08788c10 |
|
MD5 | 0927e48f7437b537f46dcbb887ebb360 |
|
BLAKE2b-256 | 12716f2ba9a5391c7c15d89151e6c8c64682b0669137a3e9c838d3e20caf87ab |
Hashes for llm_rs-0.2.8-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 35238631f4ddb02e1f67e578e300c9ed30ef5a3a9f69e8cd4aa162927e4301cd |
|
MD5 | e95052cc6679926175883a9ffceca146 |
|
BLAKE2b-256 | 0c6983f22cb2031047fb091eab267244cdac31742601f115ab75dd2beda17fd0 |
Hashes for llm_rs-0.2.8-cp310-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fdaf0146e43607089a40077f98434cba08d6d27148c62e07afb2e702be4d095b |
|
MD5 | 9ada8295c1ef18792651845f636bb5da |
|
BLAKE2b-256 | 70963fdaa02e6d346c71068d3a066f0d4d75c3ee2af4f21e8c756efd2f0789bc |
Hashes for llm_rs-0.2.8-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f49d6dad97fbb4f39cdc1a6618ceea50f10ef4eb9dfc5eb9a72e1353f9a6845a |
|
MD5 | 56582713870af280f676c3e730b72698 |
|
BLAKE2b-256 | 06018d7ef6c75ff0cef72541076ea41c70427bf8fa9c7952071259955e42471f |
Hashes for llm_rs-0.2.8-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 817801e7466bfcc19412ee9d2a178c008ed181aa1e5979f195662edb629cc717 |
|
MD5 | dcd1d20b5a35d91e8cf74915c032d255 |
|
BLAKE2b-256 | def6824fe20a42dac5cef65e418b70b9ac289ff9bb674a9692c112adf283637a |
Hashes for llm_rs-0.2.8-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2bb6b11c947d660e931a4946bb8c04e908cae785c53ca8aaf66f3620bd68278d |
|
MD5 | 8204675da08357618f6247b700e30d06 |
|
BLAKE2b-256 | af2753d37a2365982c06ef9042c4b8ce01c3611695d2869f6277a98d4542128a |
Hashes for llm_rs-0.2.8-cp310-cp310-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6deb13b0ad8af3f171b820b0c609d24628491c8004cff0ca38dff5c8392f6c9c |
|
MD5 | 8b1b93c3dfaffaf326126795a985a766 |
|
BLAKE2b-256 | 2a673ac0ded6494efbb57c43adc312eaca02118ca824ca28d7d7d4989e3dcc59 |
Hashes for llm_rs-0.2.8-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | af6339c20094ec302b5b4d0b2b6235f8d13db3e3340c7f3d8ab792e4304b6180 |
|
MD5 | ea9e4502d7d3ee50bf192fbea6842c2f |
|
BLAKE2b-256 | 88fada342d888b6f6add5f63f0fa8f2322030492d0001561556ac709b98273b4 |
Hashes for llm_rs-0.2.8-cp39-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0bc15d0e339e7d1b768dd6a1144f7f5bc931cfa3f8bd1bc8bb73899e94028df7 |
|
MD5 | d78123f3a1d80e7ceffc47d83dc8ae90 |
|
BLAKE2b-256 | 69aa140e60b3488f1cd45d5fcfdd11992cfa4ff8ed9f110c34cd777f0b863ce4 |
Hashes for llm_rs-0.2.8-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 31fb42145e8cf9be21c40510dfd2cec80fc587452c9c9370c2f426294d6bf0d9 |
|
MD5 | 65b90f674f69b9256a0071d7bf2201da |
|
BLAKE2b-256 | 7113e3ed9fdf06e00208b002a7334c1cd4ff81b5f6a9a13626c341eca6af2d99 |
Hashes for llm_rs-0.2.8-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3297183bca2901a14b8c2b05bdece6da8489b4518eec8127eeab62b146fa77f6 |
|
MD5 | 1d42db676033abafa6f9fba7331ab485 |
|
BLAKE2b-256 | b2d0ed1d53e52dde31685660335647a4d4f09acfec0183857a1be99e2e4b3c51 |
Hashes for llm_rs-0.2.8-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5d461fb08cecf106d05914cfeac190d7d9eaa11b312961eb347ab1cb2a10c79b |
|
MD5 | 5bacc8783dac13f176730b9d4060ac47 |
|
BLAKE2b-256 | 7d58f04c493c89942ce0b85ef6ef2292bfba76285a3816ea88f345d925713e26 |
Hashes for llm_rs-0.2.8-cp38-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 30d375ca4d952b015f8be38cea1eded71db174ecf8934c0d475d8844011b3757 |
|
MD5 | 59ac76aa99f17a12537e838dfbf42746 |
|
BLAKE2b-256 | 4bb6b09e8edea967799212b97b5560ad3362c32c812ae09f70009e68f3ba2907 |
Hashes for llm_rs-0.2.8-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bae7e1a31e3ee5b03ec55dcad2eb41c300d838140d98808b3eb8c999dfdb328a |
|
MD5 | c90a7c17b7d4e76001edd7d7264d02c1 |
|
BLAKE2b-256 | 9676a7c60fdeaf23763b985c525592a47e57fc978cfea2e7ec7b4655d4f9be8a |
Hashes for llm_rs-0.2.8-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 80a7e5e72307b4583aaa8be35589501d9bface7d92c9ccf8859f6b968ac1c8ed |
|
MD5 | d8d5d36586628a03b5670f9991a642ff |
|
BLAKE2b-256 | fdca110c05e2efa716995ac9870ac004221ca2cd49da0ec65b1315afd1fc10c2 |
Hashes for llm_rs-0.2.8-cp37-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2244e27b947325c31052aa2460549e747e4ec2d657d24efc496dce760520ac43 |
|
MD5 | 2474d7bdfdcaef289e809e5ff2759b63 |
|
BLAKE2b-256 | 02b50997718e712ff08dd51d5a309458b50231368dda111f6fc25a8e2b9e8337 |
Hashes for llm_rs-0.2.8-cp37-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e1485381b358ff4bd6184fc2588ae3c47779b0196d45f9e0fdfbcfa3387a9038 |
|
MD5 | d2ac785f3ab7939a10987a0834bdc237 |
|
BLAKE2b-256 | c6385d56b470d661719c921bcacbdd780a1ba82e423a2b1e75603355d032c436 |
Hashes for llm_rs-0.2.8-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4ff884b3f2ddc81ca2b19b0b89b3635058db6975f24e8cdbec5cbc40b75c57e7 |
|
MD5 | 367b80f5ad16c7e1ba8b29a706e25a60 |
|
BLAKE2b-256 | af775d5fd0ed8ce1a0ee2245545a9367b46afe2b48736fd08829fa80ae51faee |
Hashes for llm_rs-0.2.8-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b0b659cbf429e3ffe8db29172b67a3c36a435b2f3b36bd229cd4086bb2673178 |
|
MD5 | a9740bcdc04e7783c5f522e8d3d786f5 |
|
BLAKE2b-256 | 21a3343fb1ca26c1fc958dbcaa3da07c2bd7994e47a191b9bb5c594b36d15274 |