Unofficial python bindings for llm-rs. 🐍❤️🦀
Project description
llm-rs-python: Python Bindings for Rust's llm Library
Welcome to llm-rs
, an unofficial Python interface for the Rust-based llm library, made possible through PyO3. Our package combines the convenience of Python with the performance of Rust to offer an efficient tool for your machine learning projects. 🐍❤️🦀
With llm-rs
, you can operate a variety of Large Language Models (LLMs) including LLama and GPT-NeoX directly on your CPU.
For a detailed overview of all the supported architectures, visit the llm project page.
Installation
Simply install it via pip: pip install llm-rs
Usage
Running local GGML models:
Models can be loaded via the AutoModel
interface.
from llm_rs import AutoModel, KnownModels
#load the model
model = AutoModel.from_pretrained("path/to/model.bin",model_type=KnownModels.Llama)
#generate
print(model.generate("The meaning of life is"))
Streaming Text
Text can be yielded from a generator via the stream
function:
from llm_rs import AutoModel, KnownModels
#load the model
model = AutoModel.from_pretrained("path/to/model.bin",model_type=KnownModels.Llama)
#generate
for token in model.stream("The meaning of life is"):
print(token)
Running GGML models from the Hugging Face Hub
GGML converted models can be directly downloaded and run from the hub.
from llm_rs import AutoModel
model = AutoModel.from_pretrained("rustformers/mpt-7b-ggml",model_file="mpt-7b-q4_0-ggjt.bin")
If there are multiple models in a repo the model_file
has to be specified.
If you want to load repositories which were not created throught this library, you have to specify the model_type
parameter as the metadata files needed to infer the architecture are missing.
Running Pytorch Transfomer models from the Hugging Face Hub
llm-rs
supports automatic conversion of all supported transformer architectures on the Huggingface Hub.
To run covnersions additional dependencies are needed which can be installed via pip install llm-rs[convert]
.
The models can then be loaded and automatically converted via the from_pretrained
function.
from llm_rs import AutoModel
model = AutoModel.from_pretrained("mosaicml/mpt-7b")
Convert Huggingface Hub Models
The following example shows how a Pythia model can be covnverted, quantized and run.
from llm_rs.convert import AutoConverter
from llm_rs import AutoModel, AutoQuantizer
import sys
#define the model which should be converted and an output directory
export_directory = "path/to/directory"
base_model = "EleutherAI/pythia-410m"
#convert the model
converted_model = AutoConverter.convert(base_model, export_directory)
#quantize the model (this step is optional)
quantized_model = AutoQuantizer.quantize(converted_model)
#load the quantized model
model = AutoModel.load(quantized_model,verbose=True)
#generate text
def callback(text):
print(text,end="")
sys.stdout.flush()
model.generate("The meaning of life is",callback=callback)
Documentation
For in-depth information on customizing the loading and generation processes, refer to our detailed documentation.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for llm_rs-0.2.7-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d8482621c3f8171c6c7927c9c1c42e47832101d517e253339ac44d2150781c18 |
|
MD5 | 71ffaae014ee8417a15cefeb9a5befaf |
|
BLAKE2b-256 | 15b795474c8e73cd3f0e7501178f040ba2e0c7cbc8ea8e82af2e9b73625fdbd6 |
Hashes for llm_rs-0.2.7-pp39-pypy39_pp73-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e8193347f36f9505aa92f97d9aae7a3b9c68696cfb3266d3376ca37f544b7519 |
|
MD5 | 236e18d8adaae2f2b8b67805eeb6a8da |
|
BLAKE2b-256 | 482a2db4e1a60e475e294eb68176f1845ba8e5b6206f090621b062b97c206366 |
Hashes for llm_rs-0.2.7-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a89e71fc2224da140c23e491631d849c626a70a468b53d41c8d5987993f0afe0 |
|
MD5 | a5d6b54e6b1f4fe1705e6f789c03b4fa |
|
BLAKE2b-256 | 7a535f9e8a8fa6fb90e20f67570313ba441d1f11b7b6a5fe750e573d80f58b56 |
Hashes for llm_rs-0.2.7-pp38-pypy38_pp73-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dbd9e3b831ff6e80ed335999507fa68824a980cc6aad20c57fbd9b3f282af967 |
|
MD5 | 686f3fd28fef45af10f4a132bba029c3 |
|
BLAKE2b-256 | 777064a89d3cc54a93d0938d6c49478ab169f259d557a40d4bf2faacba87ba3e |
Hashes for llm_rs-0.2.7-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 06a8af47e632150a2e41556592e03e8d106f42a157e32ceb0d98e2366686bc6a |
|
MD5 | c55d43cdfc01e5a40b288403f60215cf |
|
BLAKE2b-256 | 1f6007546974ba84273fce37eb2ea8531b237f29ebcd66c64bb3406ea4075ac5 |
Hashes for llm_rs-0.2.7-pp37-pypy37_pp73-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c799fe3bc7a95ff12ebb352569ba6aec830b9bfd1108d1bcb52011011d453017 |
|
MD5 | c069acb4be853a70873d1c71df872312 |
|
BLAKE2b-256 | 4a05502fae6992575c1e58b1aecab9ef7fd3cad045d113ab14604a69cc80f890 |
Hashes for llm_rs-0.2.7-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b04dba4bc93afdcaa9292357c120962636c16e5c78c51e97de7ace7f8eefdf5a |
|
MD5 | 295f7549ba083376500b9e1023ef879c |
|
BLAKE2b-256 | 25e596916e14a30dfebc3e91848952a29153cfd23ae914d1620e55866ae4e19e |
Hashes for llm_rs-0.2.7-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ee3bec75ceb8b448e71a1a031edf7fb957b25c6f8777cbabbae710239bc25da4 |
|
MD5 | 6b11f8af0d867a43fb3711205f699f01 |
|
BLAKE2b-256 | d16fc2e617a101c47835ee448bcf442996a9dce08bb6dcca34a2dc3cebb46e9d |
Hashes for llm_rs-0.2.7-cp311-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 52caea34af568085ecee913e459d29081a899a9fec7222f134f52b83482dc4cb |
|
MD5 | a7925d8dfa293aba82b7d0bf08c5bddd |
|
BLAKE2b-256 | 0cf0844955169385441374c205cca614c97b8bfb8fce0fcff2c17c9eaedbf091 |
Hashes for llm_rs-0.2.7-cp311-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ce224b335dfac3125be05546de026bed10a962365767da460f755695a2f012e8 |
|
MD5 | 8514252ab8598473b1b7417227353a05 |
|
BLAKE2b-256 | 37afa22da4123d501fed7386098bd5eb01b0fa03cd73027e4c6e744fadbfc7ce |
Hashes for llm_rs-0.2.7-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9c8ed8852782c0b4de5dc44ce169c0c2409398c6f5ba3cec28f6b2ab2282d837 |
|
MD5 | f7931c0b3aa9b8a6713e695cd6d5b991 |
|
BLAKE2b-256 | 64caf2631ed75a389952e2d5f0a30f56b4076a77ea003d269d121f418b4899a8 |
Hashes for llm_rs-0.2.7-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6e8ee3b004e1ee6ea5a9d5759e7bef5c36801a95164e09734d14cf4d15286863 |
|
MD5 | dabcdc2b6415d9d678f6ec3419dccff0 |
|
BLAKE2b-256 | ddcff5fdd3dbef9cb3dd47a2b731fdc16ffb1c6fc71b54b59358d6a4cb013806 |
Hashes for llm_rs-0.2.7-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 75d838e7e07d277c31554f1455f4cbf107a774add8f9bf51aa4186b24b2d9891 |
|
MD5 | 8bec77bf3ecb299afdee5a59359f7101 |
|
BLAKE2b-256 | 12070f5a2319e2cf33ad63c4f847c733bdc8af4f8fd6a0fd1522ffaab998618e |
Hashes for llm_rs-0.2.7-cp311-cp311-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 582ac05bff2ad8a99b19db277843113c13ef84f80ebaf1c49f7051a137a4df5e |
|
MD5 | 938e5961e719efa280f86bd2ef4cc939 |
|
BLAKE2b-256 | 580409f4581c2acb6f426b12c70a2e0d155f3a79301616a7cf3a04b23d1db89a |
Hashes for llm_rs-0.2.7-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 94daacbeda43fe475d126cea09ce824ed3fac11fab06f0afaf1a4d6d3b4d9bdf |
|
MD5 | c0076d447b30fc61c7c6624fc7b4b5c2 |
|
BLAKE2b-256 | d54da097bf4008646313f3d0d97e48f2fc03edab1e64bb25a8ec1c624cf56d9b |
Hashes for llm_rs-0.2.7-cp310-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 84cf0c7c0b924f31d23e0fd90972b7de6bc0ccfe101f3669c301b8a623874627 |
|
MD5 | 615e62407938e9cdbc31941047bdbd55 |
|
BLAKE2b-256 | 9965d3d2e1fd79f890a0dde1504cd112e543310852afcd9405ebaa5aebf762e4 |
Hashes for llm_rs-0.2.7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f67c2cfb78ec15460f18a512a33639efc80f96956f64084f803c35e66c0bdf8e |
|
MD5 | 039d6f1b6b2453795a0ef73273364860 |
|
BLAKE2b-256 | f49976dca9aba719ef181997f49e9f60897c769513d961c0acaaafe532e7ae3b |
Hashes for llm_rs-0.2.7-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 067496a952d8e3dc3f5811a1af91140817ebbab40be09854a84bad2d6390e0ef |
|
MD5 | b7607385860267a040efcc249c18fe00 |
|
BLAKE2b-256 | 831131466fd33b42cc1b301ca225f4c10404f5364d5aa8dd9847e3ac2b4cdbeb |
Hashes for llm_rs-0.2.7-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e37d508f1487648793a6ed1b2178985099c11031d1f0bc5cfbd5c65afbb1384e |
|
MD5 | a481f8437d8044fba9c7dae1cbdb184b |
|
BLAKE2b-256 | a6ecd39d395a3eaa2e5364db3d6f878dd3650ae1ecaebb164e4cb322a9a6d212 |
Hashes for llm_rs-0.2.7-cp310-cp310-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f3a8893def83ebf5e510409cc9e33d71fff034d739989e89756185063aadd3c5 |
|
MD5 | 531c994e1a90b776ce000906aaefa2db |
|
BLAKE2b-256 | b88acab988d6bd562d790657c36fbd45d275335b2c2d6fe45814927daa94e7a4 |
Hashes for llm_rs-0.2.7-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6e1b5ed84a8221a452fe68ddc8ff5c72a0f648ce1ef9966c8c9adbc51210597e |
|
MD5 | 4927714b1188b1891ab51ceec736b0f9 |
|
BLAKE2b-256 | 5649c03ee3221d017506f84aece029340cbf91f564aec150655705d4f7f4a4e8 |
Hashes for llm_rs-0.2.7-cp39-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 673b088fada6a2074aaba52ccaf85424209b1d4c3549b9039874227bfb18b46b |
|
MD5 | 9b2d7f833ce91766c56450aca40d5b5a |
|
BLAKE2b-256 | 3dadf1bd13854fee6fa740bf9d05371a131114823bd6b63780a2ccb628fe769d |
Hashes for llm_rs-0.2.7-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3921bf6c3a87af0b3463c303a3514cbee3d4323f430d6afcf318c5b5d3dad041 |
|
MD5 | 3b0e876fab74e59ea2a99e2adc6ea4be |
|
BLAKE2b-256 | fbac4f01f8fbae819869a0ecbd619acd137b9bd58760b6a694961c2154dace7b |
Hashes for llm_rs-0.2.7-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d06eef132cfff62d60388609da5e1decd6345e1930540a23dbdd046c1937eb15 |
|
MD5 | 4cada3aa345db89f0a63a1fc8910b184 |
|
BLAKE2b-256 | dda6207d049a09b7975fa271fe10e3b06a81aa851d788b993db445d27243434b |
Hashes for llm_rs-0.2.7-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b10ab67a5884aa31a1eea523e329369b4779f3a346c3e1d7ce0a3c8788ba15bb |
|
MD5 | cd66b3a105010709a497cfe64624635c |
|
BLAKE2b-256 | ea7ff08b6a7c52c82d4ec90b6e0431cace9bfe588161c883b1e9e66417d3c5c0 |
Hashes for llm_rs-0.2.7-cp38-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 59185b9bb6b57706645f0219e78db01447a65fe726f77a52f4a7911c022c3e7f |
|
MD5 | 977f0b6d4863fea9ed39713df0260a69 |
|
BLAKE2b-256 | 73ed2ace250dc971e83b320e61d9e248715a537067be9b0219b10c0c97d7bb58 |
Hashes for llm_rs-0.2.7-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dab4d8604e73f61a5bad2625b74e72070f0c80d9d8bf27a1221cc3893d2de140 |
|
MD5 | 05de4f4982718d8293dae6230867f92e |
|
BLAKE2b-256 | 1694fb1ee1759634dc8f171bf76cdaaad04c19dc5dd27fa415afb7cb28c45c93 |
Hashes for llm_rs-0.2.7-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 44b27704e154d822ee638116b529ea77a445392973a7bda95a44c4929bb4f98b |
|
MD5 | 4382f7ef70aa5baee256f4e5c133baaf |
|
BLAKE2b-256 | 478e942ad8d2f09b92939c7161628c20109f9d6ff72ef1d5013fef02ff80aabd |
Hashes for llm_rs-0.2.7-cp37-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 260613e0fd10a8fc60783c513d930aa65d131395d72da5ed3f999e58a60258cc |
|
MD5 | 8ef2ed2739babeb637b8d30d1a6e458a |
|
BLAKE2b-256 | f15decae5ced55dfe828af861588b07c3d8755d5af2739efbca55efc95d74033 |
Hashes for llm_rs-0.2.7-cp37-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 58e834cb56692d34e9975dcd26e8b3006f50e2179abbc2cdda4c64bd5c8a71c7 |
|
MD5 | c5f204073001ed14749b0c5f05b91e42 |
|
BLAKE2b-256 | e31d39632d71f0a43c2727c5753d34003184af8fa63e8686f1714790e825b5a5 |
Hashes for llm_rs-0.2.7-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1709caa717bccab8814fbdd514ef73d029f0b8bbf894cc8d0de015abee8f71bd |
|
MD5 | 18c5a8ea979dbfd539bf24a1ef355558 |
|
BLAKE2b-256 | b8c267500a52d56349b330b5f1a577a4d7386b245cc0a42e324dd0713d317f05 |
Hashes for llm_rs-0.2.7-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 15df8a8f64593e6f1974b4659c48f57a1eb6716b87797b02f566f6c4c7a10bf7 |
|
MD5 | f17bb0421339e03b686ba509a59d357b |
|
BLAKE2b-256 | 4d430e7d2a352bff76446bfa5919388fbfecf88257221404f7b160da2cb85271 |