Unofficial python bindings for llm-rs. 🐍❤️🦀
Project description
llm-rs-python: Python Bindings for Rust's llm Library
Welcome to llm-rs
, an unofficial Python interface for the Rust-based llm library, made possible through PyO3. Our package combines the convenience of Python with the performance of Rust to offer an efficient tool for your machine learning projects. 🐍❤️🦀
With llm-rs
, you can operate a variety of Large Language Models (LLMs) including LLama and GPT-NeoX directly on your CPU.
For a detailed overview of all the supported architectures, visit the llm project page.
Integrations:
Installation
Simply install it via pip: pip install llm-rs
Usage
Running local GGML models:
Models can be loaded via the AutoModel
interface.
from llm_rs import AutoModel, KnownModels
#load the model
model = AutoModel.from_pretrained("path/to/model.bin",model_type=KnownModels.Llama)
#generate
print(model.generate("The meaning of life is"))
Streaming Text
Text can be yielded from a generator via the stream
function:
from llm_rs import AutoModel, KnownModels
#load the model
model = AutoModel.from_pretrained("path/to/model.bin",model_type=KnownModels.Llama)
#generate
for token in model.stream("The meaning of life is"):
print(token)
Running GGML models from the Hugging Face Hub
GGML converted models can be directly downloaded and run from the hub.
from llm_rs import AutoModel
model = AutoModel.from_pretrained("rustformers/mpt-7b-ggml",model_file="mpt-7b-q4_0-ggjt.bin")
If there are multiple models in a repo the model_file
has to be specified.
If you want to load repositories which were not created throught this library, you have to specify the model_type
parameter as the metadata files needed to infer the architecture are missing.
Running Pytorch Transfomer models from the Hugging Face Hub
llm-rs
supports automatic conversion of all supported transformer architectures on the Huggingface Hub.
To run covnersions additional dependencies are needed which can be installed via pip install llm-rs[convert]
.
The models can then be loaded and automatically converted via the from_pretrained
function.
from llm_rs import AutoModel
model = AutoModel.from_pretrained("mosaicml/mpt-7b")
Convert Huggingface Hub Models
The following example shows how a Pythia model can be covnverted, quantized and run.
from llm_rs.convert import AutoConverter
from llm_rs import AutoModel, AutoQuantizer
import sys
#define the model which should be converted and an output directory
export_directory = "path/to/directory"
base_model = "EleutherAI/pythia-410m"
#convert the model
converted_model = AutoConverter.convert(base_model, export_directory)
#quantize the model (this step is optional)
quantized_model = AutoQuantizer.quantize(converted_model)
#load the quantized model
model = AutoModel.load(quantized_model,verbose=True)
#generate text
def callback(text):
print(text,end="")
sys.stdout.flush()
model.generate("The meaning of life is",callback=callback)
🦜️🔗 LangChain Usage
Utilizing llm-rs-python
through langchain requires additional dependencies. You can install these using pip install llm-rs[langchain]
. Once installed, you gain access to the RustformersLLM
model through the llm_rs.langchain
module. This particular model offers features for text generation and embeddings.
Consider the example below, demonstrating a straightforward LLMchain implementation with MPT-Instruct:
from llm_rs.langchain import RustformersLLM
from langchain import PromptTemplate
from langchain.chains import LLMChain
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
template="""Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
{instruction}
### Response:
Answer:"""
prompt = PromptTemplate(input_variables=["instruction"],template=template,)
llm = RustformersLLM(model_path_or_repo_id="rustformers/mpt-7b-ggml",model_file="mpt-7b-instruct-q5_1-ggjt.bin",callbacks=[StreamingStdOutCallbackHandler()])
chain = LLMChain(llm=llm, prompt=prompt)
chain.run("Write a short post congratulating rustformers on their new release of their langchain integration.")
🌾🔱 Haystack Usage
Utilizing llm-rs-python
through haystack requires additional dependencies. You can install these using pip install llm-rs[haystack]
. Once installed, you gain access to the RustformersInvocationLayer
model through the llm_rs.haystack
module. This particular model offers features for text generation.
Consider the example below, demonstrating a straightforward Haystack-Pipeline implementation with OpenLLama-3B:
from haystack.nodes import PromptNode, PromptModel
from llm_rs.haystack import RustformersInvocationLayer
model = PromptModel("rustformers/open-llama-ggml",
max_length=1024,
invocation_layer_class=RustformersInvocationLayer,
model_kwargs={"model_file":"open_llama_3b-q5_1-ggjt.bin"})
pn = PromptNode(
model,
max_length=1024
)
pn("Write me a short story about a lama riding a crab.",stream=True)
Documentation
For in-depth information on customizing the loading and generation processes, refer to our detailed documentation.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for llm_rs-0.2.11-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 868142f2b5472f64d70afef8fa2f64ac18546b1d54f6bba9dbdec3a242335daa |
|
MD5 | eb2090e07017345541e7d4f733e09952 |
|
BLAKE2b-256 | f9732450f9d34b8a6370b309078f7727617279c1b4212e849bd30dd09d90e5d2 |
Hashes for llm_rs-0.2.11-pp310-pypy310_pp73-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e243f6dc37cb2acc8e4425ecdc1c43237eff5c55e4c1bcc44c4c8c43dd2774d2 |
|
MD5 | f1d7e0547cf952a0a444d8993b863ac8 |
|
BLAKE2b-256 | d362e9ff61255e570dca23ea0a6e7f5e29163b38b956cc62040630c7162a1c8d |
Hashes for llm_rs-0.2.11-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 074bbaf5efbcad57de17c602cb87b2d22989288e7bc4056944441ad843afe597 |
|
MD5 | 7ccca314f960dcbccf6c3f7165657e70 |
|
BLAKE2b-256 | b942b513fe5bc06a40bfbabb57489b8d8dd86c9d3e4347d9bd74b0fe517528bd |
Hashes for llm_rs-0.2.11-pp39-pypy39_pp73-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cabc01f370d10255803cd45f6267db4acd5358b0290e935145add14928dd3c87 |
|
MD5 | e751c49a671cc5da400643eb2ae7f78b |
|
BLAKE2b-256 | 3bbc6ca3595f621563084d8b1e2e5b5b14ea3ae24416d795b67f950e22e32bc1 |
Hashes for llm_rs-0.2.11-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c5d35f43b0ce1a0c9fd374fa9913805dcbee6200b50f900ee453fa7fdbc6932a |
|
MD5 | cc9f98f7d3266328c8d056377bfd7c71 |
|
BLAKE2b-256 | 7ae54b92928b74cba1a7324cbc0a5859d0730127ea6bf0210efefe1213c2d32d |
Hashes for llm_rs-0.2.11-pp38-pypy38_pp73-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 94e473b652ff6a1345b15afbdb242b66e02a4f967e00ec3100c082d6809b8a6a |
|
MD5 | a61df20c41d18e45be1c2a06afd53ac5 |
|
BLAKE2b-256 | a7602f2b64315d65d10c656142080e8dcdaed6b48ff7a8eb518334b08b0f0707 |
Hashes for llm_rs-0.2.11-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 302edfdcf4c41feb3b8b6bb0f4949596050fc8b856d1e5639f0ff9981805070a |
|
MD5 | 85ee598ed487ca5ad622795436bd62e2 |
|
BLAKE2b-256 | 83a884f338767895770c819709b3efddc20b29c980dcad4a84406cbbbbde92c6 |
Hashes for llm_rs-0.2.11-pp37-pypy37_pp73-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 706b8b2ce0d45680eb3d1e8d25af922bcd440ce41c3d9fafff46c294b3a11f6a |
|
MD5 | aaccf13f44f1a811139e1c242c51feff |
|
BLAKE2b-256 | 23b40b9272d353f00461761692c65d14f96c3e1c3489968cb61af1ef683c7fb3 |
Hashes for llm_rs-0.2.11-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a90f59b0982343365b2fc4ae0f83a517d25ee66db8e9958e72b0626fdb9f771d |
|
MD5 | 2abf3ea62af17d9c3a12c940c04585b4 |
|
BLAKE2b-256 | dbb1c4fe614667b0ca12f561bd922a4a58607ddb2a7ed3bc194437453712bac9 |
Hashes for llm_rs-0.2.11-cp312-cp312-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 369ccdf0a81c39f503355329841df50dc328ab09346fa50dd5f9b5c739105d73 |
|
MD5 | 6d70c732bf116d16811c5a6f031e98ac |
|
BLAKE2b-256 | 8916af9fabb1040ba2420804ca9591cc0454508e0a9218ab3ced0fee751abcf4 |
Hashes for llm_rs-0.2.11-cp311-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d8476744972a3a0c867abf74084215574a68bf7399931bbd2c6aec52033a8947 |
|
MD5 | 1bb12e37eb7041f2913763f61b0f8ecc |
|
BLAKE2b-256 | 4b5ccc410bf7b3ff5e6c1f82c61d3cedbecd33a2c60fad3c933421eee60a6a0d |
Hashes for llm_rs-0.2.11-cp311-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 49890f4dc3a07a81703f92ea65833820ee3e92fbae397744118fdfe64085fd8f |
|
MD5 | 7e83b936bf28f8b7836b1fb21c332377 |
|
BLAKE2b-256 | 5447da77a191877f7416c41a8ab6b337d6c2ad527ee1bb1ca9347542e4911393 |
Hashes for llm_rs-0.2.11-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bbe2d0ac5c3eaeeeccfd2c9066c804f076cecb1d8313e001aec89bbbe34c8495 |
|
MD5 | 0de4b2fe9a1c9931e608c9a4161c1b03 |
|
BLAKE2b-256 | 0b0013d7a19f6bae59655512fd579b6b4f7745da04159c3f5faa5c4a7489e70e |
Hashes for llm_rs-0.2.11-cp311-cp311-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2fcb72c780fc0febd5a853235d73d45635337d24a31621ca42f1b0f1d4218782 |
|
MD5 | 53caac21a895f9cc7f6ecfe545b61f20 |
|
BLAKE2b-256 | a4880359d3041f9a6d2eb89c2d766f2181588ccec22dd7187b1486e482860b4b |
Hashes for llm_rs-0.2.11-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | aa1973ef8edaf1b911b6f460ea4be6edbbf9b16afc33cafc263c864280cca49f |
|
MD5 | 961b3b1c46a37fe592f4ee35b03a4b4a |
|
BLAKE2b-256 | 6e5b0a3b3ac29b9e8059636504b7111f2781ad3e5d9e9a0b855443a2b3029bd4 |
Hashes for llm_rs-0.2.11-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0450d72a0a4a9ef9b1c6804707cf89a6c960e2d41627e2161e528305b0287371 |
|
MD5 | a5a64909807840dd9d4f0a0e40cf96a6 |
|
BLAKE2b-256 | fd8e6cf5e7badda5a04d04d271ed3d833f304e1cce8f5a8d4db100193dadda5e |
Hashes for llm_rs-0.2.11-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 12c050f5fd9238a216ad83f17c8310fd63664884fa66c4a33f71cff95ad7c56d |
|
MD5 | 2ab7c50a0db193f613a746da0398b5bc |
|
BLAKE2b-256 | e22b465e47b2e4e0646ab1eea6abd6a4d597bbce03337b0f70f005f86b503ad6 |
Hashes for llm_rs-0.2.11-cp310-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ca4b1b246c8edcf9f6ec65fbf8a69c118c8960cd29da77b10440e6fc055a86d8 |
|
MD5 | 4972db2f447bcfa90f79e5d08c65c06b |
|
BLAKE2b-256 | 5817b46ea9d6acf6f3057a140813b53cb4b1263fd6bea386617ab615adf7c5c8 |
Hashes for llm_rs-0.2.11-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ff7d1fd12c178f4a5be316f66c47649cfbd9eb3d13fb05ba58ed735846c3275a |
|
MD5 | 9b6b1ca394d51e0e9030c7c6a0ef3ceb |
|
BLAKE2b-256 | 7c9a2daddbb08760d923b821062b154c6b04e723dfb1c8ca1128b1b667111f90 |
Hashes for llm_rs-0.2.11-cp310-cp310-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6ab9091bbf6b9b291d0018c5106dabd14fc1b6833b9e40bf37ae4f941d435b0e |
|
MD5 | 4e858f278219dfa45a1e52bec87edf85 |
|
BLAKE2b-256 | 3f92d8b2459ca914fb273692e3aa604be0678ae69db844272c21f2bca93a617a |
Hashes for llm_rs-0.2.11-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 11a64301abc7b9b435e413a9efe013c9280050ab2d77e7ffd41e6941ba0ab881 |
|
MD5 | e28559289f8db109cf27454d1d8363e0 |
|
BLAKE2b-256 | f225f94909a37f36a5276c26e9166b9da4423fd2982df06043b55ba5c2e3c604 |
Hashes for llm_rs-0.2.11-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 23dbd6db293a48f4c3d57166c12581ea20c84a85b3525eae3075fbb8d4eed4c1 |
|
MD5 | 31a1bacd95ee05d49f0ea77cb625759d |
|
BLAKE2b-256 | ca5d49f000842124142328c959fbb9960f1b6a6655f5339aab56a4567e6d4b7e |
Hashes for llm_rs-0.2.11-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6adbf4c36534ed4ddc507435be0cf5b06bb7797ceb60886205248d618039803c |
|
MD5 | 93a1c34577f5923f7864ce60646db4d9 |
|
BLAKE2b-256 | 0e2b1d17f9f5989da8fcb6c76f7c4c759ec6f9f38397b4885efa307ea76a3d4e |
Hashes for llm_rs-0.2.11-cp39-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 972da5de03e30b4aefc6b21eb40052a700806fa816461cf60c57d22740a71db5 |
|
MD5 | 5c01081905fac36a85ec96d2dadad636 |
|
BLAKE2b-256 | 0e0572cbdb4f491a5ee4630f03e9594dd361fd49a276674f8b2cb29ee34bacd1 |
Hashes for llm_rs-0.2.11-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a92d9106ad0b5dde87aac8f6b469e0f6c2dd8e64ab71363f6636b36293b3132c |
|
MD5 | c55b3c4ed56e964fb2c6b6dd178aad65 |
|
BLAKE2b-256 | c5179a699d2f8fb14d749c9daa0d2198360c23056cc66c6ca1c4ecfab9dee616 |
Hashes for llm_rs-0.2.11-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 889ee5f12e5e2e359841e4da66549c7cec574941304b1f993cd53b56d54d1279 |
|
MD5 | c153fca92552299f2ae70c815bf26930 |
|
BLAKE2b-256 | f1d7bc410800bd63d7d5a6ee3da1b469d78f5cf53fd4fecbd8783eedc0a73b69 |
Hashes for llm_rs-0.2.11-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 932e850f81a357008e2b820d9fc147c88a5bb4068faa29f3bd939e218750f3cc |
|
MD5 | 59bab324f023e1b54f9ed100bb1b943f |
|
BLAKE2b-256 | 02f332fde990c96e92ac50fd6dbc13de177e25ac353a18e1538670b084ef3183 |
Hashes for llm_rs-0.2.11-cp38-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b2c5a1deb5955ecd32f9580d339541e3f11d0790567e9bf4ae392a27786d353e |
|
MD5 | 91504dee64d880648f34dc6b018a7222 |
|
BLAKE2b-256 | 58682617ed5f95f412372db1bde1009b4b7569c12f03fc04c8be3bc9bd50629e |
Hashes for llm_rs-0.2.11-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 44d4cc777020668f498f2a8e6f98d235d3ce3ff305c9d324eec525ac5da23ceb |
|
MD5 | 94ee69e303c5ae1121f5331f5e5c1701 |
|
BLAKE2b-256 | 08a5a3d33f346a2d99d9a01e7a9da97420892c8393ca2efbbbf92ae20f948e53 |
Hashes for llm_rs-0.2.11-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f0b3754f2273cb8057c66a9e03281c562554f9752b211029b1bf2759f771e70e |
|
MD5 | 2c2108ba5178cc9153933e4efc1e2ec5 |
|
BLAKE2b-256 | 812c5b0a3461d857e7e20975cfa359c0e9ea9d77290173432e270990cbaac956 |
Hashes for llm_rs-0.2.11-cp37-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | aaf7aa00990e4b035aebcfbc03d12f7fafeaea63fccab70f123690be339aab45 |
|
MD5 | 2f8c7d01b8466e40c510f30dcf507715 |
|
BLAKE2b-256 | 19b47079d3c2646804e3fda7e8c083f0aab10675de5d0f43b2c590d41a61bfc8 |
Hashes for llm_rs-0.2.11-cp37-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 52270829144881fc31143607211ac14af0b5ac1526c277564a59f846a50941b8 |
|
MD5 | f1ba0c2ceb14df7454cf921a4804b4e3 |
|
BLAKE2b-256 | 2dfca1915cc009f8625a448b1093e7d4d0c3f30d97207dacf14cee13cad056cb |
Hashes for llm_rs-0.2.11-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 12171740b47f23f11208623f528d2a6e517172c883dc69ee09f294cc278529ac |
|
MD5 | 4964145e0ab82590e0d2a10d161ce068 |
|
BLAKE2b-256 | 673d006c32db7b6570f2fe89255b9ce94b8eef67b4aabf2830de31b461076ca8 |
Hashes for llm_rs-0.2.11-cp37-cp37m-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8401450f9ac1abca531b5f5c8f90ac5b43aad3c27821601ca63975ab30eda02b |
|
MD5 | 0fcc94020c6e61b1d8294436cc9fbc42 |
|
BLAKE2b-256 | b0f1a4ebe93060a94bee2d5b0bde45f309814d4b66e993180d40e6cad9647d93 |