Unofficial python bindings for llm-rs. 🐍❤️🦀
Project description
llm-rs-python: Python Bindings for Rust's llm Library
Welcome to llm-rs
, an unofficial Python interface for the Rust-based llm library, made possible through PyO3. Our package combines the convenience of Python with the performance of Rust to offer an efficient tool for your machine learning projects. 🐍❤️🦀
With llm-rs
, you can operate a variety of Large Language Models (LLMs) including LLama and GPT-NeoX directly on your CPU.
For a detailed overview of all the supported architectures, visit the llm project page.
Integrations:
- 🦜️🔗 LangChain
Installation
Simply install it via pip: pip install llm-rs
Usage
Running local GGML models:
Models can be loaded via the AutoModel
interface.
from llm_rs import AutoModel, KnownModels
#load the model
model = AutoModel.from_pretrained("path/to/model.bin",model_type=KnownModels.Llama)
#generate
print(model.generate("The meaning of life is"))
Streaming Text
Text can be yielded from a generator via the stream
function:
from llm_rs import AutoModel, KnownModels
#load the model
model = AutoModel.from_pretrained("path/to/model.bin",model_type=KnownModels.Llama)
#generate
for token in model.stream("The meaning of life is"):
print(token)
Running GGML models from the Hugging Face Hub
GGML converted models can be directly downloaded and run from the hub.
from llm_rs import AutoModel
model = AutoModel.from_pretrained("rustformers/mpt-7b-ggml",model_file="mpt-7b-q4_0-ggjt.bin")
If there are multiple models in a repo the model_file
has to be specified.
If you want to load repositories which were not created throught this library, you have to specify the model_type
parameter as the metadata files needed to infer the architecture are missing.
Running Pytorch Transfomer models from the Hugging Face Hub
llm-rs
supports automatic conversion of all supported transformer architectures on the Huggingface Hub.
To run covnersions additional dependencies are needed which can be installed via pip install llm-rs[convert]
.
The models can then be loaded and automatically converted via the from_pretrained
function.
from llm_rs import AutoModel
model = AutoModel.from_pretrained("mosaicml/mpt-7b")
Convert Huggingface Hub Models
The following example shows how a Pythia model can be covnverted, quantized and run.
from llm_rs.convert import AutoConverter
from llm_rs import AutoModel, AutoQuantizer
import sys
#define the model which should be converted and an output directory
export_directory = "path/to/directory"
base_model = "EleutherAI/pythia-410m"
#convert the model
converted_model = AutoConverter.convert(base_model, export_directory)
#quantize the model (this step is optional)
quantized_model = AutoQuantizer.quantize(converted_model)
#load the quantized model
model = AutoModel.load(quantized_model,verbose=True)
#generate text
def callback(text):
print(text,end="")
sys.stdout.flush()
model.generate("The meaning of life is",callback=callback)
🦜️🔗 LangChain Usage
Utilizing llm-rs-python
through langchain requires additional dependencies. You can install these using pip install llm-rs[langchain]
. Once installed, you gain access to the RustformersLLM
model through the llm_rs.langchain
module. This particular model offers features for text generation and embeddings.
Consider the example below, demonstrating a straightforward LLMchain implementation with MPT-Instruct:
from llm_rs.langchain import RustformersLLM
from langchain import PromptTemplate
from langchain.chains import LLMChain
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
template="""Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
{instruction}
### Response:
Answer:"""
prompt = PromptTemplate(input_variables=["instruction"],template=template,)
llm = RustformersLLM(model_path_or_repo_id="rustformers/mpt-7b-ggml",model_file="mpt-7b-instruct-q5_1-ggjt.bin",callbacks=[StreamingStdOutCallbackHandler()])
chain = LLMChain(llm=llm, prompt=prompt)
chain.run("Write a short post congratulating rustformers on their new release of their langchain integration.")
Documentation
For in-depth information on customizing the loading and generation processes, refer to our detailed documentation.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for llm_rs-0.2.10-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 613e67aca555fe3739dd3118f5e864d282bde8b215958beea53ffa5d7fa6f8da |
|
MD5 | 35294f62cdef89ede55bb227e43f4e74 |
|
BLAKE2b-256 | 65246d1502c000afce4e1e00770c01f509f51aa64000d1f2a6ee4e3d22719bf8 |
Hashes for llm_rs-0.2.10-pp39-pypy39_pp73-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 512a2253123f344f3d570988c1a6e761488a9e73c0b36ae343c318c50fd72ee5 |
|
MD5 | 861458518683b98ad4facfa64b067c1b |
|
BLAKE2b-256 | 27ea6cc010d25e439854e4bceb55ee69ff8403c25b31ffbffc9edd8cfaa69aa1 |
Hashes for llm_rs-0.2.10-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 68d01287e1350f9a4a927533d3d6362fa1cf37752e88a276fc99b27c3c90314a |
|
MD5 | 323d17467fbeeca59f09a52b81261d23 |
|
BLAKE2b-256 | 3f97809826efb259402bca002e0e68dc060be2a524c4b09cac6065533f18b38d |
Hashes for llm_rs-0.2.10-pp38-pypy38_pp73-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6d6d9e08b154766a0f3db08709e6ddc4bafafb427c1912ca3e2c8eca4d3ff0d1 |
|
MD5 | 707453b01549cdaff618cc2c6ae5af60 |
|
BLAKE2b-256 | 0144f3b854be336b4a39adedd9cea8c9c40c62a93103b0564ac0c3875a67bb57 |
Hashes for llm_rs-0.2.10-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4c59d2cc5dc118a366062b9d4f2af163ee2511f6b73e464179d5457c398ae458 |
|
MD5 | 7b71193b009cadfaf338fd5ec7d910f4 |
|
BLAKE2b-256 | 3c1208761020181b52a1fbf07f61a577b7eecc122fcc5f1af19edce4cd7dfe84 |
Hashes for llm_rs-0.2.10-pp37-pypy37_pp73-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b3455ad5aa2c5ab219c4170da7c68a81c294ccefa3c34ff498440d7c6372611f |
|
MD5 | 825db06d16d5233ee567dd16d7753f13 |
|
BLAKE2b-256 | c3952e3131723f75cc1ddb6590cda0d98439696ec216ce77681f9a45f577c8b5 |
Hashes for llm_rs-0.2.10-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 01ad42a8e840a2985270ff3eede1e667397a93689e69390de07409ba4444aba3 |
|
MD5 | 8bfe05124b3a6ce80c47293e8f4f7abf |
|
BLAKE2b-256 | 55a4b37e7876f1d57a561215c54a120780aff3cacb0eca90c0fca313d2f2a1f2 |
Hashes for llm_rs-0.2.10-cp312-cp312-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1490933f6c439a767d6ba824dc66df42f4752ff3bccf98bed33351e60949691c |
|
MD5 | b15dffa55dbfc4acb76985c1392c86b3 |
|
BLAKE2b-256 | a95a252acb427bf12d5205c40b7b774f3019ded0646fe52852df70266aeff605 |
Hashes for llm_rs-0.2.10-cp311-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7c34c2846e5d839505f25769ac68204630a904c32a6ed7e0142d222d754dae8e |
|
MD5 | 143014c30267f212084b20637cb831fc |
|
BLAKE2b-256 | 13a709faf6c7f9db8423864ad7485d6f55c08e0571d8ddd72df83e80e47925cf |
Hashes for llm_rs-0.2.10-cp311-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1ff5a454bc1257597db1edd108c37854b68549c2c78bf5bec4db576abab277df |
|
MD5 | 2df75fe2b6fc96589e685e663beed726 |
|
BLAKE2b-256 | 9c79a86e1a96963da7e91970c04656cb79cf33f0ec8c9892e43d76da48ba1362 |
Hashes for llm_rs-0.2.10-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5443fbe43f7fea00f4a54f2129b569bd92093f73b515082c0c77ee974627f1a8 |
|
MD5 | a3c1c165c68ad1ec4a342196bd52101f |
|
BLAKE2b-256 | 76167e86e7399ec12ed33a0a9f5e433978ef906a7cc07fe5b4c642f0211ada30 |
Hashes for llm_rs-0.2.10-cp311-cp311-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | aa07e385fa9ecbdbd5f65155d9cc95d711e29807213651772740ffb66163a8f7 |
|
MD5 | 8683861f9d87009f69ba88333c3a72ff |
|
BLAKE2b-256 | 2a7da1b45dcc0c36426762f661a1b8e1b510d631edb3735c755afa57604c1815 |
Hashes for llm_rs-0.2.10-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2545c1d736a4da9438856913ba185256a3cb87b42059e8757363b8113be63bf0 |
|
MD5 | 60b8e8143883f476e0ec2857904d0f23 |
|
BLAKE2b-256 | ff9d76611437758758d28c6c28e57a1619d914dfb8cc689e14e4fad0968f78ce |
Hashes for llm_rs-0.2.10-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dc83bbae2abdff31cdea1c6279ddfa9f0c7656648b35469369f72e50a8757e1d |
|
MD5 | 78f4586d3a5fe11640e7976993f296f9 |
|
BLAKE2b-256 | f06262fec4798f9e54193f0c0c820c3568b82db83435466d6b950ccae85545b0 |
Hashes for llm_rs-0.2.10-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e929d6c450f92f7687a4f916a6a3fa84b2efe2e37a0a58b4ac3653284ae22d32 |
|
MD5 | bf998fe48398ec1fcbf1ed249b8829c9 |
|
BLAKE2b-256 | a5ae93804e2f79e66abf93fa19fc8de60fa4f35e7c4ba1cc559b902373bcd9e0 |
Hashes for llm_rs-0.2.10-cp310-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 385f95934390e7689d0057a941b3a962f210013405a1f7996ac323ba5c9bb2cc |
|
MD5 | be682227586761d80a6a86465cf94735 |
|
BLAKE2b-256 | 70ae52da9b1c69521b809e8d39cfbb120e576d65cc41f40adcf61c550337acb7 |
Hashes for llm_rs-0.2.10-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a8d81801cd6a2305ed437159805edc3c3d78dd5e44d66f8adfd157ab814a416e |
|
MD5 | 076115c0f99a5ff5423161f7634a4512 |
|
BLAKE2b-256 | d87faf59d2ffd4296010655e828e300940e1411c5399e79b6df297fdf742ab4f |
Hashes for llm_rs-0.2.10-cp310-cp310-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f1db0b57b8a3304a5253cfa21dbdcf4509333b2106ecdf23c2c23af99195bcf9 |
|
MD5 | 52effa89a6aad34adbee52341d94a537 |
|
BLAKE2b-256 | 791084611af05c9b310862e9554473274e1c48ebe19e554029ee9f265ce60db6 |
Hashes for llm_rs-0.2.10-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 70c3d282207ba4f54cbb5193dd14afbaf2174c0c8263ec95151c477e262532b0 |
|
MD5 | 9e57845a7403a05160f746f14f0d64d6 |
|
BLAKE2b-256 | b42498505d8c8ab7cf83117e2752e792d194303069c319671f60f1a6a571c5b5 |
Hashes for llm_rs-0.2.10-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 853b37a6d122239342168b1c5b44e255ae77b42bf2a16d20bf39929d4893f78c |
|
MD5 | 7824b6210df623550c30eec13cb97892 |
|
BLAKE2b-256 | 248d46f4e7c76df26b5045f68dee67a257f534e677245191aea0a76f23a9acb4 |
Hashes for llm_rs-0.2.10-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8e261893f6bdf72a071191ad7a66725c5530c72c87aecca26d163d3a0fe4d621 |
|
MD5 | ca56fe268831f2e34b536df9c70061a7 |
|
BLAKE2b-256 | bf4e0e6e570913fbea2637ce09896ef107e34608c9efd3a1c0fe05534055ba5d |
Hashes for llm_rs-0.2.10-cp39-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 309b6d542927d9c10d58acc7d16fc4b9ced34ba9c91f558d87883507d5072366 |
|
MD5 | 4b456f8fe386fcb8de82cb9f4b6e63b2 |
|
BLAKE2b-256 | 9b4b3a8f7516e918e7a957dddf9144e1a51c2955b4ceebe27e2ddb475f1588ae |
Hashes for llm_rs-0.2.10-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dfa245b0cfa09e8ba5e949718b803f66c1dfe6ee5ea54951417f4f1cd7db1af6 |
|
MD5 | bbb4bf7aabf96a53b24beeba420d83e5 |
|
BLAKE2b-256 | 60b4172ced0cefd95c621d573365ad39d86c0b2732194048044644d387745c57 |
Hashes for llm_rs-0.2.10-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 24f61c2b4bdc4cc6d5214ecfba06881de4ce8c795cf865165451f3f0558d3ef1 |
|
MD5 | caf0408b537b83c080d3943eb95c1b1d |
|
BLAKE2b-256 | 6b2aace6c83a2267aa447e07517d25268c58593e4ff8bc89b13facf1c9876469 |
Hashes for llm_rs-0.2.10-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 29959cd1ee76d626406c39c0415de58eb6c78cacac69eb79d67461bb4154ad19 |
|
MD5 | 825f35e023e1032a817de7bc6c41801f |
|
BLAKE2b-256 | 004b617be1d5e9a172eedb9be1a6a42df57a84d534dd3adbe1580dd3b4259483 |
Hashes for llm_rs-0.2.10-cp38-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3c79e89ca6c0a7e6d25630fd6b02e065d8453558cfd3797dc8e95bc110e524d7 |
|
MD5 | 760a76664125ef1869eee80a92237a28 |
|
BLAKE2b-256 | c26f77c3ebd419ed21c982f5959d8a877953f76a0b59f323a1cf350c9f06c70f |
Hashes for llm_rs-0.2.10-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1b94ab5d83e245a2a11ee5fc61a048185017a46c6f7e226477faf29ce504f829 |
|
MD5 | 05cdce6052decaf38b29115f08ab032f |
|
BLAKE2b-256 | eb6a4eaf38430039be1e2a5cc1100d287fcea4bd43742b4ddcf46703006c7eeb |
Hashes for llm_rs-0.2.10-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5a5f17c78e5de8bcda903f3d3776843118165b1c5bc5597e9b83d786320de8b1 |
|
MD5 | 2a3251e676c9144be05e65a59276105e |
|
BLAKE2b-256 | e60c78dcf21b7c99a519c12f0bc038fcc55465da5652aefd51ce8944ea4cb014 |
Hashes for llm_rs-0.2.10-cp37-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 54ffdd513183833f0111993d5eb0593a9e4c4bcdd1ecd905bed71d2e4210bc73 |
|
MD5 | 601ec90cf9066ece0f3fe864c4f0150e |
|
BLAKE2b-256 | 919bf3bce7af938a1f4cb183c0ecf84435814238bd48d87d5f7cc9bbaf9be2fe |
Hashes for llm_rs-0.2.10-cp37-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6cc848410d7f1cdd34d4c6b5ec11d40d215ec9db49bb2b3e257da0483bd4a1b8 |
|
MD5 | dd9ce3faf8960595aaa6f6edccdf0d84 |
|
BLAKE2b-256 | 2923d06817e0397f4711490abddd4c8966dfa166010a3760324294f14e6d24a8 |
Hashes for llm_rs-0.2.10-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 06dbb8b5747504490714bf92435d8e2ad61d91b16420a2e6e77ec93c03abd83a |
|
MD5 | fbfa5c9546168069e5042cb2dd69ef11 |
|
BLAKE2b-256 | dc52c3b45b1b5c103255ace9f376bfcda72487f8a1529c687019bc4c6c18f333 |
Hashes for llm_rs-0.2.10-cp37-cp37m-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 256861dbab958d81977aa4b55e64d9febf89323621cafcd64dd9bd91f3a5c61e |
|
MD5 | 5bb9c655fe27dc23ebc094e6fd020f28 |
|
BLAKE2b-256 | c86aa0e1b3520cf06d836bf5017150ee30ebe85ead059ee420534b0e04e46b0e |