Unofficial python bindings for llm-rs. 🐍❤️🦀
Project description
llm-rs-python: Python Bindings for Rust's llm Library
Welcome to llm-rs
, an unofficial Python interface for the Rust-based llm library, made possible through PyO3. Our package combines the convenience of Python with the performance of Rust to offer an efficient tool for your machine learning projects. 🐍❤️🦀
With llm-rs
, you can operate a variety of Large Language Models (LLMs) including LLama and GPT-NeoX directly on your CPU.
For a detailed overview of all the supported architectures, visit the llm project page.
Installation
Simply install it via pip: pip install llm-rs
Usage
Running local GGML models:
Models can be loaded via the AutoModel
interface.
from llm_rs import AutoModel, KnownModels
#load the model
model = AutoModel.from_pretrained("path/to/model.bin",model_type=KnownModels.Llama)
#generate
print(model.generate("The meaning of life is"))
Streaming Text
Text can be yielded from a generator via the stream
function:
from llm_rs import AutoModel, KnownModels
#load the model
model = AutoModel.from_pretrained("path/to/model.bin",model_type=KnownModels.Llama)
#generate
for token in model.stream("The meaning of life is"):
print(token)
Running GGML models from the Hugging Face Hub
GGML converted models can be directly downloaded and run from the hub.
from llm_rs import AutoModel
model = AutoModel.from_pretrained("rustformers/mpt-7b-ggml",model_file="mpt-7b-q4_0-ggjt.bin")
If there are multiple models in a repo the model_file
has to be specified.
If you want to load repositories which were not created throught this library, you have to specify the model_type
parameter as the metadata files needed to infer the architecture are missing.
Running Pytorch Transfomer models from the Hugging Face Hub
llm-rs
supports automatic conversion of all supported transformer architectures on the Huggingface Hub.
To run covnersions additional dependencies are needed which can be installed via pip install llm-rs[convert]
.
The models can then be loaded and automatically converted via the from_pretrained
function.
from llm_rs import AutoModel
model = AutoModel.from_pretrained("mosaicml/mpt-7b")
Convert Huggingface Hub Models
The following example shows how a Pythia model can be covnverted, quantized and run.
from llm_rs.convert import AutoConverter
from llm_rs import AutoModel, AutoQuantizer
import sys
#define the model which should be converted and an output directory
export_directory = "path/to/directory"
base_model = "EleutherAI/pythia-410m"
#convert the model
converted_model = AutoConverter.convert(base_model, export_directory)
#quantize the model (this step is optional)
quantized_model = AutoQuantizer.quantize(converted_model)
#load the quantized model
model = AutoModel.load(quantized_model,verbose=True)
#generate text
def callback(text):
print(text,end="")
sys.stdout.flush()
model.generate("The meaning of life is",callback=callback)
Documentation
For in-depth information on customizing the loading and generation processes, refer to our detailed documentation.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for llm_rs-0.2.9-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e1bdcf05189f7c72ba54e18e36bb2abcafc4e7f924e511226dde8c53b50d5de4 |
|
MD5 | 8050781873f62c2215e3312134d73e8b |
|
BLAKE2b-256 | 47614873d6c143fd0bddd205d639360a67ad096b3d87275fc1a4e5e348ab6c06 |
Hashes for llm_rs-0.2.9-pp39-pypy39_pp73-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a917643850469b6c68277fb78830b00accf9ec102268acb79d0182916c1f1973 |
|
MD5 | 87d60d1370cd46ae6c708d3567dd5be2 |
|
BLAKE2b-256 | 2b4b0d65140e18c88131ad09b11255636752347a98fd1851426899e2d32e11e8 |
Hashes for llm_rs-0.2.9-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 30f881cae61679d4750880df8cf49ce971035ca5581365de9ed67da0877b1030 |
|
MD5 | 69f78c535c4f8915c378d88c1d953686 |
|
BLAKE2b-256 | fdc86585aa77c30683e2589245ce8fa965dec1a7ba1fa2a284681b5e9020b57b |
Hashes for llm_rs-0.2.9-pp38-pypy38_pp73-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6704700bce21c7a2a24e1718327507179b15a53b1bc7006124d8457c3db9bcdc |
|
MD5 | 085de78a5d9a772ceea79966b89c7335 |
|
BLAKE2b-256 | 91c40caaebc3975ad292a5c4b427881ce56109b98c76e79ba5c5ed1f6a806531 |
Hashes for llm_rs-0.2.9-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b8772601a35a1289bb2f015afaed4487a8aaa391762f387921b9c47bf6518b32 |
|
MD5 | 213b57361f6d3dee09e3baf4e9b32db1 |
|
BLAKE2b-256 | a2a1c0e4c05dcb35f507dd666b27ebd3f6633dabddf60b55a583bfb8a8b97ff7 |
Hashes for llm_rs-0.2.9-pp37-pypy37_pp73-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d89a14e4162743df7946f32779674290c1452bae68eecba70602ec813d2e9109 |
|
MD5 | 92fa2e954a8b6dda97e1e2e94579f595 |
|
BLAKE2b-256 | c1d915481a1c083fe0f50ca29b2f4ad40a6c50b0f5476bc083c1c7262eeaee6a |
Hashes for llm_rs-0.2.9-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 92338e0fdb78e4c1e3c972e11f15ef98b7c8b18ffdd4cd5052962154652e0f6a |
|
MD5 | 13be6568537079448caad113f5942ba9 |
|
BLAKE2b-256 | b4254f574d042614a2abebfea926689d3c9d4b86067f0c6d1030e3a06304a3d5 |
Hashes for llm_rs-0.2.9-cp312-cp312-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ce4bdfe720ca33ec1c32d4b08d1841604901e05d361c8777ddb49eae40a63b27 |
|
MD5 | dca690e14c61c72f8037ec34f4368ecf |
|
BLAKE2b-256 | dba168c1839bcde638d371c849e92811ae572f8c5bf5e1b020160e1574aa2f4a |
Hashes for llm_rs-0.2.9-cp311-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d2dde2ad98f1ff177dfbd1f778471541baeb49f4ccaea390e174435c6686f43f |
|
MD5 | 5de7d7481a17e37f863b7de7943fc694 |
|
BLAKE2b-256 | d0efc27051d67cd0751ba3459bcf55f9b05d95a764fc508e1f7b713d5372f3f9 |
Hashes for llm_rs-0.2.9-cp311-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | af5e43cb0a59662bf348434eeecce6337eaccb969e7bfd6da7f235201943f7c0 |
|
MD5 | 7fe87c89654f0ddabfc114806f3f8787 |
|
BLAKE2b-256 | 26d241b4954cfac3fa7c754b54bc64cbc5e51e1628bb75001315b7c6c542e0ef |
Hashes for llm_rs-0.2.9-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 20f98aeb0aaedf5962e11b82ce0615b2e0f00d4d86dc3547a873fe2c8ef781e6 |
|
MD5 | 3214775c97d0216aa75a273bce09fa81 |
|
BLAKE2b-256 | 2547f8ebeaa08b941aa0bc509b395ce31a76db5b8198cb828c03f1139ec5258f |
Hashes for llm_rs-0.2.9-cp311-cp311-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a93b10fa3e6d3c6b87dd2c1d93854c7bf45fb6e29fa6c5297002b8b74a0749c5 |
|
MD5 | 12adf0761e637c6135acd709db93b633 |
|
BLAKE2b-256 | 3a520d8a3c8019eb4f73dacc08e8cef8aa522dffde04f945bc3ea30eb918b06d |
Hashes for llm_rs-0.2.9-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 04404ba5b595b93487a11a053581b52378103e46ca9030cd90c8137e5cd0f04b |
|
MD5 | d98fd679065bc6c928f02dbee17f1344 |
|
BLAKE2b-256 | ebdf545f2ee6863d31cda543ccf590decedaf741e4567bd05fa2e786fa27cf5b |
Hashes for llm_rs-0.2.9-cp310-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a3b8c259a4a8bc24ee20700cd9374d9e69926a69ab6c1543f70092a59a3df411 |
|
MD5 | cd18a249cfc848220a7f8f58ad13d66b |
|
BLAKE2b-256 | d03b490a522819089b86962e01ece23c1385c03348fabe12fbf899cf834585d8 |
Hashes for llm_rs-0.2.9-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 263009af33e29fee4cbf276948eae00cdbfdb6a30674b411983797d6a70bca41 |
|
MD5 | 5764880f4b29867a16068a2677c0c467 |
|
BLAKE2b-256 | 650801aadd438be75cb8f570feeda653a7562f02f610fe6500a8efb916efbf63 |
Hashes for llm_rs-0.2.9-cp310-cp310-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b41106fb60117ea258aeb3b02c9dddbedbe915a4a2526df7fdf4b7da58f7d6dc |
|
MD5 | 00554d5306d2d529529c1cb00ace42b2 |
|
BLAKE2b-256 | e2232c71d6db728c029edf24518806313b9e9fdb0d0e605a093d89c00febc3d0 |
Hashes for llm_rs-0.2.9-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1bb443de84d105fd553fca4c5fa01290d42dbd66a2958873160a46f5bfae539e |
|
MD5 | cb8f944ef2463d0670253f0ff509299e |
|
BLAKE2b-256 | fd70880804b03c6bcd685a9eea98942c3b50ca6d19884b04bacb21243afa65dc |
Hashes for llm_rs-0.2.9-cp39-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 07406aba93948eac8f5dbbeb762962d45a07ba7e70cd5f2ac3fd8efa76f3faf6 |
|
MD5 | 24861d3ea20e659d38b5edb63eb56ae1 |
|
BLAKE2b-256 | 91c15015d07ffa6c4450ed82054cb6af201eeda4d4200de20f3b481ce5e3c61b |
Hashes for llm_rs-0.2.9-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c3455a80d3b10ed6a5022c0574d0a552fce808558f04fb8abe035e83f3991b5b |
|
MD5 | 897aedd2776ddaede57ff54e882aa09a |
|
BLAKE2b-256 | 951457b963f5aea9aebac35f274601b217eeb42343905e05a47f9b43eca75376 |
Hashes for llm_rs-0.2.9-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 29ed73648876c3168e81bede51c0266f6b2342238aff9dacdebd6f64d1204a84 |
|
MD5 | ee99271de77c420cd352be2aef6dd119 |
|
BLAKE2b-256 | 3e3bf8e4c5972764ae3a42d297bd1e43d3caa6c58a60a71843dc3c9e00ebdfb7 |
Hashes for llm_rs-0.2.9-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c6f29d46500979169a40c9dfb50b50294e09cf02fe602a1cd7ec4501a04b34ad |
|
MD5 | 032a811ca2f5aff8994a27146c3abd18 |
|
BLAKE2b-256 | dcdd13ed682750987bbf6f59dbe30ef8b2af393c27bc3ec41c7a543e3fac02cc |
Hashes for llm_rs-0.2.9-cp38-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 60a1d73641345d56db2d33b0b04017f16335b567985b3904b4c1c838a8dc9106 |
|
MD5 | f6691cadb45e654d11f1372bbc1dcc4a |
|
BLAKE2b-256 | 479c0ad8c48b104e28bfce7b42034c0d93ae1fb7b67ba3a59e5ce999693fbdb4 |
Hashes for llm_rs-0.2.9-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | af780d38e2e61b28d8238c04274179446ea682516ace01fa53e3e9f22a82243b |
|
MD5 | a120feab5aafe073107bed410022ce82 |
|
BLAKE2b-256 | 8a8f35fe28be4eadf150e181855314513d54fe3d59a56ba65f9ac80d427f14c3 |
Hashes for llm_rs-0.2.9-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 56ba9c24895ddb09a5e1d052d78f69a81ed57dc1b931497a52e2ef9f5fdce2a0 |
|
MD5 | 8d73883bde8596c36ef00e7ef8eeae6b |
|
BLAKE2b-256 | dafaef6ec92c7b31d3598747be873cd0f27a11b1f27aa16460f1b655bde140e0 |
Hashes for llm_rs-0.2.9-cp37-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0490ef8b4cce79dc166ad187a72c293a099faf01fd5f78943449f36da3eb64c2 |
|
MD5 | 2bdb21f258a20856f339ddde2546fcd1 |
|
BLAKE2b-256 | 48cb2448ef305969b46502d5abd68a8585ebdeffe5433f54e594ed8e167c8c5b |
Hashes for llm_rs-0.2.9-cp37-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fd6a53036eaf85c8cc9b81923fa674c803f1ee9c0efb03c1f07a275afde69606 |
|
MD5 | 796e0bd054ae2b77f3716903f74ebc6e |
|
BLAKE2b-256 | b77568c2e6fc63d6acbc78d23871dd2265b1436ba143f18aa6f80606edfeeb51 |
Hashes for llm_rs-0.2.9-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c86c69fc67fb1820a29c74ad3bbe5c070175a015c01548c3dab08ba8a1967ea0 |
|
MD5 | 230b5cc7c1e43795d3b5dee9b2d4ed1d |
|
BLAKE2b-256 | 2589dbe14dac6713f9d6142158ffec35aec420461f7f05a80c6db7b66e7f40b0 |
Hashes for llm_rs-0.2.9-cp37-cp37m-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 56829d82a51cf0d9307524e5b860d051d04d437841f396464b55c6705ced9ee0 |
|
MD5 | 7c5d2e40abeaab3de6b03e64442dcd06 |
|
BLAKE2b-256 | 454e61c9028248693704a974ee08bed76bad5a2123e8c4f2122414c809833036 |