Unofficial python bindings for llm-rs. 🐍❤️🦀
Project description
llm-rs-python: Python Bindings for Rust's llm Library
Welcome to llm-rs
, an unofficial Python interface for the Rust-based llm library, made possible through PyO3. Our package combines the convenience of Python with the performance of Rust to offer an efficient tool for your machine learning projects. 🐍❤️🦀
With llm-rs
, you can operate a variety of Large Language Models (LLMs) including LLama and GPT-NeoX directly on your CPU.
For a detailed overview of all the supported architectures, visit the llm project page.
Installation
Simply install it via pip: pip install llm-rs
Usage
Running local GGML models:
Models can be loaded via the AutoModel
interface.
from llm_rs import AutoModel, KnownModels
#load the model
model = AutoModel.from_pretrained("path/to/model.bin",model_type=KnownModels.Llama)
#generate
print(model.generate("The meaning of life is"))
Streaming Text
Text can be yielded from a generator via the stream
function:
from llm_rs import AutoModel, KnownModels
#load the model
model = AutoModel.from_pretrained("path/to/model.bin",model_type=KnownModels.Llama)
#generate
for token in model.stream("The meaning of life is"):
print(token)
Running GGML models from the Hugging Face Hub
GGML converted models can be directly downloaded and run from the hub.
from llm_rs import AutoModel
model = AutoModel.from_pretrained("rustformers/mpt-7b-ggml",model_file="mpt-7b-q4_0-ggjt.bin")
If there are multiple models in a repo the model_file
has to be specified.
If you want to load repositories which were not created throught this library, you have to specify the model_type
parameter as the metadata files needed to infer the architecture are missing.
Running Pytorch Transfomer models from the Hugging Face Hub
llm-rs
supports automatic conversion of all supported transformer architectures on the Huggingface Hub.
To run covnersions additional dependencies are needed which can be installed via pip install llm-rs[convert]
.
The models can then be loaded and automatically converted via the from_pretrained
function.
from llm_rs import AutoModel
model = AutoModel.from_pretrained("mosaicml/mpt-7b")
Convert Huggingface Hub Models
The following example shows how a Pythia model can be covnverted, quantized and run.
from llm_rs.convert import AutoConverter
from llm_rs import AutoModel, AutoQuantizer
import sys
#define the model which should be converted and an output directory
export_directory = "path/to/directory"
base_model = "EleutherAI/pythia-410m"
#convert the model
converted_model = AutoConverter.convert(base_model, export_directory)
#quantize the model (this step is optional)
quantized_model = AutoQuantizer.quantize(converted_model)
#load the quantized model
model = AutoModel.load(quantized_model,verbose=True)
#generate text
def callback(text):
print(text,end="")
sys.stdout.flush()
model.generate("The meaning of life is",callback=callback)
Documentation
For in-depth information on customizing the loading and generation processes, refer to our detailed documentation.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for llm_rs-0.2.6-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b6b972c3c9fd79529f7bd01a8a55d1198c7322292f9a03eb2d8ab90547e39f86 |
|
MD5 | f3042b2d8a718a9978292e5eb32b3b62 |
|
BLAKE2b-256 | 95980e10a1c5d70a15645225cca6066c5ac13dc339fd53efaa1241d77cdd5605 |
Hashes for llm_rs-0.2.6-pp39-pypy39_pp73-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3e4b08030f56e2e43db5c3de5125135a09bd2ba0e3e6d77bf24a6499b1e33f57 |
|
MD5 | d077557f45adc9d99d70671f7d303ea0 |
|
BLAKE2b-256 | dd517d3d449d7a945542a2a8e6930f776844cef2601dc87471047d873f8c2cca |
Hashes for llm_rs-0.2.6-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 84db593c4939508887da3dd1dc881e9ec89317d138c106897f017a74be1ae02f |
|
MD5 | b106918c5d9d85caaed6b2209c631c72 |
|
BLAKE2b-256 | cc58ce6a7f2e468c9af44d66ceac605fcbb1e1aae1d10357251443cf33d7ebd9 |
Hashes for llm_rs-0.2.6-pp38-pypy38_pp73-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 127ddabacb96889949a61bc8fe58919df0dbfa2075b72370f161cfd8c1a6db1f |
|
MD5 | 214698ebd0c7e607c7fb0da8cc44324e |
|
BLAKE2b-256 | f45f7d8c2dfe73e7e07911299a9b79fcf42c0360eb54d444cb39892b330fd011 |
Hashes for llm_rs-0.2.6-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9b716c7f46b1fc35496afe0e801150f5ad32a97ebf428c695dc0a1d5b38829ab |
|
MD5 | 128eb68728e73e8814b91b413db68121 |
|
BLAKE2b-256 | 350fdff938e1e060d15d6c4de13b11ec9c67ec6e76f6a664fadfaec1a6d57a00 |
Hashes for llm_rs-0.2.6-pp37-pypy37_pp73-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | feab4e76d609168c314b499d3e0e745ca450c0db60f2113495f0c320e79d983e |
|
MD5 | 17aa8c48126bccd0458762a64311f921 |
|
BLAKE2b-256 | 202e5f33e7559f7cfd04a31a04dcc0fc68f4474319dfc417357dc6200effffca |
Hashes for llm_rs-0.2.6-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 86d1444f2ecf22a0b9fa6688a627031aa346330475e00294a33d53a4c40e2851 |
|
MD5 | 842050f6137846c1bed2dc71cf8d1603 |
|
BLAKE2b-256 | efb330218013b4883faa32ab2d890f7e7cdcddf1f20c12a70fcbadc9a3211f91 |
Hashes for llm_rs-0.2.6-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 067a9412f414397d5965954a714725457356fde2e318e758cc2b572894dba79f |
|
MD5 | 5fedcc142c6985e5dabd538a420d8b1b |
|
BLAKE2b-256 | 64a0abde5effac796f908fd340cc78ce014e4026aca0354032ec02aac5bddab3 |
Hashes for llm_rs-0.2.6-cp311-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 021d8bbbc364ed0f2b194c238c5dffc434dd29fa3ff70529f37107293e15a50e |
|
MD5 | 2d41736cec7b487901d85d3ab8893bb2 |
|
BLAKE2b-256 | 96f67b7d2ae4b87255c7bcf8d7de6fb99bb8458f0e1bebf599f44be7afc61f36 |
Hashes for llm_rs-0.2.6-cp311-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 58928a9b7338403717c4b17fb697910bb1b9a68a2bbe7ef052f9e070508444db |
|
MD5 | 1cdb197fe2d31f3b1c93eb44aa0813ab |
|
BLAKE2b-256 | aa9b18eb863319f8b1781b936716688335d70d9b99e2d8ab6f489d291c2b7ac1 |
Hashes for llm_rs-0.2.6-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 716bb02679602c564dde3d35964ed19ec1f54782cf20834275a77a69a13a5ae8 |
|
MD5 | c2bfd55cbd3d9d17992dbaae1cc83937 |
|
BLAKE2b-256 | 65ed99d85b2080d8eedd12b1a3c1a802437d08b5110bb2e0627bd4fe4a80a15b |
Hashes for llm_rs-0.2.6-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fecc960fa6c16c9247dd96611677504d732336f676b30fbd30262d01b13640bb |
|
MD5 | 3105c3422659ce79bac3da2ec41999c0 |
|
BLAKE2b-256 | deffe420dc96f2a4d61095ce5d2db88a1c1ad8b78ba18b968695911203841d7f |
Hashes for llm_rs-0.2.6-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | acf58284a33cc5753153923c2b7f139ef98485730490406a74b601191c7e0e20 |
|
MD5 | 63c096b60193ffbc13d3d1199082c70b |
|
BLAKE2b-256 | a183993bf664ae9c252cdc6da5a5e5093522da9c1099aade3dc81ce8390305ef |
Hashes for llm_rs-0.2.6-cp311-cp311-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4c2d66f251fa1d0ff3d412dfcb1ec50f95360af3ffc4968a18fd439a0dc9f336 |
|
MD5 | 07da96b419ccf20e18359c4cd7a2ed3a |
|
BLAKE2b-256 | 29ee41cb7412aa2478b4824e13edd5481261093a913ca125ffbf5810d49f9a72 |
Hashes for llm_rs-0.2.6-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7b02fe15282da4bc6c5ccf734d00d00c19151730de57e66976745ad8e8ed3bd6 |
|
MD5 | 8eac30be1ce8dabb85b10f1acdcf3c18 |
|
BLAKE2b-256 | b397d63bc168b65518cc1f3d116bcdef8776625de8944ec8cb92cbb6bf428652 |
Hashes for llm_rs-0.2.6-cp310-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 378625602afc585467886088f028424e7bd2fc9aa791597a6a1468e424d232cd |
|
MD5 | 642282497603886141041bd8de587aac |
|
BLAKE2b-256 | db8b73e69adb0472e40ec1a20eee1bfdf0b324d77a3548535795482c5a333f7f |
Hashes for llm_rs-0.2.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 567755482d1ce899949719cc09829a7ad698c4588b1f76af0dcb701357dbcef0 |
|
MD5 | 1c864b26950b3decf0dfaae6acd7e391 |
|
BLAKE2b-256 | f262ad2609db008353c0835754bfbfa44f78ea17aa96b7f8cc98efcf0a7d618d |
Hashes for llm_rs-0.2.6-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f233bc6e7540902f234e79e100ca7d6b11f41c13796e6b5ccd109612bd70e264 |
|
MD5 | 4fc927abeff79194fcb6a2365d477730 |
|
BLAKE2b-256 | 9ac223e4d2117507f129e1b373d993bf90d1acf6c2d45fb3b508ca0d1019b837 |
Hashes for llm_rs-0.2.6-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 047c979f0d5babb600d4abe50823251b6d78f229a4319c3c321a69dddd810c0c |
|
MD5 | 92cad17bc7c24f5e5f1b119cc9f1bcf2 |
|
BLAKE2b-256 | 998f2075f4209a3161d13e0f1b4be40df037bc072ea1d86b28251f35993ba36a |
Hashes for llm_rs-0.2.6-cp310-cp310-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e4481dfa1d3b615f6bc7bbf648129dfc1054c175b78e7d866f114aee74c89236 |
|
MD5 | b0e957f0b9730e902d0972e490db380f |
|
BLAKE2b-256 | d88bee02ddad89889497be2e3c45fd38e720fa2c3e351bd775dc3861c83070e0 |
Hashes for llm_rs-0.2.6-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b3b5c5d58b2399b793c0e57e7561af72123c2378e3d20be6cdd3c72bacdb4a2e |
|
MD5 | 2ebea64b6a521f9cd114679d8b4f6685 |
|
BLAKE2b-256 | 649bce8afbabc1791fc605bffb41ae6c986dd16334ec344b72974a5358f63627 |
Hashes for llm_rs-0.2.6-cp39-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 360f8e303c88c4003266359e938316726d5db995e38ed51b3e6c5794bff90779 |
|
MD5 | a5006a89f0b4625195168c8579c81c64 |
|
BLAKE2b-256 | 767a8a7824cff9a072b119c6358b667423a3a422138cc9cbea57e0767413a34a |
Hashes for llm_rs-0.2.6-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 564fd0cebed161b1636910368e1fc5951658eb649b3f38e7877676a22e9a3854 |
|
MD5 | 00996782bf37322191d01094c09afbce |
|
BLAKE2b-256 | db7160efd0ab7d7f4bbae24f4d8530ba26a76505b637f9dc41fcec2311e30603 |
Hashes for llm_rs-0.2.6-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b4c88209ff811e9047296e0217ff6df2e02e974d9d817df2a5ea4463b23648dd |
|
MD5 | cee39339411a6aafcee574b629f29318 |
|
BLAKE2b-256 | d4bd2dd542a4e40f4e172f642db19bd4bb11f4660e52ed78242335ba33805985 |
Hashes for llm_rs-0.2.6-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 301b582caa49151553b6c7462bf370847470d22dac8210a290d61950738d10fd |
|
MD5 | dd978ed8c0958f11ae3198dbeb45adaa |
|
BLAKE2b-256 | bff388048f6c80fd758bb1bf9d3675615e6c6717d203e145ad2e1fad49d11eb7 |
Hashes for llm_rs-0.2.6-cp38-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cdf351bd5c881002e62b62b85f320aecd39eefe51f939efb2c11c90df686d7b2 |
|
MD5 | 0c7da614b8d36314153720872696e669 |
|
BLAKE2b-256 | 4599dc8a32941db8200dce23061934f7200c9e0cc2691fbfeefdd2ae25e80924 |
Hashes for llm_rs-0.2.6-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 03d5517a5b527cb323261eff0233bf265db9b3a441f8e77226286f44d1ccedbd |
|
MD5 | 6627cee7be79101803172cce13a73611 |
|
BLAKE2b-256 | bfad654c7a0827937e2599dc69174458e515977cdda66723ea40b40b049b42d0 |
Hashes for llm_rs-0.2.6-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9a254f525b0c61d63e613852df65eb919791e9825ba0a65d8697e83ce5051f22 |
|
MD5 | f31f8a10c875454cd9e273ed474aac2a |
|
BLAKE2b-256 | 80885f013f25c1cba18cb322e1191388c4f67f2c84f9d18c9e197bbb9010d0d2 |
Hashes for llm_rs-0.2.6-cp37-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 94050bc5d2fd52218c4e4b61ac4def458a57e7d9bd8f326411fca4bba30efcce |
|
MD5 | 54f894196c02e3edc0cf90007a703b36 |
|
BLAKE2b-256 | 93192dd1009c7ed74760724986dc3e2fefdb1e90413370d94bcb1fe1d58cd0a6 |
Hashes for llm_rs-0.2.6-cp37-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a4f9492f695ecf6b8d5712bcaab465c601852397681edbd420221c25c71fb347 |
|
MD5 | 2b889b4f396b71092e4405624c750b01 |
|
BLAKE2b-256 | bbf5a1c73639f9071f6b4ea3b86ce445e46666c305a668085ff24724d9d9313e |
Hashes for llm_rs-0.2.6-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 053361ec80ca0b94b26a365e7f28ffe5b36366ba49b29c47946dcff1bb77713d |
|
MD5 | 830798887349c690fc10bbca54f6a243 |
|
BLAKE2b-256 | 91eb46f8ea5585b04e7a53c803e9aa58b8b41c0c06b645cb6adceacf00fcd72c |
Hashes for llm_rs-0.2.6-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c2de6e6dc705ff3aee9555a079f7abc699625c7c9c184ee249107cbc4233a39a |
|
MD5 | 63a0435bcbda5ba160ab2be6a1a5eeab |
|
BLAKE2b-256 | b9912f474a2a2d627ed393f040aa5754469da6418f472b2b84c3099e9ceade36 |