Unofficial python bindings for llm-rs. 🐍❤️🦀
Reason this release was yanked:
invalid import
Project description
llm-rs-python: Python Bindings for Rust's llm Library
Welcome to llm-rs
, an unofficial Python interface for the Rust-based llm library, made possible through PyO3. Our package combines the convenience of Python with the performance of Rust to offer an efficient tool for your machine learning projects. 🐍❤️🦀
With llm-rs
, you can operate a variety of Large Language Models (LLMs) including LLama and GPT-NeoX directly on your CPU.
For a detailed overview of all the supported architectures, visit the llm project page.
Installation
Simply install it via pip: pip install llm-rs
Usage
Running GGML converted models:
This example shows how a Llama model can be loaded.
from llm_rs import Llama
#load the model
model = Llama("path/to/model.bin")
#generate
print(model.generate("The meaning of life is"))
Running Huggingface Hub Models
llm-rs
supports automatic conversion of all supported transformer architectures on the Huggingface Hub.
To run covnersions additional dependencies are needed which can be installed via pip install llm-rs[convert]
.
The following example shows how a Pythia model can be covnverted, quantized and run.
from llm_rs.convert import AutoConverter
from llm_rs import AutoModel, AutoQuantizer
import sys
#define the model which should be converted and an output folder
export_folder = "path/to/folder"
base_model = "EleutherAI/pythia-410m"
#convert the model
converted_model = AutoConverter.convert(base_model, export_folder)
#quantize the model (this step is optional)
quantized_model = AutoQuantizer.quantize(converted_model)
#load the quantized model
model = AutoModel.load(quantized_model,verbose=True)
#generate text
def callback(text):
print(text,end="")
sys.stdout.flush()
model.generate("The meaning of life is",callback=callback)
Documentation
For in-depth information on customizing the loading and generation processes, refer to our detailed documentation.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for llm_rs-0.2.2-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8009268070654cfb003ab6226c745e7fa6371c99528d4dfc8e8af88778eed888 |
|
MD5 | 80fcd64124579a30c242adbe8c6bed94 |
|
BLAKE2b-256 | c23a7646882ea869122ae3e48fee4c01ede369a3e10167c434e01497c83ac954 |
Hashes for llm_rs-0.2.2-pp39-pypy39_pp73-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0f6653b1f90acacef070ba60c686a4e1eee40a62073b1c4b053a076e9ecd3572 |
|
MD5 | 8b2d46d913deeadb16b5c0990194c6da |
|
BLAKE2b-256 | f538b2d0b2712bf12eec88594363b13e8ec259e9b18b1ce5c64e900f8e22f342 |
Hashes for llm_rs-0.2.2-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cd70aecbe01697417ce2d06d8f4fba7dbd97043278ca4086f9e408557664c408 |
|
MD5 | 3e5862a56049a56701d50dc122684006 |
|
BLAKE2b-256 | d030f01b811405a41034cee1193784d48255f49c482e869527ad6da34339dfcb |
Hashes for llm_rs-0.2.2-pp38-pypy38_pp73-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 616984c94512362189e7f47dc303b41e7f2487a8253ce1a89e07b60230b59b70 |
|
MD5 | 46c2493c93763afa8783b13e53fe13c3 |
|
BLAKE2b-256 | ef0986f09a40a79207d06f52f87371edc7cc31e0d612c1c2432505d7163ddcf2 |
Hashes for llm_rs-0.2.2-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4c60df7ac25b1dd424eed73665ee2cbd5d45ef4e6cb017f636ac9253e1b7821a |
|
MD5 | 69750d361205ba30d00ca9876cdc9d04 |
|
BLAKE2b-256 | fcfb5ac7e04c086d2674db5f4015a8a00be0957ef230165a06c9045fb87cdfed |
Hashes for llm_rs-0.2.2-pp37-pypy37_pp73-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e024eec9ca7740e9699affb573ed31c8426fa0d57bd8438702a09144bfbad569 |
|
MD5 | 8938713a69197bffcfcf3472f382098d |
|
BLAKE2b-256 | c653c4280ce5ba80cbb83e7025aa05954dc29eaceb2037cc0893bad6ae577bc6 |
Hashes for llm_rs-0.2.2-cp311-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7705f1a10d10f2f747670f49d03653d54530344389ea3442d436449bd4deb942 |
|
MD5 | 1b0327d4120a6017a9aa763e77c37f2f |
|
BLAKE2b-256 | f227093e3f3c65c0d4e0e1b42639fa23f3f1b32f60b45664f6c7e7df3c83cc92 |
Hashes for llm_rs-0.2.2-cp311-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8a9a47d4b699485b44ef92d52f29e5630c068cc2fdf28bed8bcb4f9613b34221 |
|
MD5 | b13a2f6f6ae5467849bff5ba2b380754 |
|
BLAKE2b-256 | 23ec6f5a427f5c41695242ae809e43d8b58fb659dfe5de2905312e85151a1f4a |
Hashes for llm_rs-0.2.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7997242d30de0e8b4d8c6e08ff268043304a119dd4a467d765446e9cba164dc2 |
|
MD5 | cea9b50a7c0a61287a47955878b2cc32 |
|
BLAKE2b-256 | 20c0352b462dba7b528b8b59a8a4dc9906f413330fb544f763dcccbd3f5fc1a7 |
Hashes for llm_rs-0.2.2-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c60cf40c761ac80dc9c653835a71a007a5480996e683ab1477078fdc86db3819 |
|
MD5 | d8d0be7f46b7bd8059f48340877ef2bb |
|
BLAKE2b-256 | e753e53f7848feb35a06d53b9e4782717e91025822e166e738920bfbf516739e |
Hashes for llm_rs-0.2.2-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a6bad3e50121fd6360af2e21a8e75d33c14941598f91cc5650065c6c504b4281 |
|
MD5 | fd9c80ba70be836ac966116d077303d0 |
|
BLAKE2b-256 | df2c1ce95af51a12bfa603e69854cf4e6ea3ada7e859d812459740f71a407501 |
Hashes for llm_rs-0.2.2-cp311-cp311-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fe4958eaf94219ef01be0eba372e9165140dee161ec16fa4a4a3646a1873d4ec |
|
MD5 | 7853132a4307484c9151c07d9c9e350e |
|
BLAKE2b-256 | c49ea3a52101ad9f5a18624d7c14e861e1948012e9d58d9a7c44c6e53606786f |
Hashes for llm_rs-0.2.2-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 317eea4ec84cbb57d923516aadc3f779870db60c85952b2646f3a878e3f55c91 |
|
MD5 | f9b06cb6ec16e0c023de699ec8d82d9e |
|
BLAKE2b-256 | d40260d80c174f2a52a0e9769629507e83810d69c7db672a70d37bcb84d4c15a |
Hashes for llm_rs-0.2.2-cp310-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 59e3d69fa8f22e04db093a709c0c3e5631b7bea28d9c6ae290f0cec839539321 |
|
MD5 | 49cd80e533053bdc9a5be403d63ed64a |
|
BLAKE2b-256 | a8c1dd12c026aa2749e6817d22a542610add28aeea91839e9e36543d8b62037c |
Hashes for llm_rs-0.2.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3942a73194ff66a07e55180862c957bee75eb9ce039fb6d6b3bd427c07176006 |
|
MD5 | d046ab185d77e7ed0b3b304d749ee373 |
|
BLAKE2b-256 | 7edda1b900ddad0f435820597b0744f96ca162bd91e43b5c8606f38d7b2b314c |
Hashes for llm_rs-0.2.2-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9ac02f14fecb01e74a460583508b21098c49dc0968a130f5d62a4da14b3a8b53 |
|
MD5 | 7bfca5eb9496a6980406ff2aa2e356c4 |
|
BLAKE2b-256 | 999ba4d416a30fcfa43356d9038cd68796f7405410125972b062c3987bf8c6be |
Hashes for llm_rs-0.2.2-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 503ef7a34bf4f4be43e43ccbfd71725e7fb78e6653ac96d17a7e8e24cc704526 |
|
MD5 | dd18a947678eaad339b1ff4271ffd265 |
|
BLAKE2b-256 | 5433da2b92466abce163286514e15fa8969d6df8972a959b0e35a28db3b2ae18 |
Hashes for llm_rs-0.2.2-cp310-cp310-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9441406c1e274cd8fd377f391aacbe1616b29a2cdea9fc8e26c768d3b22e7ae9 |
|
MD5 | 63820a895c4626fd6899c1869cb399c3 |
|
BLAKE2b-256 | 00c20677c93772247161fa5197ef37adb8fbe30be23e71cd55d861b31c471012 |
Hashes for llm_rs-0.2.2-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 55908e87e0602583e169c414b31db3935de7f2786e862eaddc72f03c5a87579f |
|
MD5 | 107627a9903828e9dd8346a425e78ef7 |
|
BLAKE2b-256 | 4fc91b7c8f88714ad657efa3692ca70789427f1b423b67d770c1a776d9ff5fdd |
Hashes for llm_rs-0.2.2-cp39-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e8f18e239e7218c845218dac2f9e2243443ff8bcd0be74dba9fc54f96bb996f2 |
|
MD5 | 0a634147bcddd7c9f77d2cb19ccd22ef |
|
BLAKE2b-256 | 8f408b23e0eb51f053ce9b1b487e8d5fb5825346fd77f2872b5139d91954bcf7 |
Hashes for llm_rs-0.2.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 71b0a06decdd71cb162a7d724f0e199c13869bb86019a5cb126a7be17e6f4588 |
|
MD5 | 86ce6336f7dfa102f1924639a311f492 |
|
BLAKE2b-256 | 75d78403de715d78e5365ae127c62f20c1afb5a99e824545f4b29bd09fd10ed3 |
Hashes for llm_rs-0.2.2-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4412a5478ea16201fc24960b129c6bd8beece6aa76556778ae2a914a759f9903 |
|
MD5 | d17b2a5d003c0c2990d96525c62218a1 |
|
BLAKE2b-256 | 915ada2a121382689adfb55e8d2ec8286d0c0b101e05e3d190770c132cff1eca |
Hashes for llm_rs-0.2.2-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6ec998d098e91bbb036ce6150c0c5a6d00d6620738289fe760860128982f876b |
|
MD5 | f51bca4a4b73bea70626092e4e1af4b4 |
|
BLAKE2b-256 | 34427d7edd1acebb06ced85360021cd90183bd891a0d4bb917642897cc8f9db5 |
Hashes for llm_rs-0.2.2-cp38-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 52906e0c36af33bd4c1bcb2fcbe4283a07c252c7f16b9ba2e6c209e883b1ab5d |
|
MD5 | b06dbe4b9e437cc1b903981fe80cdbe1 |
|
BLAKE2b-256 | d0e704a9f523d6497a933fb4324256a98245de305e3dcbf1d7facfab86ac5753 |
Hashes for llm_rs-0.2.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 741f42fcc5df9331623ab1838e3a0b2e85a12c00f575dabca65386629ecfec7d |
|
MD5 | 5bb12f8baa28687f4a1da2fd021799aa |
|
BLAKE2b-256 | 80c7f4f601fdda6301fc6aa77ea06993a8f01633847535a0e507469b93ba0a9c |
Hashes for llm_rs-0.2.2-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | deffe9079f0b94f5c568b09ba77d51261d467aa460653f92b2e44da7b57b4635 |
|
MD5 | 1a46d5d41c9583ba57d1560d19a58bd0 |
|
BLAKE2b-256 | e85d35c8d212082d77bb38dbb95f68e8d947e6b21220a722e32d6ed3eb02fbaa |
Hashes for llm_rs-0.2.2-cp37-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5540843c9ceead2136872db7a371caa10b8533e8d2f7dcd3051df5b7e47ce5a6 |
|
MD5 | 9fe40240e12c32e377bbbe6bd9444316 |
|
BLAKE2b-256 | cf26e026972d949f706c54237d8ff838fb4a7f5c3c543fa449a3fbdf99f437a9 |
Hashes for llm_rs-0.2.2-cp37-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | efea04d363e2708531707c77ad8e58682bad862269fbe184f704a4b68bef0b68 |
|
MD5 | 3d5fd11481ed0def66d6193e1ad4626c |
|
BLAKE2b-256 | e193b6cc6ee967cdc67d6a2d420f6e3066ba3c26727da7c3672d6651bca26da3 |
Hashes for llm_rs-0.2.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4a03f3339da3dd69a78d9d2d90519ec20ad5e736a2428674adefda67e3910560 |
|
MD5 | ed080cb0a8cce8bbc60c37cc1581b197 |
|
BLAKE2b-256 | e59733f47d001bb480ffb966f2eb37a6e3ee4c34c8b8f4501bad92f3aff92139 |
Hashes for llm_rs-0.2.2-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b5525340c23b0da804af62c79358cd8d4930a700ee2b522cbacbc575e630269a |
|
MD5 | 9aab91bc5107b0fbd1d77fec8049ae0d |
|
BLAKE2b-256 | 9301ed8695ecfc32719af733571b0a6607f94bc64804bd9a57518febd2c00137 |