Unofficial python bindings for llm-rs. 🐍❤️🦀
Project description
llm-rs-python
Unofficial python bindings for the rust llm library created with PyO3. 🐍❤️🦀
This package allows you to run multiple different Large Language Models (LLMs) like LLama or GPT-NeoX on your CPU.
All supported architectures are listed on the llm project page.
Installation
Simply install it via pip: pip install llm-rs
Usage
The package is type-hinted for easy usage.
A llama model can be run like this:
from llm_rs import Llama
#load the model
model = Llama("path/to/model.bin")
#generate
print(model.generate("The meaning of life is"))
The package also supports callbacks to get each token as it is generated.
The callback-function also supports canceling the generation by returning a True
value from the python side.
from llm_rs import Llama
import sys
from typing import Optional
#load the model
model = Llama("path/to/model.bin")
#define the callback
def callback(token:str)->Optional[bool]:
print(token,end="")
sys.stdout.flush()
# (return True here to cancel the generation)
#start generation
model.generate("The meaning of life is",callback=callback)
The configuration of the generation is handled by the GenerationConfig
class.
from llm_rs import Llama, GenerationConfig
#load the model
model = Llama("path/to/model.bin")
#create a config
config = GenerationConfig(top_p=0.9,seed=1441,max_new_tokens=1024)
#generate
print(model.generate("The meaning of life is",generation_config=config))
To configure model specific settings the SessionConfig
class can be used.
from llm_rs import Llama, SessionConfig
#define the session
session_config = SessionConfig(threads=8,context_length=512)
#load the model
model = Llama("path/to/model.bin",session_config=session_config)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for llm_rs-0.1.1-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1cf87820e31c2b6091ac9a187bb6a6f4ae8cfe31d309053aca118ab4dfa48904 |
|
MD5 | b72b41876e7dcef8d3682ce62c7d1bb4 |
|
BLAKE2b-256 | 0d6b3c696348fe269937ce1d3eb095ea28f8e8a4d83bbdba56b28c13aadcab34 |
Hashes for llm_rs-0.1.1-pp39-pypy39_pp73-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b91eadaaa15bd13a93c9af4d58b8182ff97464cc59f7812f49bf427e444d1679 |
|
MD5 | ac7afb87c38c2828314b9c2cf1482d1c |
|
BLAKE2b-256 | fb2d483e2bdc7c685760131cf3681740629d7ba1d2bb8abc74c0c24942a1e799 |
Hashes for llm_rs-0.1.1-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 81d5e1ecd25b457c515290918757c2bb6ec54d61cf7367725432e7b12d9cd198 |
|
MD5 | 2d81b632fba4deaffb3af155b99264fa |
|
BLAKE2b-256 | 127fa12228199c345153505c47342babc7308905a3251b5876057c00b7c61214 |
Hashes for llm_rs-0.1.1-pp38-pypy38_pp73-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e9fa6cdd3fa319f1ff237128d463735ab5ec67051b449215f397fe8d76b7faca |
|
MD5 | f6e221c1763162f5f93ac0bb3e2a7a55 |
|
BLAKE2b-256 | c1d9483fd31b8e8ca14f0a5728ed1c0109c6019ebb17e19bbc7296f436030378 |
Hashes for llm_rs-0.1.1-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5c8e5bad85e3ff884fbee14f08416461a7da571b31d093dbd74ed62fe659b778 |
|
MD5 | 4180a52c0ed6cd8fe516673113612431 |
|
BLAKE2b-256 | 742c4f1a88e081cfc761ca0a851a6ae9758e58f8ded967c8b33085610f43aafd |
Hashes for llm_rs-0.1.1-pp37-pypy37_pp73-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4cd3c5dacfbd134533115a71b40c4dc4d863890ecd4d5dfae5f5ceba843ec2a1 |
|
MD5 | 64482ab474254f66a3dfc6f2295067b0 |
|
BLAKE2b-256 | df576fdaaa9381f7ef44a00662b6228b878b03e59a5eb50bff3a3ea2958772e5 |
Hashes for llm_rs-0.1.1-cp311-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 72579f61f29a5a7bf8cd399319d532d278fd62fd73d8d8ea8bb2e9f248c12d51 |
|
MD5 | 1ae57842d51d01ca8fc03c82101b68e9 |
|
BLAKE2b-256 | 2404d6fbbdb9b01349865db9f843d7a023deb0637112c4c809fa89197e8426b1 |
Hashes for llm_rs-0.1.1-cp311-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2aec66648f378be186429fed3ef9ba31c9749c1171bc8a0b560ad2ccb045de68 |
|
MD5 | fe22ab9531ac26d17123aee2a7e4c4bc |
|
BLAKE2b-256 | e2d2403fc47d5113a3b512723e00bac7b054e246993c524b2ffbed31629eaf03 |
Hashes for llm_rs-0.1.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4da6628acbb38d225e4e1fee400cb3c6b1d71b1f4f2deb9d39c5b68ae08fb6ac |
|
MD5 | fac37c5ade8be81468d37ca218362611 |
|
BLAKE2b-256 | 235cd8e25c884dbe0f64bca12d860dc7a1c05bbf240200ad5e3b9d742c09eaf0 |
Hashes for llm_rs-0.1.1-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0fb489fceea933172e40b637ce8bc55d1c909c46bd7aa59d3e1017db539b2541 |
|
MD5 | 395d849a62641fe7b458491626a442a1 |
|
BLAKE2b-256 | a5f964a20e2105ede8dd1dbd3877b82f18aff1e068fbb249a95aee25c782dab9 |
Hashes for llm_rs-0.1.1-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ca8886c34bba6bf8144c7a08c8b6a6ebaf6a35e90ba1635919872b176dc9c03b |
|
MD5 | 769ea8f212c18916786b7b432e64729c |
|
BLAKE2b-256 | c9f2e285417a3f1820c7fa7b41759933f0a4cdc16f207d469db051688ef34044 |
Hashes for llm_rs-0.1.1-cp311-cp311-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3d6adfe410c2c360df69dd381dca9ad5880295a442e40df1d18c681cbd148b8b |
|
MD5 | cd5871d9b0e233ccf3ed24dc9922fe12 |
|
BLAKE2b-256 | c16f34a13ea2b703fb04a7f19c92f8774cda6ad26bb69cd18e5964c10d82a18f |
Hashes for llm_rs-0.1.1-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bc21dc52599cd478ada21cfe8f45b47f5200918d982a37c7e7b11eaaafb04c8b |
|
MD5 | 445df4493d469d529d06a5d72df33f4d |
|
BLAKE2b-256 | ba21c2b8a5d82247218a8976e075f695259e6171912015982bcad91ef612737f |
Hashes for llm_rs-0.1.1-cp310-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f3598b24e2adf55ad1183e2cfef0a198f3f71881a8cb92143d73c6fd0772ff8b |
|
MD5 | c2941a8cff8b6919066d81bb6f2e7f45 |
|
BLAKE2b-256 | 620b9a08ee8e0f5f4df9b994d3ab555d87eadb3b8c520fe2bdb5d0e1d32bfcbb |
Hashes for llm_rs-0.1.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a0dc5f753e2b3a2fc410d3f6be6524270c92b8a019170d091ca5a46b3acf6465 |
|
MD5 | d20d2edac26e1c44e059ec04344edeb1 |
|
BLAKE2b-256 | 60c669bc34a358882eacd85901bc2cf9d09b7d36c6c75b497bb87b1800687f8d |
Hashes for llm_rs-0.1.1-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3c6cba6b1ab92dbc915a10f8319916a4328770e8add427833cdbb13937161efc |
|
MD5 | 43f9701ecfa439030c639c94ee0d9c08 |
|
BLAKE2b-256 | e504537862021f321e935fb05b06e1cfca7224339f3862ca9bf706efa01bbbe3 |
Hashes for llm_rs-0.1.1-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b1f73eae7175c69f5e616d640c2b0ab8e350ba0e43fac5b913dbd30b50f5743c |
|
MD5 | 99edd744fb9c73dfa4fb8abad0f3810c |
|
BLAKE2b-256 | bbf83e57c839568ff5ef138e5d9da323655c08649447c06453a8b64c464b681b |
Hashes for llm_rs-0.1.1-cp310-cp310-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f06f17221e566888968020f6a67d0fcae9e26fab466370352f01833946966c7b |
|
MD5 | bc051f8cd47d446c02c91f6b1e777466 |
|
BLAKE2b-256 | 5475ab1e92ed88413d31c7e0f17032711ac983e9bfb34a2273e486f972c6d1ae |
Hashes for llm_rs-0.1.1-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ef0b21bed6829acdde52162fe1b6fe2cb99c056586c8cec1bab26a28e6c2b0db |
|
MD5 | 6d3bc56d6e49fb4a5fcab7d1f8a5ac3f |
|
BLAKE2b-256 | c81b39431c1056b87a86166511204fa402d4c625ef3dd71ae20436d1fd3b228b |
Hashes for llm_rs-0.1.1-cp39-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f3fd7ce58ffd92405971fe5983ff91b31dd2c5d7d9ac5dfbe76a295e93c642d3 |
|
MD5 | 55837bf4a10acec1a10dd683fe5f980e |
|
BLAKE2b-256 | 65c5a9639af798829c8e28af216f6f09c3b3dcf1903e948fc308c2417b7e25bb |
Hashes for llm_rs-0.1.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5a9b2b5df4d5f4474f66db12a5a97e3b0bd9e29816315e23e5d05430fa8241c2 |
|
MD5 | 471d0b036f20dc9a1a2fa26e4e09ca12 |
|
BLAKE2b-256 | 4b55351f8e33d84a0607319396d2a237cb7013e2425c63a5b71fed13c8526d4f |
Hashes for llm_rs-0.1.1-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 671e10f92e963e1a84d2d70ac96e5cd392823ee4f9d4a243b00a4e6f60a0601c |
|
MD5 | b78e507f9a81944312188d16919c45b8 |
|
BLAKE2b-256 | 4f4243c5629abc43f0be720ec6a2b074c82ab309174c5e123c5c1688615e516a |
Hashes for llm_rs-0.1.1-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 486e652fe230bdb991308e8aebd36107701e5fe793adc466ddf8864289453057 |
|
MD5 | baa6b4f97b011d6927bb6a77f853788d |
|
BLAKE2b-256 | a91ff9cc0a5a71069bf9f4a81328d6f466529508eb879377a8934d2edc9dfbe4 |
Hashes for llm_rs-0.1.1-cp38-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2f0713ff037cd6be11d0cea5ad47fbaf498c675bff0c24c4e05a25e986991c1d |
|
MD5 | 251198911fc78f44f1c3e0ec3f083511 |
|
BLAKE2b-256 | 920be68aadbb9834d3687e3cdb175b579ef31c8e9bd13d3b974e2b8633186327 |
Hashes for llm_rs-0.1.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d70059ffea4b24ecc5ec8bc9ebd384a241541f8bc3863e248415936aabf0822d |
|
MD5 | 80767311d9b825a23ab2469cee595f5e |
|
BLAKE2b-256 | 6d6265965dbbff57ceeeee9e3042dfe7b006a9007ef6c99aa428eaa367083214 |
Hashes for llm_rs-0.1.1-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5b0684036010dd82083d18d471886977cb3d19cff8c846ea8df50de3d61cf3f7 |
|
MD5 | a8b68c9936cb98199b0a30eb0fd972f1 |
|
BLAKE2b-256 | 941fa01bf342ec73345c2fb95e18c31aa0ad99be1af50c0185ae7efb3eb535d7 |
Hashes for llm_rs-0.1.1-cp37-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 34dab56ac2558a80524af0cdbffac49dea036c969c9e84c936c38e94c19586fd |
|
MD5 | dfc37d2e021c56a1328f5a217bcbeb60 |
|
BLAKE2b-256 | 1a847c6dbc0e180ba09929f37f9b0a814648699ce61cddc28520caf213afe9fa |
Hashes for llm_rs-0.1.1-cp37-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7d0888731bccdfe8704e11fcdd148affd950dc80b0f7e290f25352a7ec0b2bc5 |
|
MD5 | a154c2f8b8f5ba970813e7392b24032d |
|
BLAKE2b-256 | a04b21a9d3e594a901beaa3171bb913e70514d877d84e7190cb1288d1350a949 |
Hashes for llm_rs-0.1.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d83bf6588acbae695bd3019bd137bdf323a68e4b4ee1f7f0e0d6488db357fa5a |
|
MD5 | fcb095b9edd23530e5230105bb6c1bed |
|
BLAKE2b-256 | f524029a90f02c766ed0dbf5ea3e4ebac0809fba0d879638d57a4be813ec9284 |
Hashes for llm_rs-0.1.1-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 08a4275b6b8bcc8bc8503cae408cf6ada16e26f03006a69f891af1d5e5fd40a7 |
|
MD5 | ea521f79d37c059e0fe8beaacc40fffc |
|
BLAKE2b-256 | a8883272b9df3e8ebaeb9ffd864bd4e1b21183c96649784f15f7db7a05b53faf |