Unofficial python bindings for llm-rs. 🐍❤️🦀
Project description
llm-rs-python
Unofficial python bindings for the rust llm library created with PyO3. 🐍❤️🦀
This package allows you to run multiple different Large Language Models (LLMs) like LLama or GPT-NeoX on your CPU.
All supported architectures are listed on the llm project page.
Installation
Simply install it via pip: pip install llm-rs
Usage
The package is type-hinted for easy usage.
A llama model can be run like this:
from llm_rs import Llama
#load the model
model = Llama("path/to/model.bin")
#generate
print(model.generate("The meaning of life is"))
The package also supports callbacks to get each token as it is generated.
The callback-function also supports canceling the generation by returning a True
value from the pytohn side.
from llm_rs import Llama
import sys
from typing import Optional
#load the model
model = Llama("path/to/model.bin")
#define the callback
def callback(token:str)->Optional[bool]:
print(token,end="")
sys.stdout.flush()
# (return True here to cancel the generation)
#start generation
model.generate("The meaning of life is",callback=callback)
The configuration of the generation is handled by the GenerationConfig
class.
from llm_rs import Llama, GenerationConfig
#load the model
model = Llama("path/to/model.bin")
#create a config
config = GenerationConfig(top_p=0.9,seed=1441,max_new_tokens=1024)
#generate
print(model.generate("The meaning of life is",generation_config=config))
To configure model specific settings the SessionConfig
class can be used.
from llm_rs import Llama, SessionConfig
#define the session
session_config = SessionConfig(threads=8,context_length=512)
#load the model
model = Llama("path/to/model.bin",session_config=session_config)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for llm_rs-0.1.0-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0f553ca896437fd148fa47b9f7492063f780a8120ddff9f867ae60642644b32f |
|
MD5 | 05b061e0c72d6d639429179d58e6b14e |
|
BLAKE2b-256 | bc2edf45cb2ba8fa6a24fa5417405d8639df7b40f3d8cca62f254b0d6f95b882 |
Hashes for llm_rs-0.1.0-pp39-pypy39_pp73-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ed3bc2fe55ab50eca170da143be91bf7e8b6d9ede29e869d9ba3d8a506250981 |
|
MD5 | 8bed688da678d5a7d530f1d4636d47d6 |
|
BLAKE2b-256 | 2149bdbd7725026465e3c09278aeb9b08f032190bb997ae43edec8b799dcfba4 |
Hashes for llm_rs-0.1.0-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | adf2371f7f117d1cd9ecea0e47b9bf545c09e443127bba9724873c70e2af6266 |
|
MD5 | 6a7c116c633b57c295d5129ede8dc353 |
|
BLAKE2b-256 | a2dd4ed8d6760155efa7ffab9b9418e814fdb0d3d5b2af56a2617680957ab341 |
Hashes for llm_rs-0.1.0-pp38-pypy38_pp73-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f183f989d2add396a6d906e06e78a179f8c3eaf1717c03c64f324afe84ebec6d |
|
MD5 | e524428371cf7b19ab0a728e6d416cfa |
|
BLAKE2b-256 | 3dab7aafa6a937bf4d2c4699e8562e2440e0e3c6a57d0badc5ccc16ce00c302a |
Hashes for llm_rs-0.1.0-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9aee4c64dcbf83a6507db09b0f64a4d4a80ae069eca26a897d366d285b7f68f8 |
|
MD5 | df0ebf6d0fb5832cc72a806d8c46bc4f |
|
BLAKE2b-256 | b8191c9daa9bde3cf61ec275a63aa722c36dcf1418453b5c9867cd32dc0a7b6c |
Hashes for llm_rs-0.1.0-pp37-pypy37_pp73-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c7b6f7cdd662c8b3802d980834af49f8fc2ef492c72ce6159c7f4905329c7fba |
|
MD5 | 0ed4e46853650443fd82aa23f3ee42aa |
|
BLAKE2b-256 | cfa8a5a880ac01c7449de2211a9e773f523fea3c5a8b79605dfcbc73d4899bec |
Hashes for llm_rs-0.1.0-cp311-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 86578938d830bd5838270a0a506ea8fe2829f0fbf5be816a643e7f231469d33a |
|
MD5 | 1ef017d6612221b6f3a720931afd5b68 |
|
BLAKE2b-256 | cbf577957de22da9a9577bfabb02601f7d66cb340d5c11465e1a2a34880833d4 |
Hashes for llm_rs-0.1.0-cp311-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0bc6d78067551ea06de67538f603aa6a8bf45d177af4a8f0e5b9c6ab788292c6 |
|
MD5 | 3be59c61de8cd32224e1e6d72387707a |
|
BLAKE2b-256 | 8e4a09b67f40e9136d559397511d3ba1d7140de066aa2d1f3c71f9a351080c25 |
Hashes for llm_rs-0.1.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a91d9c61751dc90236bd4b37ddab9d676ea550e17ea901ce741db50e4ebe1c24 |
|
MD5 | bdae770751338d866c9b009d87d7ba1c |
|
BLAKE2b-256 | 49894a8874ca3809641ace2af0c1b6a350ff75e117230280f252c2dceb0c3381 |
Hashes for llm_rs-0.1.0-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a4184e2f674cb51da40ae666bc301d493c764e2df556b88442d49bebe8c0721c |
|
MD5 | 044f673d06469737e71b6a8bca6708a7 |
|
BLAKE2b-256 | b96dfa9d2adfe5942df377f7d98062582835f9ffcad12c75ff60efd10a9fa161 |
Hashes for llm_rs-0.1.0-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a90fed9fdeeec5710b79fb59290c55b574f99873eec752491372b77dafc42434 |
|
MD5 | e1a75005fd4265f83db6fb4e3fa1c618 |
|
BLAKE2b-256 | db2bf36f2d245257a996aaefc9de22e4ed3c5f07f198ffbb91a04f7a76d77058 |
Hashes for llm_rs-0.1.0-cp311-cp311-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 29f0f7f262d93c17659adc0584e77f74750b70d11b96f9387de519de860259fb |
|
MD5 | 0e8e3da9dcd25d6a61c0811801957b7a |
|
BLAKE2b-256 | fa10b94a1454d7a9553727a1e780765152228f7a52a8eb44d8f3231e63dd9dea |
Hashes for llm_rs-0.1.0-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 01ffb8313c5069d7620911a711fe93009ec2ef9e8e6bd7a06ab2a2370d0a4f99 |
|
MD5 | 9dcc9338eea4368f90114d459bdbad40 |
|
BLAKE2b-256 | c19c874c56733e2f4f174bca4de34492fed5e5fcddc757cd3bf86dfd0c67eef3 |
Hashes for llm_rs-0.1.0-cp310-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1a59323c6571abc7fe3f73ba5fd8d4a7a8ed8432ff3fdbfd5ecc98fcc23de511 |
|
MD5 | 28172b328616db2bc2fe8b16c549f29b |
|
BLAKE2b-256 | f373e919a60d7a55bb51a7915e918bd257fc1583a24b9b8dbb44cba0d11e2a31 |
Hashes for llm_rs-0.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1835b09d1158bb137913daba44ca54894e3a8299a06309b28c899c5034cd915e |
|
MD5 | 8b78dc2ee1d7fbc8170ef093ef39e4bf |
|
BLAKE2b-256 | b1fed4d445ca78068b209b8b71d21a86c37bb462d9313408b803b580a90359d7 |
Hashes for llm_rs-0.1.0-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bbd6c06de1bb81300bff3a5769e330f386b8bf5ce2133e025a61bcdeb3e87aa1 |
|
MD5 | 94024714bdd2a70301b54e1121343db9 |
|
BLAKE2b-256 | 884a044a16520073cef9199a5449cdd004d6db402077c8a0488437a8a9ae8f67 |
Hashes for llm_rs-0.1.0-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 478fa0f566ec604bd7f52af026412a3b40a55dbdaab15def82b838f3e3802099 |
|
MD5 | f271812f8220bc91f3e8d040d8a51ea4 |
|
BLAKE2b-256 | 530901e5cb0cd4e38891eb998090bd95005f63e52934ea332337937421e30205 |
Hashes for llm_rs-0.1.0-cp310-cp310-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8fac5db1249a7c04141553ec0f15f08a9762563f4c4181fbabef52bc0e91f7a1 |
|
MD5 | d155dfd0dfd2ffe4213c553ad41edaca |
|
BLAKE2b-256 | 0d5a638016d2f200b82ae670d15f76d71eabf566100624d4851a7278dec7e099 |
Hashes for llm_rs-0.1.0-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 766a8b57a13ad023fff64f10af72844be976336737f0bb0069cb6dd1714b1e75 |
|
MD5 | b04f6340e4713b8e186dfbd7270c7716 |
|
BLAKE2b-256 | 57bfb27fe8ba918e26f1ed99c5e5e7179141e20c2f3439e05ace0c651342a4c0 |
Hashes for llm_rs-0.1.0-cp39-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6615ab699737c62dddfc2818cb77a90e7cd1f552ab25d7df01437160e48e4df7 |
|
MD5 | 11afdc84a3d3f82708e25715185736b2 |
|
BLAKE2b-256 | f0b97e27417014e0573932b7eddfa55e56c5034642f0080bbf453086316cddba |
Hashes for llm_rs-0.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fbc00965cbd52bee62468ff7a004e1e87be5e21584399f00789242a3ebd51977 |
|
MD5 | 1e9aeff20a6c2ec11957548712a2b091 |
|
BLAKE2b-256 | c51b3c548f20623bba7d1a98f4f66672c22a36726ef007a73cec128cca3aca2f |
Hashes for llm_rs-0.1.0-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ba0c30309b22f8ff091f04a98b23d735051d1dad52590853e38a6e11def04141 |
|
MD5 | 4010b332550fac32b26a48e27c6789ea |
|
BLAKE2b-256 | 54399ce7e6fade6d5d05727fa0eef6b9e220135df3836ce3d27cd1bc03359e2c |
Hashes for llm_rs-0.1.0-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 91e7a6aaac6ef5679b5cc41277db44a7827a337eecb68d87c1a46ce5edfe1bfe |
|
MD5 | aedc7f31b055c4996bde981879987d90 |
|
BLAKE2b-256 | 4b1cbd2d1e1198db85c012bb1f81b5af30cb6420019338db9a41c0738463a17f |
Hashes for llm_rs-0.1.0-cp38-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1960e4cca1a009e4d7b43f02d75396c5099ea15a2a9a94a696494b9578df8b6b |
|
MD5 | e1f634bc1871dab8d4d31860b1dffba0 |
|
BLAKE2b-256 | 63b82c231123420ac76e300c402175eca3d8bb1b15dde79631e89b4c8119a039 |
Hashes for llm_rs-0.1.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 95cba4cdd333096406ae336523023a490aecc7b2aaaf358a8f4a87fbb9b21fee |
|
MD5 | 01fd25da7b123230a75be631a2452682 |
|
BLAKE2b-256 | 475f004729d034f615fe7128820fca336285d11b2278d22c72b0eac62a44c11b |
Hashes for llm_rs-0.1.0-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4d90d85554f36917979c892e49e8874f85c7add8cf91194daca759a5d68381d6 |
|
MD5 | 39aa9c12b4cca8182092f4f461b2eddd |
|
BLAKE2b-256 | 7ff80302085c6f2af4ee2a8becb701751b479005ad4b18763074e3f8111ae676 |
Hashes for llm_rs-0.1.0-cp37-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0d8ca1e57faf7a2fba1b360896c74a0ede90a9fca5f0d19ad98b8458294c7f23 |
|
MD5 | 1f0bb0f61fd0acdfcaf1e804aad01279 |
|
BLAKE2b-256 | 0040dcf40ede759689043ccf5ad627cd4d1823a09e5e2b2b73bfa1c5544a63fb |
Hashes for llm_rs-0.1.0-cp37-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ec5d6e460bc7e3a9662d91b5b22742a73928494427b3e96c47f5e5a06e5bbf09 |
|
MD5 | b9edb66ad046633c372f33c6e12f7512 |
|
BLAKE2b-256 | 475516b69906f8c1ef220e1c05a7f70f53ba82ed7739ab60ed4a7dbaae82a7d1 |
Hashes for llm_rs-0.1.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 06838b752bc9a79348070ab710c6e6e449f0dae00a22eb153dc1522f2c561a1e |
|
MD5 | bd12969abfe72ec336fc81f1979f2d74 |
|
BLAKE2b-256 | 29c9a335e10c2eeb5f3c3f85eccc8037488266d9c937826dfcdc4cd020af112e |
Hashes for llm_rs-0.1.0-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 40d8a29ffd3b29cf27e1c6173af40c14ef0fe47b56ea167196af11fc18d88e3e |
|
MD5 | 7c41377c44bce3018ae76b27f6cb7e8e |
|
BLAKE2b-256 | e030de85fc28382aee073592e8a0887c1b4f74a2808b1b474986a7c64326f84b |