Python binding for llama.cpp using cffi
Project description
llama-cpp-cffi
Python binding for llama.cpp using cffi and ctypes. Supports CPU and CUDA 12.5 execution.
Install
pip install llama-cpp-cffi
Example
from llama.llama_cli_cffi_cpu import llama_generate, Model, Options
# from llama.llama_cli_cffi_cuda_12_5 import llama_generate, Model, Options
# from llama.llama_cli_ctypes_cuda import llama_generate, Model, Options
# from llama.llama_cli_ctypes_cuda_12_5 import llama_generate, Model, Options
from llama.formatter import get_config
model = Model(
'TinyLlama/TinyLlama-1.1B-Chat-v1.0',
'TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF',
'tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf',
)
config = get_config(model.creator_hf_repo)
messages = [
{'role': 'system', 'content': 'You are a helpful assistant.'},
{'role': 'user', 'content': 'Evaluate 1 + 2 in Python.'},
]
options = Options(
ctx_size=config.max_position_embeddings,
predict=-2,
model=model,
prompt=messages,
)
for chunk in llama_generate(options):
print(chunk, flush=True, end='')
# newline
print()
Demos
#
# run demos
#
python -B examples/demo_cffi_cpu.py
python -B examples/demo_cffi_cuda_12_5.py
python -B examples/demo_ctypes_cpu.py
python -B examples/demo_ctypes_cuda_12_5.py
# python -m http.server -d examples/demo_pyonide -b "0.0.0.0" 5000
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
No source distribution files available for this release.See tutorial on generating distribution archives.
Built Distributions
Close
Hashes for llama_cpp_cffi-0.1.0-cp312-cp312-musllinux_1_2_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 719a23ea28a13060168813c065ee06260096234e044e07b8d541cbcb19c892f6 |
|
MD5 | 89092c46411a38c0f41a951587b86ca1 |
|
BLAKE2b-256 | f55934de251f9ba67307fbf998f80835b75f0cca906d8fa1e486851ec7c55ca0 |
Close
Hashes for llama_cpp_cffi-0.1.0-cp312-cp312-manylinux_2_17_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 09f1b30946f506ff082da68ccdce9cb320b6100c448650d0c21ccc3f5910a668 |
|
MD5 | ee61a23b6972c7bb75d9ea1246e6b182 |
|
BLAKE2b-256 | 84e488f484d218dbbbe7d5af9e14fcec8e42f8ede82ab2664c53698db434b754 |
Close
Hashes for llama_cpp_cffi-0.1.0-cp312-cp312-manylinux_2_17_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 589719b250e5a08caec5f9de6f2de78923ed117f64283f23e3ea2a09b678a77d |
|
MD5 | 04de45f95ba4331636af4eb14d8dfb03 |
|
BLAKE2b-256 | 7c85db8e533bfedeb2eb8f7bf6b334317e4eaf97866bbadad824fa6022c1fb33 |
Close
Hashes for llama_cpp_cffi-0.1.0-cp311-cp311-musllinux_1_2_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4a4c6c43c2e058ccb1928ffcbf17dd3e176c2d3a50a03c9c3aadfaa4c1f62d58 |
|
MD5 | ef13cdab2405ff1fb0ed7229a0a812fd |
|
BLAKE2b-256 | 93fd7636ee99b147d21d6466efff77f12989e6720a2440b8e1a83ba61613aed9 |
Close
Hashes for llama_cpp_cffi-0.1.0-cp311-cp311-manylinux_2_17_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a07a51b147123918a2378b005232dec274bbcc7a3c7b59af493a6b1c94bebdd1 |
|
MD5 | 3fafb7c6b82e64c3d8657dfe88a79713 |
|
BLAKE2b-256 | 7632b77f1ee5a655d71a70bc6b99b6403a42aeba44b8b7dad71c6d119b0f138e |
Close
Hashes for llama_cpp_cffi-0.1.0-cp311-cp311-manylinux_2_17_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | de39a39033e558214acd609ab918a769775f1261ab727799a0754995006b49dd |
|
MD5 | f4e96ed09170b12dd9b9362423389c69 |
|
BLAKE2b-256 | 9b243b5b7c55388fde494eee1ec990d712d0c3ef80521a3aa2181d617856f751 |
Close
Hashes for llama_cpp_cffi-0.1.0-cp310-cp310-musllinux_1_2_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7bea12f8e862a1a844b32e165ba0bdc53f8e4495ccc4105af566b8a82c2d5545 |
|
MD5 | 33621b67c8be4e142e081532e2958e4b |
|
BLAKE2b-256 | b918d2fc0e15c9829bfe91e08a949fae0dacc1180f42ded2f76eff2d44e69c10 |
Close
Hashes for llama_cpp_cffi-0.1.0-cp310-cp310-manylinux_2_17_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 86294063912fa9d6b5f28c31eb736d7d04d8cef87e5b9f1fab6f54a836f7dc7f |
|
MD5 | 030df050f0d12ff72cdc9804292ad998 |
|
BLAKE2b-256 | 065796158a6acdb137356fd1e9139edea508bca3dd7ec01a999913915dc19e8b |
Close
Hashes for llama_cpp_cffi-0.1.0-cp310-cp310-manylinux_2_17_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 69380ff7fba23189844da6383ddfc3c1ded4cf1dec488b9e27d1a0fad9d4a975 |
|
MD5 | 4561d30ce802de89f3477e566737662b |
|
BLAKE2b-256 | f604711e3e70ac0882991720aa938f99149bd3907281f0f89ecad7a93f939741 |