Python bindings for @ggerganov's llama.cpp
Project description
Python bindings for llama.cpp
Install
From PyPI
pip install llamacpp
Build from Source
pip install .
Get the model weights
You will need to obtain the weights for LLaMA yourself. There are a few torrents floating around as well as some huggingface repositories (e.g https://huggingface.co/nyanko7/LLaMA-7B/). Once you have them, copy them into the models folder.
ls ./models
65B 30B 13B 7B tokenizer_checklist.chk tokenizer.model
Convert the weights to GGML format using llamacpp-convert
. Then use llamacpp-quantize
to quantize them into INT4. For example, for the 7B parameter model, run
llamacpp-convert ./models/7B/ 1
llamacpp-quantize ./models/7B/
llamacpp-cli
Note that running llamacpp-convert
requires torch
, sentencepiece
and numpy
to be installed. These packages are not installed by default when your install llamacpp
.
Command line interface
The package installs the command line entry point llamacpp-cli
that points to llamacpp/cli.py
and should provide about the same functionality as the main
program in the original C++ repository. There is also an experimental llamacpp-chat
that is supposed to bring up a chat interface but this is not working correctly yet.
API
Documentation is TBD. But the long and short of it is that there are two interfaces
LlamaInference
- this one is a high level interface that tries to take care of most things for you. The demo script below uses this.LlamaContext
- this is a low level interface to the underlying llama.cpp API. You can use this similar to how the main example inllama.cpp
does uses the C API. This is a rough implementation and currently untested except for compiling successfully.
Demo script
See llamacpp/cli.py
for a detailed example. The simplest demo would be something like the following:
import sys
import llamacpp
def progress_callback(progress):
print("Progress: {:.2f}%".format(progress * 100))
sys.stdout.flush()
params = llamacpp.InferenceParams.default_with_callback(progress_callback)
params.path_model = './models/7B/ggml-model-q4_0.bin'
model = llamacpp.LlamaInference(params)
prompt = "A llama is a"
prompt_tokens = model.tokenize(prompt, True)
model.update_input(prompt_tokens)
model.ingest_all_pending_input()
model.print_system_info()
for i in range(20):
model.eval()
token = model.sample()
text = model.token_to_str(token)
print(text, end="")
# Flush stdout
sys.stdout.flush()
model.print_timings()
ToDo
- Investigate using dynamic versions using setuptools-scm (Example: https://github.com/pypa/setuptools_scm/blob/main/scm_hack_build_backend.py)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for llamacpp-0.1.13-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fb11e5db1f096b603e432216ca8c0d99acbd0e2641c854492b869439faaaa796 |
|
MD5 | f0595b9cd5b98c5edf666275db5e693c |
|
BLAKE2b-256 | 36555d4f1d7964dea2abb964a6bfd2dca09f3799c146ff459d0ae7e778eb3cb3 |
Hashes for llamacpp-0.1.13-cp311-cp311-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5d7618a77d59ecd8f1ed99b60a780ebfcd2ff733e4aecf97204bbc587b9c59c0 |
|
MD5 | 4298c53399b8191f97abada65ad41754 |
|
BLAKE2b-256 | 666efee5569e879d788173bb6b6caa0cc424b833e8a5c8cc04ed6177c5ecc1eb |
Hashes for llamacpp-0.1.13-cp311-cp311-musllinux_1_1_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 800271ff57de413bd72596523b7621ecbcdbb7bab48741a1402de451d9d63a9a |
|
MD5 | ab23250339058925617355ccdeaa7f59 |
|
BLAKE2b-256 | a7a5367540fc48ea4277deece10c821f30ab16c38e3287bd5bcbb6f76ada5ddd |
Hashes for llamacpp-0.1.13-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 75fb5d7e6b9ecdbd11d9c9c4be5a0ca8766975cde841822787885e7f9e20cee2 |
|
MD5 | 9ac85a7457142267682c8151aff44cd1 |
|
BLAKE2b-256 | b024b8ad19f1254de06b888fefdb7369752ab34e25ededa953a167dacdc81b5b |
Hashes for llamacpp-0.1.13-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 70fe857521ea31057c599b5e0a6b704c71b9d3f56489bc133a7c1f8a9fa687b6 |
|
MD5 | 0780a9a8c37deb5bc8f4fee14bee62ad |
|
BLAKE2b-256 | 6cf296fcef2175291b0b372e517500cab6ee9ae189761f671e591cb7023c42a8 |
Hashes for llamacpp-0.1.13-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0c387726877a8b672c55cf0660fde123c712d79a32735bcef94dcb0e570499eb |
|
MD5 | 5eb1b93ce89e5990247f6eda14151d1f |
|
BLAKE2b-256 | 3a3e5fc51fcffa06747f4484fec36882bd18dca8f76ff2df2e3f520743f5131f |
Hashes for llamacpp-0.1.13-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2ac59dd3946ce69f49da965340b74284ba078cb87c9098e9451fabe9b2189f96 |
|
MD5 | 53f3cf9c730c464fee673c509a20e910 |
|
BLAKE2b-256 | cfe4519817bfcecadf5183eb76122089085498e5f43bdc19fe7b2f73e912664f |
Hashes for llamacpp-0.1.13-cp311-cp311-macosx_10_9_universal2.macosx_10_9_x86_64.macosx_11_0_arm64.macosx_11_0_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 27f8e26962a59c7fffb80ec87c6f13573d17262ffe1e06afc14a2c0d58ad1230 |
|
MD5 | 7ae4146bd73ccabc87ecc16f9c1dd777 |
|
BLAKE2b-256 | 1aa9bdbb6b50eb9328abef2beeb618a70fabc8f5513d5beb0aabd92790f56f1f |
Hashes for llamacpp-0.1.13-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e3a1674e5526156d9adf3b4a2db192a01cc7ed8a5eefe31f4f11526377b3c2ca |
|
MD5 | d63477699a1ce7a513a2de37191b31fb |
|
BLAKE2b-256 | 149329384f95b1e2d29eccc2d823151ee9f867fa16bfaf3c08d14391346453f7 |
Hashes for llamacpp-0.1.13-cp310-cp310-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d07e2494d6ddb87a1fe30af7a5c5a0225704b7acfa27ed9ba50b932c16dd744f |
|
MD5 | e907d5d78bd5b826aa5d543ea70ace73 |
|
BLAKE2b-256 | 557e329c46fde2359a138853a028d9af9d7d505fc7400d4eb928ad92c62da484 |
Hashes for llamacpp-0.1.13-cp310-cp310-musllinux_1_1_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6b254885302d23c0e23b81ec2ab048555ee07e4d5da5f86dbcc52545bb1be2e5 |
|
MD5 | 21cd1e0d7676d2f08b94f526df9962b8 |
|
BLAKE2b-256 | 4b30e0a8aa916021e5a9e0e696c7178acea62a8fb2b147f3edf28759299d798e |
Hashes for llamacpp-0.1.13-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | de3aeedf0111bdbe7ce50e0ced556dd201744e517bfab729c1029ec9b1825295 |
|
MD5 | 4ae74894a00e354cfd0d71878ba09c92 |
|
BLAKE2b-256 | c6cabd3db743434464f2937206ccf0c803477365bbf91f0e0f7051928199f9f1 |
Hashes for llamacpp-0.1.13-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | eb4478ca05f2d1315ee6d267e87872d601f9776b0a86a4460decc25cf9c46e87 |
|
MD5 | b5ce2e1c8fc8b5972623c82cab939975 |
|
BLAKE2b-256 | 81115bdfe022c9f9ca717401c3d5372dbde1c81bd0a7f4c3a7090deff1276a7b |
Hashes for llamacpp-0.1.13-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | afba13ec89c89d72402bf27241f4b1dd1d0c4b3d9ddd98b669b6ae48d5342f6e |
|
MD5 | cc68c700ba7267adf40cc2661d2ff96d |
|
BLAKE2b-256 | 42eb1edf72647f7a465237ae90b2bdedc4d5399b60cee76d6a890b76efb1c69a |
Hashes for llamacpp-0.1.13-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 80eef87e3ee8eb4341f0ed7e24b0436f518b57782d8498307d544e2f6b552b9b |
|
MD5 | d08d5fb444a19fede257880b38296c0a |
|
BLAKE2b-256 | 4a9f2d8f4f4f9085f6915a99edd83c70f99c9ca14ae3e945a355961ab3cdba9a |
Hashes for llamacpp-0.1.13-cp310-cp310-macosx_10_9_universal2.macosx_10_9_x86_64.macosx_11_0_arm64.macosx_11_0_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f44794f011df62733c887c2ffed47f7d21bf1c025c8bdcc7dbbb084c0b59fd7c |
|
MD5 | 3eda91f64c4f36b357c0883c91eded1a |
|
BLAKE2b-256 | 761aea546f4b32aa79bcd839c510be1e63ee637edba2315c767767b8eafa16aa |
Hashes for llamacpp-0.1.13-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ca25dac7f75a84a33f2931ce61e9d48dd145dc2ddc27c48074401a40eb3971f2 |
|
MD5 | 50821fe67703e636d1e2cda5cee772cd |
|
BLAKE2b-256 | 3d89a5f5f73b8b0e8e876855ccaa0b25985cc7f5d7a6d02b4857f688d85e6b6f |
Hashes for llamacpp-0.1.13-cp39-cp39-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e9102a0b9d903f7294d06e32c3c1b8ea53c99aafe0ce407e201c6e033d112d8b |
|
MD5 | d6e14c4189ef95154051b9f68108f192 |
|
BLAKE2b-256 | 308988840344e05a83b58eeb377c3671242213b2e9242b659485b0967ec09f1e |
Hashes for llamacpp-0.1.13-cp39-cp39-musllinux_1_1_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0ca00aa7f66b9f869bd5268befc66f14658ff4c2308ef7ffacae62c2497c9c0d |
|
MD5 | 5a70143f8afe0bc4204536508c568690 |
|
BLAKE2b-256 | 7bc7d33ff70f835fcff67b8ba7b94d4272fc75b44f9fbf163092f0c28a0276ac |
Hashes for llamacpp-0.1.13-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a245ea6af1fa42a00246c001a6df46c947bb93987c9a78380aecb563d65f0cbd |
|
MD5 | 97b68b5f89daa7fff9a6f56c1e8eaab3 |
|
BLAKE2b-256 | e157972f0120bc53b9e66e9d60be6ebfaf0f500c9ba6060ab8e926aaa6e69dd5 |
Hashes for llamacpp-0.1.13-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8ce749ca4ed2159b83588f9a4bd9f728bb9190e3e3f321d957386bc5f7017c07 |
|
MD5 | bee5d35e982f3c5771c4c4927293d8ad |
|
BLAKE2b-256 | 20716e3ab7e8664c5e1f6d0c3572807f4ef6f04b0c8d09ff948e3b520f164c7d |
Hashes for llamacpp-0.1.13-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ddf516e3b5f9219848acafe75c4a98c8919b24128b25e86c68c24d98d263f58a |
|
MD5 | d325fbc6e934f308b2d85e28917c4b94 |
|
BLAKE2b-256 | 8f26dac05e68ad41e40e485ae2a736d30aaf2f2131f7515adb74b72fc71c8846 |
Hashes for llamacpp-0.1.13-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 37e757a6233e91390758e31f062e842f8ece94d0a126d2472fe5ed9ef665bce8 |
|
MD5 | 75fa18e4c3bde183aaf02d2cb62fe3bd |
|
BLAKE2b-256 | fdca7b541f27388f580a72221b24da0b40277926cdf5657c95b85e5c7ec1796f |
Hashes for llamacpp-0.1.13-cp39-cp39-macosx_10_9_universal2.macosx_10_9_x86_64.macosx_11_0_arm64.macosx_11_0_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6fdb8a88c035a9acb0dba7262173c3c5358fb6d4e6e6e13d99f14b35fdef49d8 |
|
MD5 | b4bf3d27396ccb15e80715614ef291a1 |
|
BLAKE2b-256 | 75595908c63659b867f2abd42ef214f86e6dcdbd94e225a1646f0ac741db4d6d |
Hashes for llamacpp-0.1.13-cp38-cp38-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9f2cedc4fe82a03445013c2362dd056a9b97949cab94fee011a647da53df979b |
|
MD5 | 4c817cd99ee5d67e202ca7e8afcaa230 |
|
BLAKE2b-256 | c5261539f1dc046fe7c078703c3713d34ce29d8845d250edddbcd6cf3ee9d314 |
Hashes for llamacpp-0.1.13-cp38-cp38-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 446c0c38a9efa22645f5c66a9bf0e730a7bc5cdc1f7203a9719a47eea819994e |
|
MD5 | a8c5f2912f4f059453b381ed1347dd20 |
|
BLAKE2b-256 | 946f7c09ecc2b34076f72f9e342029f4d8abc14257f8f9f3b8bc1f58b288b3eb |
Hashes for llamacpp-0.1.13-cp38-cp38-musllinux_1_1_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b528f2c521f7bd0946390a674de1194adac96b02df863b8017f61e63fba95e2a |
|
MD5 | 70fa8a0578023a1428908078aad73a21 |
|
BLAKE2b-256 | e7ba51931db3df06db0d553d3ba101bb87f9e346414b7f09aef45e4c4889757f |
Hashes for llamacpp-0.1.13-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 65ff61e6615480487641cccc4837b808f0ca61b85881a36c63db1231127e9e24 |
|
MD5 | 78c2e63371b73652df31e4fa0cc79aa8 |
|
BLAKE2b-256 | 055dbbe51e90478be39a3b91b9cdf8b819905a386801926c3c52dd8a3606b096 |
Hashes for llamacpp-0.1.13-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d75d7151b8d75b1e7cdb3c288792ae64b23e08c84e76776ea96c5b9cd8a47789 |
|
MD5 | 1a2d21b7290b0023d1ff83895bf39098 |
|
BLAKE2b-256 | 1f667a870c47535b3f7fdbb8d5d7aa8969ca5354d3d5a656b12af6d0d9ab935a |
Hashes for llamacpp-0.1.13-cp38-cp38-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9610f17e7bcc92fb2eef91e5e6341df5f0e497376fa1985b85cd3df8d9977ac9 |
|
MD5 | 3c642d6d0926bd93fd96eef08da01859 |
|
BLAKE2b-256 | 6b401a2620a11b1a5c98a8ffba0b9d638775eff96e2a8c360d2599eeac8c9bf2 |
Hashes for llamacpp-0.1.13-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cd7c9753a692b357c1600d8c58d87e98e3288a3e4439526652f6a24739bfd97e |
|
MD5 | 01819f542c8b0e54d7904b1d7d8397a7 |
|
BLAKE2b-256 | 07682dc11ae9f4792ca520c241042c5bc99f346a4990f718be9fe4d18ded819b |
Hashes for llamacpp-0.1.13-cp38-cp38-macosx_10_9_universal2.macosx_10_9_x86_64.macosx_11_0_arm64.macosx_11_0_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bc5911ad653502cc63d73c8dc50a22db94d9ab3560d2e9bb674d71fecf1e9072 |
|
MD5 | f025e8877a78bbe534ee8b439d760ece |
|
BLAKE2b-256 | f1941081481bee89922f05dd2595e1707b03f0b1a1e4a621044da09c1c54fccf |
Hashes for llamacpp-0.1.13-cp37-cp37m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 564fb7019a51c2404f66981dbd548bf691007e05001477c9b10645cba4d9d1af |
|
MD5 | cef3400614f6181d85cdee21dfe1e104 |
|
BLAKE2b-256 | 8feed7acc724c0db7602d80586d2fcf4dcedbb3480136829296332e7dab448c6 |
Hashes for llamacpp-0.1.13-cp37-cp37m-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a334e473bce5b6bf3b231239654232449df2e30fe6ac80bc57aceefd56bde664 |
|
MD5 | 46f4f1dabee1b3a23d050f75feb7c8c7 |
|
BLAKE2b-256 | 3334e0ffc86f9e0ac67ef7a280b85c471f28f2926ed7c1a5d823a426ede2beb7 |
Hashes for llamacpp-0.1.13-cp37-cp37m-musllinux_1_1_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f11aed908b4d1c52da9a3394857b05636fe39ccadc83f15decf61d44a88f1709 |
|
MD5 | 64b985ae8e8948433a6344fdb979a33a |
|
BLAKE2b-256 | 62fc646fc02e5a2591b5c22a99afcf2c2a8930ddbcf49173d0b466e1da4ddee1 |
Hashes for llamacpp-0.1.13-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 87f69dc772cdb10849e6c58efdbd71dd58fd4e0cf50fe48ce80177d4d3e97c12 |
|
MD5 | 5a6617e64eaaecdcc0f6e6a34e1bb23c |
|
BLAKE2b-256 | ab181eeba499a5e9ce2adbe1841700d12f8a7b60de738ae113dac7461cb4b9ab |
Hashes for llamacpp-0.1.13-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f51b3085939949aa266ca9606384dd64e0fe095c7aeebf0125b0f627602e1ae1 |
|
MD5 | 275080cedb81da8b2e601527b05df99a |
|
BLAKE2b-256 | bce1164b1cb536c1cb5eed4a7aeb78dde9a351adfe6989d0cdd8b10a8e9ca579 |
Hashes for llamacpp-0.1.13-cp37-cp37m-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | abd8ef24ba79831cbd352e7ccd613522a05c70b114f02c098e353b1c0e3385bd |
|
MD5 | e84bc3fe6fa237fadff250454715846f |
|
BLAKE2b-256 | 03179754674719987585d074dc200fb40bdfb191e551b9a241cb77ab819f7324 |