Python bindings for @ggerganov's llama.cpp
Project description
Python bindings for llama.cpp
Install
From PyPI
pip install llamacpp
Build from Source
pip install .
Get the model weights
You will need to obtain the weights for LLaMA yourself. There are a few torrents floating around as well as some huggingface repositories (e.g https://huggingface.co/nyanko7/LLaMA-7B/). Once you have them, copy them into the models folder.
ls ./models
65B 30B 13B 7B tokenizer_checklist.chk tokenizer.model
Convert the weights to GGML format using llamacpp-convert
. Then use llamacpp-quantize
to quantize them into INT4. For example, for the 7B parameter model, run
llamacpp-convert ./models/7B/ 1
llamacpp-quantize ./models/7B/
llamacpp-cli
Note that running llamacpp-convert
requires torch
, sentencepiece
and numpy
to be installed. These packages are not installed by default when your install llamacpp
.
Command line interface
The package installs the command line entry point llamacpp-cli
that points to llamacpp/cli.py
and should provide about the same functionality as the main
program in the original C++ repository. There is also an experimental llamacpp-chat
that is supposed to bring up a chat interface but this is not working correctly yet.
API
Documentation is TBD. But the long and short of it is that there are two interfaces
LlamaInference
- this one is a high level interface that tries to take care of most things for you. The demo script below uses this.LlamaContext
- this is a low level interface to the underlying llama.cpp API. You can use this similar to how the main example inllama.cpp
does uses the C API. This is a rough implementation and currently untested except for compiling successfully.
Demo script
See llamacpp/cli.py
for a detailed example. The simplest demo would be something like the following:
import sys
import llamacpp
def progress_callback(progress):
print("Progress: {:.2f}%".format(progress * 100))
sys.stdout.flush()
params = llamacpp.InferenceParams.default_with_callback(progress_callback)
params.path_model = './models/7B/ggml-model-q4_0.bin'
model = llamacpp.LlamaInference(params)
prompt = "A llama is a"
prompt_tokens = model.tokenize(prompt, True)
model.update_input(prompt_tokens)
model.ingest_all_pending_input()
model.print_system_info()
for i in range(20):
model.eval()
token = model.sample()
text = model.token_to_str(token)
print(text, end="")
# Flush stdout
sys.stdout.flush()
model.print_timings()
ToDo
- Investigate using dynamic versions using setuptools-scm (Example: https://github.com/pypa/setuptools_scm/blob/main/scm_hack_build_backend.py)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for llamacpp-0.1.10-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1d2911ad9e42ae8ecd7bb0c5853a6bd9f95a52291feb0efbb6591c41f1eb1c3d |
|
MD5 | 6eff327ffa4122769d8434cd7d5cafab |
|
BLAKE2b-256 | 6b9a6f3f2b2817598d93a02be3f12c397c73164db344ed2df92a990dd50d62cb |
Hashes for llamacpp-0.1.10-cp311-cp311-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ee7eb2496908b4c2bc65f0cf9f6ea82bfe117d6753509b137ff583e99afcf0e0 |
|
MD5 | 3d946fb9f2423803bff1a53ec9adb385 |
|
BLAKE2b-256 | 73348bf3bac94ca1824a335328c795da666334d8016d36c0257aea4b346dcefe |
Hashes for llamacpp-0.1.10-cp311-cp311-musllinux_1_1_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b03b2e9ce1cfbf76f95a1e540524b5fb64ca529c7a03b428d1ecf0fd4da928a3 |
|
MD5 | c322a0f3033258ad0d414e5f1c79228e |
|
BLAKE2b-256 | 2f249190b5468d92c0f3d992997f6d91da6a08ada3e54b673f09adf693817f20 |
Hashes for llamacpp-0.1.10-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 786223b1b5898b6b4b6ac7bbbd2a77cf07a436c37f1af0e8617e20aaa37c7403 |
|
MD5 | 61624f126e1b6ca898466a0cca319657 |
|
BLAKE2b-256 | 1ba699e87f4adea6450148f4bfa771fc60af38204f6a2b426f7277110346f7db |
Hashes for llamacpp-0.1.10-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 172a01a20aa2c7024d2161fa39ba0ed10b0651bedf8c86e586b753513b290279 |
|
MD5 | b076e209f77469b0b9ff45ed4f7c9026 |
|
BLAKE2b-256 | 0d48e515c3ec51f0129d953fe79c4820cdb33f931f5f45cd0a0d52ca2407b89e |
Hashes for llamacpp-0.1.10-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 661742fb93e920ef38136396a2726efd605095f7b89a5ccfc88322a653157eea |
|
MD5 | a69cd808c268b74afc80e1110103fa45 |
|
BLAKE2b-256 | 7ea81dc71d82c725626e6aa0dcbc836b9f205016f67a53dbe7b0171185e3a4d8 |
Hashes for llamacpp-0.1.10-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | df9701d1fcb14a37a374f4e24ff96c4608adfb8b90ff630b617e24acb96b1b3d |
|
MD5 | e1f9b08471ea240e24b65aaf5887d664 |
|
BLAKE2b-256 | 871e6c56694c5ed6420234d99d7459add278ab321c8628c461c17f03a642eab5 |
Hashes for llamacpp-0.1.10-cp311-cp311-macosx_10_9_universal2.macosx_10_9_x86_64.macosx_11_0_arm64.macosx_11_0_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e890f3fbba3acad210e87b36ab4b62fb34f54f8671aaa44fa2cda3ed132505f0 |
|
MD5 | be27659814e2e3c5e715ce2d360ef906 |
|
BLAKE2b-256 | 5f7317a3485e57434bb898e44d52260b854526ada3b27fb80b2fc067e6de83c7 |
Hashes for llamacpp-0.1.10-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8a69cd675364bf41a1c7b1c8008ebc66c6eff94861a760da9960a04f74659e0d |
|
MD5 | 82ce3ac414ff23e05a9a23254f7c5826 |
|
BLAKE2b-256 | 220ece31ad7928a8a103885c0597880f43de7f970624c94403c8d38b3aa66f1d |
Hashes for llamacpp-0.1.10-cp310-cp310-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3e84addb8f52424cc01ee070e369ec2af9fb9192a81e8d05ffadbe302bb231cc |
|
MD5 | 1cf3ba70de153148882e4c5ab568791a |
|
BLAKE2b-256 | 6e765aa6e89768ce7d84db58250ce977903d2112e1d3a544793cc1fb14ca8a67 |
Hashes for llamacpp-0.1.10-cp310-cp310-musllinux_1_1_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 536ab3ea6743ae7add5e71ca2896be3228df8bdd1e57fcde8cecda149ce3e7f7 |
|
MD5 | 80178e22445ee02f4e58e843f50acb2a |
|
BLAKE2b-256 | 078b77ab276431867426d098a6e7c25af9db7637d938b7b9d1d92761f4012b04 |
Hashes for llamacpp-0.1.10-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0000d707ad36a1447f7304732eb6cc382470a18ad1c8c33563da23c359d10ff2 |
|
MD5 | 8d61b4a9af3ead572a417f394bed9daa |
|
BLAKE2b-256 | c61015a71072c52cc3b840897345abe68d00c3dce3d40084f85a149aa84db24c |
Hashes for llamacpp-0.1.10-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bc900cc169bb29903fd14870d6af95b38777e7cc9b3e132563a233a87bf21392 |
|
MD5 | 05a53a7903180ec2e7bcf9f4b081bc01 |
|
BLAKE2b-256 | cd3daa0d1ec9ac658c45a59abd91aea66ee9fd5e1362aa1b1960a8bb831156f9 |
Hashes for llamacpp-0.1.10-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d1d1eb4c43124045864e2e5c736cc4a7a84644344b64f778104999e1fe560876 |
|
MD5 | ec7bcf79392d1f8ea1a7e175c05fdd7f |
|
BLAKE2b-256 | 096e4a91e537b6a01ff015dc0e891bd39c2e77a8ecf7c6f1c2dd36330be07822 |
Hashes for llamacpp-0.1.10-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 026c07222f7d94b1de48f24473b90be134dd4efb84fadb3e9a2a1e14ca891b37 |
|
MD5 | 4d5ff6d30f608c355696da67efc4cb0c |
|
BLAKE2b-256 | 4a32491e62b502d474755907717202db9052a8f9fa8b4323ec58713b88ebc7d1 |
Hashes for llamacpp-0.1.10-cp310-cp310-macosx_10_9_universal2.macosx_10_9_x86_64.macosx_11_0_arm64.macosx_11_0_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dbf89876d25bf0d8a89e21eee4a2d7f6589e2d8851d43caa0b63a33f168d42cb |
|
MD5 | 017368c6f3f09db9ca08c7c4d677f820 |
|
BLAKE2b-256 | 1e55a35f80847444a878e741ce1fa8b9c575c83f05d85a7b57bd9f34b8b49efe |
Hashes for llamacpp-0.1.10-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 599c343e8e322b58921f84fbbe604e8ce804b7c6d5095104feed7f37decae309 |
|
MD5 | 2725e90f59eb1d4ff2ad7e4f7e12eda9 |
|
BLAKE2b-256 | 9e1a961249a3a39ffd37878495b270823b8ca31f77e187db96ae629c8ff350b2 |
Hashes for llamacpp-0.1.10-cp39-cp39-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c0f17fdc8fa6d4995463fba0dbf43b08f85aa0ada587faf7f79df661d7870896 |
|
MD5 | 4fdc35fdf723f38dcf735e802399d750 |
|
BLAKE2b-256 | cae5af8ecb1a57c15f8bcc3b5e0e53bc01feaa8a5774bfd53bf70be66f4d37fe |
Hashes for llamacpp-0.1.10-cp39-cp39-musllinux_1_1_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4e5f0158c3e34517ac87e3d765d377e6e8d1105103d056d52c4613aa2608df13 |
|
MD5 | 946f3d09d61ea61ebcb3aeb944a97d70 |
|
BLAKE2b-256 | 35b8b0c26aa9bc2db059dc8a6914ae03c15a576d732a9f510f92ff919fed824d |
Hashes for llamacpp-0.1.10-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bc529af741db40f1f2fdd575ae61b1f185ff240287bceb08e66f292c424e4eeb |
|
MD5 | 39cf4ad5c682c4199f4a87f9bfbaf287 |
|
BLAKE2b-256 | 8a8dde29ae67ece8188402c26aee95bd73d80b07f0d74b71855c9406baff23dc |
Hashes for llamacpp-0.1.10-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ab01b71e41b797c62086168793530eb3b21efc976e43e1daec9e9f35e7811e40 |
|
MD5 | eca3ad54e98f34932503d7eb4a551523 |
|
BLAKE2b-256 | 3953cb64e60afe66b8166d75bd709a3dcc6f953518a76cb814f068fbf9b81e60 |
Hashes for llamacpp-0.1.10-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 69e06cb524b37d4d404ee80d3b7f0120eaa794fad05f1aa4ddefe8edcefa9c8c |
|
MD5 | a050d021e260794920efbff0fe9f3de3 |
|
BLAKE2b-256 | 42175ecb6f05383e9328b5a8e9fba23d2993a5cae6577f174f855b5e041a476c |
Hashes for llamacpp-0.1.10-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 29d79b417588c1494efba5a953f814eb9d2526b34ea94cd256d1ca57a4a2d031 |
|
MD5 | 05661c53e44f5bfa50e1d578c4557fcb |
|
BLAKE2b-256 | 35085c6a143ca9393cfb213c92c37b60ae246ca1f4a71d1b15162f3a165a0300 |
Hashes for llamacpp-0.1.10-cp39-cp39-macosx_10_9_universal2.macosx_10_9_x86_64.macosx_11_0_arm64.macosx_11_0_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 267c5655da3b2b7009814eba082449e4d86e85acfde6ce54e5e12721bc0fb9b3 |
|
MD5 | 436544ea16df463b3c3438bc371bd9bc |
|
BLAKE2b-256 | 336385cac311980d3eeccc568850c4a657dc7bb4c62326ab0a3a9bb09bdbcbce |
Hashes for llamacpp-0.1.10-cp38-cp38-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ddb6d1d24f82c8c3fbcef9fa31135e3522d4d2a66008c5c57d529d683ca84670 |
|
MD5 | 317402fb6d107020d38cdc624df3057f |
|
BLAKE2b-256 | 6e6b7259e80fcee0727ef60bbb6c0ec96bff302c5f2ad3c741f89ad3cdb23696 |
Hashes for llamacpp-0.1.10-cp38-cp38-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9829447a63a40def40ed1c676fca6d0f7ea023d657f2b1d64462b3636ceab45e |
|
MD5 | e662694f23d81ad7df81c4621d13bd9b |
|
BLAKE2b-256 | a494c3daaa8d43af9dfc338a9ed365ace775e6e73928106a3a44e668fa192040 |
Hashes for llamacpp-0.1.10-cp38-cp38-musllinux_1_1_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b314e082ede9e77abc39a88baaad2bf36ef9b37ba8237a5d9d41708db0ed2850 |
|
MD5 | 8cfbb8947e7039f944cc33643dabbbdb |
|
BLAKE2b-256 | 17faaee138bb50c05dd05115c6b286b1547afa249c4d7c172d1a95d9301287e5 |
Hashes for llamacpp-0.1.10-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cd585fd9090d4cb640322580953ce92a8cc5cd49e0798ed97694f89ae8ae72db |
|
MD5 | 6162e66a045d46aa6d0bff16053fa80d |
|
BLAKE2b-256 | 06025b7c1df70987bb7335202114ff0d6d7a727b0af4eb6767fb98f99def4c95 |
Hashes for llamacpp-0.1.10-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c2d94a290806ab3bfe47579c891eae0ba8704d01bdfaefbd45daeb4b2adbcd46 |
|
MD5 | aeef2ae959797c836760cfd6631d09cc |
|
BLAKE2b-256 | 17f4943bf2fe5b91b8953757d201e63a822f521694a7df439d27000482b09445 |
Hashes for llamacpp-0.1.10-cp38-cp38-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1a70ea14008d18f267389f681417bad86dd6ded428a123b26c9c54949812cc7c |
|
MD5 | 54ff3477a708fba033c3d00490ff2a30 |
|
BLAKE2b-256 | c432f0226361be13fc3cd508599579953ba71aff6e64b65bfb921dbed0b6d060 |
Hashes for llamacpp-0.1.10-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7783d05a82d4ed570720e63238306ec4bf8a9660c7397c7efe91a24b43417be1 |
|
MD5 | 0f65c66d031ea368b585d892eae14128 |
|
BLAKE2b-256 | 1dbae4c3bece041989a92653f2bc7d60f2c1bdac8326a73e49ee9c25333cd740 |
Hashes for llamacpp-0.1.10-cp38-cp38-macosx_10_9_universal2.macosx_10_9_x86_64.macosx_11_0_arm64.macosx_11_0_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0024941f22e780446620af7c5b7ca492fc9445ebc7c64651841a68adb7a08aa4 |
|
MD5 | ddf0b5fb1400a2022553cc58cdfc7adb |
|
BLAKE2b-256 | b71e01ca6b016aa13925cd34e5fbe0ca493897d5e9b8fb11575993e08d070bb9 |
Hashes for llamacpp-0.1.10-cp37-cp37m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ef3e34735336da3303c459cde96416440a73be2a07f9826c50df02da803ee082 |
|
MD5 | 4729e4bd35b31057a31724eff47262d3 |
|
BLAKE2b-256 | aa2d85a03cc553458df7b37d81f1dbeb1b8a3ff9a789dc7cbdd9cd2a83b66cfc |
Hashes for llamacpp-0.1.10-cp37-cp37m-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b69828311f06348fc3c19ddac0180bf78d5124494ef755c634551981aaa47495 |
|
MD5 | a53c1f8afcca9040ececf52b0a6716da |
|
BLAKE2b-256 | 90e716adea5a521e0619ae3c12bbb542458f4c08f134feb68b63dd52991f413d |
Hashes for llamacpp-0.1.10-cp37-cp37m-musllinux_1_1_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ee79c7b960ce46df8f7712ad65bc646ce4aeb5c7016a65398211e4c801b97d31 |
|
MD5 | ecf4e7a644c85db840f688c87734cfc5 |
|
BLAKE2b-256 | 9f8589679d33cb4aa5a6cae3b37d0d640fae442fc72e62c08d7b6fa91b22cc4e |
Hashes for llamacpp-0.1.10-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 62ee843d069390746d2ce6555079daa2530f9c81b4f38ea12daab4cf0bb7161d |
|
MD5 | d9997e22180d85eca5836360aaa62e7e |
|
BLAKE2b-256 | 4d2e8d5c8c95ea07cc79f835e25b70fad1c7732582b3085ca237055f55d3619e |
Hashes for llamacpp-0.1.10-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 68e3fb03834988e65de18d2e7c53941e01a42ffde00e59c96a8b32ed539376bf |
|
MD5 | 32ecf7ee42c81c2782ba145f44b2e5ec |
|
BLAKE2b-256 | 17499732bc6c51f7f1bd9cb1a14571273e05b4ceb7e32453ff4afc1606d4213f |
Hashes for llamacpp-0.1.10-cp37-cp37m-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 72b0715193ca1b7498fbaf6375671eeccdfa2a0253987bf584d1989ce00f9b22 |
|
MD5 | cf70c0b4dac6d8648b5887db54977234 |
|
BLAKE2b-256 | f548da4ecd3d9bb9938d12469a0a805588dea6477c3c2ef563a4e2d00a630236 |