Python bindings for @ggerganov's llama.cpp
Project description
Python bindings for llama.cpp
Install
From PyPI
pip install llamacpp
Build from Source
pip install .
Get the model weights
You will need to obtain the weights for LLaMA yourself. There are a few torrents floating around as well as some huggingface repositories (e.g https://huggingface.co/nyanko7/LLaMA-7B/). Once you have them, copy them into the models folder.
ls ./models
65B 30B 13B 7B tokenizer_checklist.chk tokenizer.model
Convert the weights to GGML format using llamacpp-convert
. Then use llamacpp-quantize
to quantize them into INT4. For example, for the 7B parameter model, run
llamacpp-convert ./models/7B/ 1
llamacpp-quantize ./models/7B/
llamacpp-cli
Note that running llamacpp-convert
requires torch
, sentencepiece
and numpy
to be installed. These packages are not installed by default when your install llamacpp
.
Command line interface
The package installs the command line entry point llamacpp-cli
that points to llamacpp/cli.py
and should provide about the same functionality as the main
program in the original C++ repository. There is also an experimental llamacpp-chat
that is supposed to bring up a chat interface but this is not working correctly yet.
API
Documentation is TBD. But the long and short of it is that there are two interfaces
LlamaInference
- this one is a high level interface that tries to take care of most things for you. The demo script below uses this.LlamaContext
- this is a low level interface to the underlying llama.cpp API. You can use this similar to how the main example inllama.cpp
does uses the C API. This is a rough implementation and currently untested except for compiling successfully.
Demo script
See llamacpp/cli.py
for a detailed example. The simplest demo would be something like the following:
import sys
import llamacpp
def progress_callback(progress):
print("Progress: {:.2f}%".format(progress * 100))
sys.stdout.flush()
params = llamacpp.InferenceParams.default_with_callback(progress_callback)
params.path_model = './models/7B/ggml-model-q4_0.bin'
model = llamacpp.LlamaInference(params)
prompt = "A llama is a"
prompt_tokens = model.tokenize(prompt, True)
model.update_input(prompt_tokens)
model.ingest_all_pending_input()
model.print_system_info()
for i in range(20):
model.eval()
token = model.sample()
text = model.token_to_str(token)
print(text, end="")
# Flush stdout
sys.stdout.flush()
model.print_timings()
ToDo
- Investigate using dynamic versions using setuptools-scm (Example: https://github.com/pypa/setuptools_scm/blob/main/scm_hack_build_backend.py)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for llamacpp-0.1.9-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9ef100c7c9ae231f9707c47fe8a95fb746616c04e9bafc349f5cad39bcee7dd9 |
|
MD5 | eac4598c1d0ff721ebc8acc00a17c2f2 |
|
BLAKE2b-256 | 50d68d347293ecc162d5313bb175d5c2d0845b3f0e492d43f23e4c7c7bb4cb3e |
Hashes for llamacpp-0.1.9-cp311-cp311-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c52aefe81931f14c8565c0ec9a25386294f14e179d39450336ed670ed76446bd |
|
MD5 | 660b90ccd3a0dd0984de17f34ce0db51 |
|
BLAKE2b-256 | 89d0ea124ba002373fde60d0b96add4ff677e22ba3de4a3c35bc79d11fdb8e06 |
Hashes for llamacpp-0.1.9-cp311-cp311-musllinux_1_1_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e4b1d6f595f2819ce23377f0e00a3a8fe67e7eda02dfb33ce0ceceed3f534f60 |
|
MD5 | ea749d1621a8a5261f2bc419f5d69753 |
|
BLAKE2b-256 | cc997b72cf7cca8d591ecf537c8e32271ed3cc87b57966128ccdd8015d51f5b2 |
Hashes for llamacpp-0.1.9-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 40cd2878e216b83e09645f5b54f39f4f1d6268e54c73a7c3a1c9ff7cc469363b |
|
MD5 | bca2545b6408b6d3131348ad90fcfadf |
|
BLAKE2b-256 | 503770f804cddb3758b0ce980f986ceca45c8c73ee6fd3d77d5cc9b9b6ad18c7 |
Hashes for llamacpp-0.1.9-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4d19d780727f06b48dcc408ebf1f3ae9386330e615a97392b8320aab94229986 |
|
MD5 | 3abb55445accbc3bd05e2b4c5f349cc4 |
|
BLAKE2b-256 | d436fac0fff66f9753ce1b8514a1b8cd04194bcf0f10712e6007529ae7f9f613 |
Hashes for llamacpp-0.1.9-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5e4e86864139bb1da180e53612866cc13fdbda0494c58d9df729282337d63fe5 |
|
MD5 | 1b23eb90911fcdb1f0d01b18f3b0a363 |
|
BLAKE2b-256 | 9d45f194389df50190359c31ebb95e4000f8fd505e041beeff8472914f7a5ea6 |
Hashes for llamacpp-0.1.9-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 703229084ef4b4deb3bd018745b6d0a0afa3e8a42469057db61ca6d2b95edaee |
|
MD5 | c9803deaff095cf4fc85fe04a5327cc7 |
|
BLAKE2b-256 | cf37d462c82a99bae5681338ae6da80fada23f3f728d7b122f328250d78ef839 |
Hashes for llamacpp-0.1.9-cp311-cp311-macosx_10_9_universal2.macosx_10_9_x86_64.macosx_11_0_arm64.macosx_11_0_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fa111f6be70458c8484222734de4e5553e6ef2a14f0f5a547a2a861f04a25ca5 |
|
MD5 | 0022154e803895f4a8eca80c6164efbe |
|
BLAKE2b-256 | a6996f14b930e20ed0bd32514a1cb6fc0535b4b870986bea4accb9c8d82c14bf |
Hashes for llamacpp-0.1.9-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4be06b553723fc3ea43f6b1183157a6efb256e3e1f4e75cdaa81869703b22f4e |
|
MD5 | 48e5fcc07282ba7ca1d81a21234b3bf6 |
|
BLAKE2b-256 | bc111f81caccc67944474edb96c546c213bc9a431c6bdc318ca376854884427b |
Hashes for llamacpp-0.1.9-cp310-cp310-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0bfb1333dddaf074aa721084e4cdc1a2196a35041da6298edbcd87f4e9d2a893 |
|
MD5 | 4a71bd9226eb1948e5eac857c7f9dbf1 |
|
BLAKE2b-256 | 2722195e5fab32289466ab5f023cf7777b05423ad7699f7de327b5c5a9455ae5 |
Hashes for llamacpp-0.1.9-cp310-cp310-musllinux_1_1_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ba83067a5128988578a1bcce1ae4d96f3aced2dcf10c5e10dfb6d1d55fa3e62f |
|
MD5 | c9a896c5c359f8c8309dc97a9c6c3977 |
|
BLAKE2b-256 | ef69edbe3feec26f1e2e6e98fa33d9409fed398918f90df5f586235daeb21fca |
Hashes for llamacpp-0.1.9-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cf888805b1ba80e6ec5008a32480f45b976ff62b3cdb74c05d9409f112d0f52d |
|
MD5 | a9283b134232a18bd8d754b721c5ccc3 |
|
BLAKE2b-256 | dda97168f24caafc501e3f488a9c349afe43e311659c82b1a3124496129bd67b |
Hashes for llamacpp-0.1.9-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2f3dc9dc65b0f6fe8bc0b25e18b6f943d8306291559e2370633052dd648dd0ca |
|
MD5 | 7736e65aa7528bb09108bb50689b25b7 |
|
BLAKE2b-256 | d50e2f03ec073dd1c875d74d28911fbbe027b6677b0da3ebb71974b588a413f9 |
Hashes for llamacpp-0.1.9-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | defe542b4c2acbd63377bb7e43667a00060bfb058cf749911cd826ba2f90ca30 |
|
MD5 | 6e34f9dc9276a7cbdf9a9ae87a109b0d |
|
BLAKE2b-256 | ca4b56c49e03a71e967e799cdd0e0acd1a3a3c6ac200605947c9f5c0e679d732 |
Hashes for llamacpp-0.1.9-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0f0c9b6678f605a8ababb5f1b48c76293d588b258d8f8c9def27ff5342dc5c69 |
|
MD5 | 028c5a9fdd54fcef7dfee6be082f9587 |
|
BLAKE2b-256 | 6838c02ec00e37fb711bfa08a69ee6f808c5d5f0853c3d25fc4c5cca4fa361ef |
Hashes for llamacpp-0.1.9-cp310-cp310-macosx_10_9_universal2.macosx_10_9_x86_64.macosx_11_0_arm64.macosx_11_0_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f9c56f8e2c865c9721a363bffc36b24c39ea7b6bc221bcb5f6c86a0b09f240e2 |
|
MD5 | 04e91f2f3fc130877975b5822063b0e6 |
|
BLAKE2b-256 | ddd425d43b486cec266e7225b28680f01765e885e81e2bf51f96d41747cb8c11 |
Hashes for llamacpp-0.1.9-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d5bdd68f663ec2b9a2a2b3c9546ef1a8ec5681e10ffd7e14fc46a8dd63a5a19e |
|
MD5 | e744b537d2e71ef64232740783581f1f |
|
BLAKE2b-256 | 445d6ec9b816b537e372613bc46bce6702f2912dd8f097396118c41049253e4f |
Hashes for llamacpp-0.1.9-cp39-cp39-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ce5c84bae65d3d038252e823ef21429144ead73c2e89339d1bcfa7ee1c1b7fb6 |
|
MD5 | 06c43eac962c3612fd798d943a03a8e9 |
|
BLAKE2b-256 | a1f9499d0831fc70317d5a354ec84b80f8e8cf18a98c8f3961f94bc09cc9e7c5 |
Hashes for llamacpp-0.1.9-cp39-cp39-musllinux_1_1_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 65fab3d10b0698e4929d2ebd00ed788f1bea6c3c30b65c9366764259a6313b19 |
|
MD5 | bb7afb93edbb783f3b9d537069004998 |
|
BLAKE2b-256 | b2f3196c7379382fcdd07d1f062707468cc1e17e680b277b50bc162f4ee129c1 |
Hashes for llamacpp-0.1.9-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d33f0bdfba5fb9d5a046dbaa79ae66f5bbc1f1a093db377b5839e428160266c8 |
|
MD5 | c2ec2cf249c74149b00616b3d420918e |
|
BLAKE2b-256 | 6b1ca0ba472855226bc86be724c3e5ce4b3a104b8b7e7bd8cf6307acb13701b4 |
Hashes for llamacpp-0.1.9-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f28ebd8183ea4ed287839aa920f2ac6097a81c92b24d590cbff28a5c8e273b90 |
|
MD5 | 9487572d6acdb44681bc461c85f26a9b |
|
BLAKE2b-256 | 758905e46a9884e7eb0d757e452368ff9da7eab65f3c47bda4c5198d650a2d7d |
Hashes for llamacpp-0.1.9-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3e7f7e6907e509847b38884575a1143cc42818bd5ecb731dccd039c10c6f765a |
|
MD5 | b24ad248affa5e8ba5671a7ff3c553e0 |
|
BLAKE2b-256 | 02ca68aa6ffc481b2f6533e241fb568c2cf8cda7a34be60597f29b7757e1297f |
Hashes for llamacpp-0.1.9-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 14bc6110b0a12b3d2efa0921e1d94d48e62daaa99572112fb4743dc5fbcb8da3 |
|
MD5 | 34760698afb4b2491a3efdce03213bda |
|
BLAKE2b-256 | 8cf5ec960009060535aca54793d7c1442e17f90f8bac936391be90135358f96d |
Hashes for llamacpp-0.1.9-cp39-cp39-macosx_10_9_universal2.macosx_10_9_x86_64.macosx_11_0_arm64.macosx_11_0_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a70ba5006b6fda45917b540957ae0f2080325f918aad0dedc245adb869737204 |
|
MD5 | 086addd9b8e5d69738f850c0d160a973 |
|
BLAKE2b-256 | 18be6769741ca226a1cf900fea280a1ecdc65c0320311d9831f567a071b9cf01 |
Hashes for llamacpp-0.1.9-cp38-cp38-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ed6cc36ed8d4e987efe31fd111b6a8504626ee6fa4eaad3c4da733d3185bd1aa |
|
MD5 | b1638b842a506caeb599f7a59f73cf88 |
|
BLAKE2b-256 | 3eaa9a092d51248f5a3bd42f49b5321ffcea2363c3399140f69e9643c9c8dce1 |
Hashes for llamacpp-0.1.9-cp38-cp38-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 45f15f1f9d2d22f5e86d8119429cfa0cea502e4133c6c50ed35a5910ef9777a0 |
|
MD5 | 9e09a51dd3c6ae00b2635620c430f716 |
|
BLAKE2b-256 | 34ee38ee880031bc375eb5a4b5ef1e9c8e20a42a4b2a786bbc69bdd22683d1dd |
Hashes for llamacpp-0.1.9-cp38-cp38-musllinux_1_1_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 47b87b9e6122b89e522f1a7c8452a17d118870b73fc0e72ddcda2d0f54929644 |
|
MD5 | d3d337bbb59c6cfcfdbac130a5c9df66 |
|
BLAKE2b-256 | cd659aaa6ed2a9d4fd2c8130a45cb140ab866d9448c4f221e7e04220ffb54070 |
Hashes for llamacpp-0.1.9-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b1fef0d031df52e8de67fb962e657288abd91eee8ef491fb80869cffd95033d9 |
|
MD5 | 816f5640e55b21f681ad595cb79131da |
|
BLAKE2b-256 | d889262d3f8fb712c2a49eace0173e98c2d68fa2e87dea1b2b835f380adf9e01 |
Hashes for llamacpp-0.1.9-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d4777949f614e7de5b9e9ca28352620316b115dfef0f0a8a01173b01ba18100d |
|
MD5 | 57a443345568d124986dc135e4ad1035 |
|
BLAKE2b-256 | e538db08bc82e2a027be47514072058062d8aab496ea097032f521aae9a69e69 |
Hashes for llamacpp-0.1.9-cp38-cp38-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d119bea2397f161d3ce34bebc3291429ac692f0c28f4bdc18a4e3557b0da9d74 |
|
MD5 | 66dbd6b921564a8ec76623f1357ed838 |
|
BLAKE2b-256 | 888a7c0ff2aa41bb2ffada9ef4f69937e2ab891fa50885bdd6a21a56c325c7a0 |
Hashes for llamacpp-0.1.9-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 084bc9f836bc412f48c40a229ec2961e3b422af2fc3537edae05c2bcf22b498c |
|
MD5 | 188ee1073409274dd12dd4d13c512b8d |
|
BLAKE2b-256 | 70371dc392877dd6d5d42649572e0d4c0400e72eafcadf34e06bdd78986d4e36 |
Hashes for llamacpp-0.1.9-cp38-cp38-macosx_10_9_universal2.macosx_10_9_x86_64.macosx_11_0_arm64.macosx_11_0_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3e6829a4d53ed6c5af5a10fbeee582960c6c33e2cdcf9e7933490574b08a8e80 |
|
MD5 | eb7373555a22e2e1603f1ad3641fb371 |
|
BLAKE2b-256 | 6089001b513a869102e56fb98d7021221633d6d50047b281590e55a0c4915a0a |
Hashes for llamacpp-0.1.9-cp37-cp37m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 35da4c4c9e4b54b9a8ec9881c5cc6d8ffc90ea5b8ab015ecf7f63e39fe13f71a |
|
MD5 | e86d7435b3cc85b974a279b7b7d1b044 |
|
BLAKE2b-256 | 768ac29e9c6a73a9cbb7a07d5605641e5e18bad289c6063392e62bfc539a36f9 |
Hashes for llamacpp-0.1.9-cp37-cp37m-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2cd682a973a6d5ca44241134093530e2faffc5ec20a5cc1a2d490c4c42e88d8f |
|
MD5 | b5aa7289ab73fde8e567db8b331b8145 |
|
BLAKE2b-256 | 48c855b5b2583cc9b43c2aa5cdff3cd738dc27e0082cc1c198caa92dd4b73708 |
Hashes for llamacpp-0.1.9-cp37-cp37m-musllinux_1_1_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9d20d6eb6427f66de58e67f27e8439ef2a2145b072d803739bceca98232a3310 |
|
MD5 | d043f4fe8cbc5a1ea1d0c1688edfffb1 |
|
BLAKE2b-256 | 1a51d32df37065493fcd3fb1d797b5a8deb7d8736c36801223f9771d1497975b |
Hashes for llamacpp-0.1.9-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f980b651f3d4b6f966273eac196ec54e34236edae07a7a2fe0f75b02efe3ce71 |
|
MD5 | 3f2d6581e6d899e2170ef2ef6bbdde83 |
|
BLAKE2b-256 | 4f13d03d99d421e817b70ec8d4e9dfa6cebdcc6c5bfda0a0dd58572ef6972967 |
Hashes for llamacpp-0.1.9-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7d6bdd63eb8cd01514a75cb70584933fe3252f09f28701caf3207c589f4ac638 |
|
MD5 | 12a4ac657aa02fe68bb9768e79469c7e |
|
BLAKE2b-256 | f7ee4fa04362491db999185cbba92998d9330b315be9f0e87b8887ae736efa70 |
Hashes for llamacpp-0.1.9-cp37-cp37m-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ef054429bc879385d481e5fdf8b386142dd07ee9acc16323f0263c9daa266460 |
|
MD5 | 392ebe580821afb85b28aa0aa80ab53d |
|
BLAKE2b-256 | 742c2b8b5bdc0d0ce42366668b9e7c477533bade7077889ef44e09ee36fe163d |