Python bindings for @ggerganov's llama.cpp
Project description
Python bindings for llama.cpp
Install
From PyPI
pip install llamacpp
Build from Source
pip install .
Get the model weights
You will need to obtain the weights for LLaMA yourself. There are a few torrents floating around as well as some huggingface repositories (e.g https://huggingface.co/nyanko7/LLaMA-7B/). Once you have them, copy them into the models folder.
ls ./models
65B 30B 13B 7B tokenizer_checklist.chk tokenizer.model
Convert the weights to GGML format using llamacpp-convert
. Then use llamacpp-quantize
to quantize them into INT4. For example, for the 7B parameter model, run
llamacpp-convert ./models/7B/ 1
llamacpp-quantize ./models/7B/
llamacpp-cli
Note that running llamacpp-convert
requires torch
, sentencepiece
and numpy
to be installed. These packages are not installed by default when your install llamacpp
.
Command line interface
The package installs the command line entry point llamacpp-cli
that points to llamacpp/cli.py
and should provide about the same functionality as the main
program in the original C++ repository. There is also an experimental llamacpp-chat
that is supposed to bring up a chat interface but this is not working correctly yet.
Demo script
See llamacpp/cli.py
for a detailed example. The simplest demo would be something like the following:
import sys
import llamacpp
params = llamacpp.gpt_params(
'./models/7B/ggml-model-q4_0.bin', # model,
512, # ctx_size
100, # n_predict
40, # top_k
0.95, # top_p
0.85, # temp
1.30, # repeat_penalty
-1, # seed
8, # threads
64, # repeat_last_n
8, # batch_size
)
model = llamacpp.PyLLAMA(params)
model.add_bos() # Adds "beginning of string" token
model.update_input("A llama is a")
model.print_startup_stats()
model.prepare_context()
model.ingest_all_pending_input(True)
while not model.is_finished():
text, is_finished = model.infer_text()
print(text, end="")
if is_finished:
break
# Flush stdout
sys.stdout.flush()
model.print_end_stats()
ToDo
- Use poetry to build package
- Add command line entry point for quantize script
- Publish wheel to PyPI
- Add chat interface based on tinygrad
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for llamacpp-0.1.7-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1eda6ca61a59be9fa52b9a405a309b4c28cca2da3374f44924802d50fe6c2dd2 |
|
MD5 | d360481abf949ab7c92adf538c72201c |
|
BLAKE2b-256 | 99dcca414eefb56f9f71fca0e81743082cc160b75488fb560be5cc2439891278 |
Hashes for llamacpp-0.1.7-cp311-cp311-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 880c45c8b8da84044b5d10e94bea289afe211960d57ff4aaf23ec9d3fcdc0774 |
|
MD5 | 98cb215c6a8e99864ab2b3e2b1f32810 |
|
BLAKE2b-256 | 64c6968fc2f492bc7f3eecad2ffd088069af1ea16567501fc6c1258823e1d05b |
Hashes for llamacpp-0.1.7-cp311-cp311-musllinux_1_1_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a9182035b3a3c744a353d85af7220e42671fe5e46230df0b14b5eca59fae14bc |
|
MD5 | e3fb4a9dfb69b74cc65c9f3d1348cc91 |
|
BLAKE2b-256 | 481049dfc3925b3a2a3a06bf3ff2c2b3c2d0c9d50dcd19a5cb243fd0505d2f40 |
Hashes for llamacpp-0.1.7-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 022af777b9e9280eeae617d3193b3a2a28a1b5db755ad4978d206c037ca71ccf |
|
MD5 | 60e6e412ffcb92f3c20b9fe29b537bdc |
|
BLAKE2b-256 | 5f800ff426339edce31134fba12318af58ae33c84130b69c7b591549f8e67624 |
Hashes for llamacpp-0.1.7-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d4c49b0336833a1828c6398e08064a5e271ad0cb777ef28c8fb39b3b14e23a8d |
|
MD5 | b8ff793a5eccb139ac3e21c417e0e3b9 |
|
BLAKE2b-256 | 8c8629dfa239bba793f896243d3500331dbd8280e92aafb6c8ef2806dea48b7d |
Hashes for llamacpp-0.1.7-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 021e5f9eaf8720ae45618a38212672232949cafa289d9fdbb99a911b92996262 |
|
MD5 | 289913b36970b97812544a41a47e9701 |
|
BLAKE2b-256 | 2b512fe579d5e26ca27b2df68a2bf4ed77cf155322c20ece1e73cd89e2e7aace |
Hashes for llamacpp-0.1.7-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5f415c176a5581da6bcb6b81a35f7e3cc5b5647132b1a2102194e1086f0b396c |
|
MD5 | e055c3fb6afe2c03a00627e737fc79e2 |
|
BLAKE2b-256 | 5b1e34a86b08ad0a7740f5ad33c08253e5f33fc87979a985c1d6a5e4ad09a0db |
Hashes for llamacpp-0.1.7-cp311-cp311-macosx_10_9_universal2.macosx_10_9_x86_64.macosx_11_0_arm64.macosx_11_0_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7dd26672ea9245be028b5c80f52801fee89c1772141f835c1da943c90ca52a00 |
|
MD5 | 4ae692bb9dda76d71bb86b804a40381c |
|
BLAKE2b-256 | 4dd4f0516e1c0ff1819e823a86deb245a61a8c7ed2d4689471eca0e4197f8a9e |
Hashes for llamacpp-0.1.7-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2ad26d003495e709274dc44d17b2e3e011404cae461b2146296a29dde599abb0 |
|
MD5 | ea5fcf9aa9813f98c34afd1350bb48ff |
|
BLAKE2b-256 | 9070f1b72dcaedad7e9c55c2cf91e353cc30c81766dfaffed035ef9bb9964117 |
Hashes for llamacpp-0.1.7-cp310-cp310-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0e28c63569137276fa93c554167fa8fba3f6821f57ad3c6d282d65853b037cb9 |
|
MD5 | 6512728fa321df52fb1b23d9356429cb |
|
BLAKE2b-256 | 72755c9a5b5142cb92dc078de6d3ce2662460febbad58c1b9fa0a908dd781762 |
Hashes for llamacpp-0.1.7-cp310-cp310-musllinux_1_1_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 937fbcf5c124be3abda402f674dd15369ba285991e49b8ca6cb021aced27ba92 |
|
MD5 | eb72456616f0ae64ac4ed08c9aac6ba1 |
|
BLAKE2b-256 | 1315e4bacc9349d681b55e3ae597da3fb045b859b618e881c1e30d590d00c0e6 |
Hashes for llamacpp-0.1.7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a1a993881292098b938d1dd638309c31e877a0f9d0274784a078bc39ab01abea |
|
MD5 | 7eede8142a4f094ef3d9ef381a3dd2c6 |
|
BLAKE2b-256 | f4280e4ddd762311dcdf7d4bced8dfef06100bbdf671d5488e7538b3990cb35c |
Hashes for llamacpp-0.1.7-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ab2bde8bace56b7e0cd70f715c3f64322d799eea58fcb42ee20fe59bf9a30808 |
|
MD5 | 1061d51207299f45fb8edc7cfa8bf5a1 |
|
BLAKE2b-256 | 063b28927bf44f8e9617b86a7ff9798e89f650be0ba255023f8fc21e1a54aa3a |
Hashes for llamacpp-0.1.7-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a25c1fca292059d4000224e1f17e11c774569f5346b7aa3dd75c53ae9d7f5e46 |
|
MD5 | 00e5954602d5ee67a31da969c56c4add |
|
BLAKE2b-256 | 8bbbd175108106a0817262b7e47aee5f11e116cedb59916c3085761213378540 |
Hashes for llamacpp-0.1.7-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3dff8b857b81b65d35d686b670a2534bdca6f02e41e01ecc8ffa9ae4b1d5e3e9 |
|
MD5 | 7d31e1373c9a1ba5569031ed6062f1a3 |
|
BLAKE2b-256 | 79998eca33a9083bafd1da2183a3dfc1ae99d1326923c46dade4ef2dbafebd56 |
Hashes for llamacpp-0.1.7-cp310-cp310-macosx_10_9_universal2.macosx_10_9_x86_64.macosx_11_0_arm64.macosx_11_0_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 537eef39b55ee57fc9c50fa228c1f45297ef3055a70853bf1ccf94b28e64a6eb |
|
MD5 | 870fcf5aeac7ffe0daf01f7f22de9051 |
|
BLAKE2b-256 | 1b02e520ec94f42dbf2c2e0261536a876eeb0664d291a936029179b78fe20645 |
Hashes for llamacpp-0.1.7-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 83ed5719cc945f3c1365a78c3ff23c6cd82a16f46994eba27f91d5ff7139070d |
|
MD5 | 2d9e0c0eb029d7a6f2cc1c64e2dc0836 |
|
BLAKE2b-256 | 286640d67542b309a556981fc0f71390588c3267b1757a1c87b400fe6e919266 |
Hashes for llamacpp-0.1.7-cp39-cp39-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e1aa57296ce6ad8b59ea47969434147ea93114247a49275ccd02d301c9d67fe4 |
|
MD5 | fcbeb9a122643b54d3a83ac89311c0dc |
|
BLAKE2b-256 | 1dfa58c6793fa5288d802f6dbf97ca75060205e63deff3e92f829b6200bf6a30 |
Hashes for llamacpp-0.1.7-cp39-cp39-musllinux_1_1_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2cbdb2e732cbe70a6cf9aac5077b8fe53004eb5c62f58c89379482be31c4c9d2 |
|
MD5 | a05e892744b08d38f03c898cd0fe48e1 |
|
BLAKE2b-256 | 6c862c3ca93503e8ed511eecba30ff9548031856f31754126b3cfb37cd23eb3c |
Hashes for llamacpp-0.1.7-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1b6fe5ecf8ed9c7bff417b84152828da032510fe9403f38269949831f7da8a9c |
|
MD5 | 2f6f41ca0a0db40d706c4d8b6194ef3e |
|
BLAKE2b-256 | c09e8e631262276b2b5e1c0e026672dccf37a6fba45bacbdf87b76df462299a4 |
Hashes for llamacpp-0.1.7-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e2bdd0ba91226aeea4bec14de84a5815ce7be3f135910d14c6c2a48cb3423825 |
|
MD5 | 3817809fdee5f13dac07fc3973d6fb96 |
|
BLAKE2b-256 | 707592bb97648c4348346d1a449a3034a496e578c63a939ee61d6802157b1a12 |
Hashes for llamacpp-0.1.7-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b0e19267f73f883d3ff04725dbb40dcbbc0c519601ca18e138d84e6a42a4a87b |
|
MD5 | 00b248d509bdf2551b71d102014a1b5b |
|
BLAKE2b-256 | 6e57bb096bd6e04d8715eb36699d9e295209fc7952f36208c2e320665d2ec9bf |
Hashes for llamacpp-0.1.7-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a81b7e6029a1cdb95b6413426376ee279d6523a08ae1d370f2b3ddae116078aa |
|
MD5 | 79dddf5de3d8e282bf1fd3a18d4d74f0 |
|
BLAKE2b-256 | 1ea41a422dec88df8777394f20fd3c0eada0f3ec24c1defb8f640116072787ae |
Hashes for llamacpp-0.1.7-cp39-cp39-macosx_10_9_universal2.macosx_10_9_x86_64.macosx_11_0_arm64.macosx_11_0_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e05d15fd664888e793f6e75b30969967a03b326baae4239d208a163333f5016e |
|
MD5 | 2f035b8cf7b788df57c9f9f2c644028f |
|
BLAKE2b-256 | 333f802bc8e5fb76b554a2086c029475c7a326cce78fd835d48cc6c6f37177ec |
Hashes for llamacpp-0.1.7-cp38-cp38-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9396f1228431c03e4042e33afb1ad2c3699d90705342de1cb47a8271e9638735 |
|
MD5 | 102c0276589227f7bd6207d80d9fb145 |
|
BLAKE2b-256 | 42fa0a745917f381558bceb238fb7743dd9d59a07fd2c494ca6a6d11abc8c24e |
Hashes for llamacpp-0.1.7-cp38-cp38-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7d8ac64f2957a472dbe7de98082a43fb1266658a9c5ce0b79f5abfadc694202f |
|
MD5 | 3fb5b30eebc0206367be4dfc7fd66d86 |
|
BLAKE2b-256 | 1b39cdd5a4e8601fd88e4dfa92f41eee586554cc3b8dedc68fbe0b5eaebc9a6d |
Hashes for llamacpp-0.1.7-cp38-cp38-musllinux_1_1_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 15b0dc0cdd6e4cf3708527562e622010d80796a206e08cf12bb2583daebe5c47 |
|
MD5 | a805c7c56f029a34d2aaa61f2693b77b |
|
BLAKE2b-256 | e084d5231f353d374b444ab50fb36c04e0ac3ab83c8c847c22315890ff561ade |
Hashes for llamacpp-0.1.7-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8031b2ba2d72be568bc4afad395f54578bf360d25b5cfff0f25283fb287be307 |
|
MD5 | ddaf32acb9c95245aa5009a7e33ee6d9 |
|
BLAKE2b-256 | 073a91ba0a93cd3e8b03d08d90f85c0a1a819a8366871e276ca8e71f7d262fcd |
Hashes for llamacpp-0.1.7-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 885ebe692ae8f0a1ab8e97ce55c6d0ca709c17247db9bac2c3d971ed849ed0e3 |
|
MD5 | 819d54e8ed164a803796d6f22e4650d5 |
|
BLAKE2b-256 | 1c89008eae17d92e17f3c528570b36bdf170d9a0a819154cdbb00c3677c2b107 |
Hashes for llamacpp-0.1.7-cp38-cp38-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9c967e829b6644def20cac8dac1170f3c8ef137a4e4cc123a71829d316d9e686 |
|
MD5 | 116cf68687f5179f464231a98dfcf384 |
|
BLAKE2b-256 | 228e38cc75c98d5eefdd9ebd69df86c75c5981690327292943216a59d1e87a5f |
Hashes for llamacpp-0.1.7-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f907fb380b9028507057fe3edba554900bb8e3c41c179063e7432b410b9f2b2e |
|
MD5 | 544fe34d42e6553c4e3c23e56ad9a455 |
|
BLAKE2b-256 | d29b789ebc3a93d81297a60de955e0c9bcea8e02d0fda6f21e51899d5d72cfa1 |
Hashes for llamacpp-0.1.7-cp38-cp38-macosx_10_9_universal2.macosx_10_9_x86_64.macosx_11_0_arm64.macosx_11_0_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6fc6f18176084864257b67c08dfa8bfdb3ff8061868c571cc2958589cd191e43 |
|
MD5 | 306883c4658b79aeff2352d2a680975b |
|
BLAKE2b-256 | 9a41bc4b35abbebbb4e89c776168bd1d47ddc6cca70090c766d6021599dad685 |
Hashes for llamacpp-0.1.7-cp37-cp37m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4cc53b40b6b9aa1e38c9349a20e0430d70022b0c2f633ff032679bdf7eb369b0 |
|
MD5 | c5d018bf0eab027ffdfb911caef10428 |
|
BLAKE2b-256 | 025ec0a27c336b3c696ef15c4ce900e32b9a91b2acbd345cfda50fd95510a6a8 |
Hashes for llamacpp-0.1.7-cp37-cp37m-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0910f87bcb05f87f1b54da28ca8752b295224ea949371014b00b865939307403 |
|
MD5 | 052e9db4dcc54f1cdb8e5a08869001f3 |
|
BLAKE2b-256 | ee37ec0aa69da35fb67272845d607a006df4de9d7f2ca0cbeff9f376b9f831de |
Hashes for llamacpp-0.1.7-cp37-cp37m-musllinux_1_1_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7ec4faad3aa643df13f21deafad441be301a8edb55ea32833062d8fe9bb86276 |
|
MD5 | 3ce73faa712a02cd1b146f628591fb47 |
|
BLAKE2b-256 | 626bb69c95f1137cd3a34cbd4e040249131beb5ec4de842b0fcee04a33de2f42 |
Hashes for llamacpp-0.1.7-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 195934e2b11a20bb9c5f21ea5478dd3de8b348a509563ac634bec2cc9b2f1434 |
|
MD5 | 3bb616fb0a7d7b8cfacf238926224533 |
|
BLAKE2b-256 | 9e99ff138d743e960a7084028e385e698d32fe164bea7ce5e4d081efddf65bb8 |
Hashes for llamacpp-0.1.7-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 494a931722667bcdc46c85a60be0cb4c8f050a1b796b1edbc8c3b62edac92202 |
|
MD5 | bd7de241ad15b2e8eef22266adea2f8a |
|
BLAKE2b-256 | bc8517eab552da8b7530363cec8133495a2cbbe075b15e9fd9fc2464c194d503 |
Hashes for llamacpp-0.1.7-cp37-cp37m-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a66ab14ee48b67435ee699746502327ffd52f5057d317f76b6a25d110a68ad7d |
|
MD5 | d24bf9f70b478e2bede8fb9ff50cbc04 |
|
BLAKE2b-256 | 8f97d50014de84c43e4f6473fbfde9b9542bfe3755f924fc65ef86b902ddeb71 |