Python bindings for @ggerganov's llama.cpp
Project description
Python bindings for llama.cpp
Install
From PyPI
pip install llamacpp
Build from Source
pip install .
Get the model weights
You will need to obtain the weights for LLaMA yourself. There are a few torrents floating around as well as some huggingface repositories (e.g https://huggingface.co/nyanko7/LLaMA-7B/). Once you have them, copy them into the models folder.
ls ./models
65B 30B 13B 7B tokenizer_checklist.chk tokenizer.model
Convert the weights to GGML format using llamacpp-convert
. Then use llamacpp-quantize
to quantize them into INT4. For example, for the 7B parameter model, run
llamacpp-convert ./models/7B/ 1
llamacpp-quantize ./models/7B/
llamacpp-cli
Note that running llamacpp-convert
requires torch
, sentencepiece
and numpy
to be installed. These packages are not installed by default when your install llamacpp
.
Command line interface
The package installs the command line entry point llamacpp-cli
that points to llamacpp/cli.py
and should provide about the same functionality as the main
program in the original C++ repository. There is also an experimental llamacpp-chat
that is supposed to bring up a chat interface but this is not working correctly yet.
Demo script
See llamacpp/cli.py
for a detailed example. The simplest demo would be something like the following:
import sys
import llamacpp
params = llamacpp.gpt_params(
'./models/7B/ggml-model-q4_0.bin', # model,
512, # ctx_size
100, # n_predict
40, # top_k
0.95, # top_p
0.85, # temp
1.30, # repeat_penalty
-1, # seed
8, # threads
64, # repeat_last_n
8, # batch_size
)
model = llamacpp.PyLLAMA(params)
model.add_bos() # Adds "beginning of string" token
model.update_input("A llama is a")
model.print_startup_stats()
model.prepare_context()
model.ingest_all_pending_input(True)
while not model.is_finished():
text, is_finished = model.infer_text()
print(text, end="")
if is_finished:
break
# Flush stdout
sys.stdout.flush()
model.print_end_stats()
ToDo
- Investigate using dynamic versions using setuptools-scm (Example: https://github.com/pypa/setuptools_scm/blob/main/scm_hack_build_backend.py)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for llamacpp-0.1.8-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9cf398e597268ad897fa97c96179ee0038cc35a42e96c6bcca79466010235f00 |
|
MD5 | 8d038c8c7466d0009301195f0f5514c9 |
|
BLAKE2b-256 | 3eca9d84de966cb711f6f447807b35c4de539d0a6668d9bf30894ca5c6c90258 |
Hashes for llamacpp-0.1.8-cp311-cp311-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2488adeae5896e66cde15e5ea8abeea17559af60eee5905270434a58a9fd3d1a |
|
MD5 | 92f0fd7f805369f7df6899278d226bfa |
|
BLAKE2b-256 | 4c1bb9085c8f6cf9b357ff02d8d6a9989dd70af4a7ad67c7b393a406e4a653f8 |
Hashes for llamacpp-0.1.8-cp311-cp311-musllinux_1_1_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bddf855f227792852824d2ba6a8494113b23b4cb3744f6b92c1f85f5dc57ebb5 |
|
MD5 | 0907bf49d3a73e31fdcbb79bcbeae6ea |
|
BLAKE2b-256 | b2162bc862b9b8fdefb147163a8687774c17c6cebe245767c8285c67b51c5096 |
Hashes for llamacpp-0.1.8-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 11e479876cd68432ed977c81b980dcd377e10dacbce2093c67afcb49705d4d16 |
|
MD5 | 9dd72706ff92e74237f13d327fbd661a |
|
BLAKE2b-256 | b5fb27956a6d05729cbcea651021b48f86b5a42f9bc6c5bfaa9a61896ea815cf |
Hashes for llamacpp-0.1.8-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f1fce172a36e31cb50f71523400a4ca1ea7142ab25411b18f65818e22acdfb52 |
|
MD5 | f67f436fd56eade4ae5e5f64a9161392 |
|
BLAKE2b-256 | 02ea9a165730b5d48d181a255eda27757fe32958ea0f31da35606aa1fbfcd107 |
Hashes for llamacpp-0.1.8-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5c3280dd2ce30134eafbc41786a8e0fb22e13685501e9669ac7e3c4bc97d9e16 |
|
MD5 | f9f04d039ff32d4ca95cfc74a22da80f |
|
BLAKE2b-256 | dea011ba49257953763bb2f58bc392521df9956c1dd690595e99b68daa6b629a |
Hashes for llamacpp-0.1.8-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a67e04e82ff11da979ecc724b48042116fac047ea5e13c233e90a774ba7fd93a |
|
MD5 | f8910e8aef6f3f197ace89bef891b044 |
|
BLAKE2b-256 | 4aa001586f527cc345c1df1362809020362729c7bd3aeb03250b5bd8e7a6ed94 |
Hashes for llamacpp-0.1.8-cp311-cp311-macosx_10_9_universal2.macosx_10_9_x86_64.macosx_11_0_arm64.macosx_11_0_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 67fc069f5b13cecf71c8b0c9ad8a7ecaa6a4ba0d41e7555427f011d09d574324 |
|
MD5 | 9fd02ba55218c82f35ba3d7a9fe97821 |
|
BLAKE2b-256 | faedee75bdaf18f4bc7b4acba272695c605eb9a7c661c6a9f73cdd5be46a5338 |
Hashes for llamacpp-0.1.8-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 855f048bfca32bbc0d1b3f47baa549d5f3e15640410bafe8a21830e5d1b9ea72 |
|
MD5 | 119c37af470ebe2eeca5ef47961a6f98 |
|
BLAKE2b-256 | f43e05b01ddade985d1a3d1757455652101d855c3bb9321e5599309e1c376367 |
Hashes for llamacpp-0.1.8-cp310-cp310-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ce9efe1d5fcdbc509498abfa56a7b39590fa12c007070f40f88ca84c4ab6c950 |
|
MD5 | e802b6c0695dc5320c69669500637943 |
|
BLAKE2b-256 | 0d4df6e35baf1e03ce1c487c594ef37478a090f069637f3df1d478eb96421a72 |
Hashes for llamacpp-0.1.8-cp310-cp310-musllinux_1_1_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0e86d0d853999d6ffb90162f355cdd53be118ac4adf7b1d5d791ec4ce3354078 |
|
MD5 | e9dc9bfda289e2ae11daa8b746754b84 |
|
BLAKE2b-256 | da5b211a1261faf069a2ccd5ef941b12474b012ab53a675ee135d5bf63ef157b |
Hashes for llamacpp-0.1.8-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 527b6d32e1a215c02854d61d34d7e0bb2bac0d8d3df194986c29cc6872a0880d |
|
MD5 | 8b490e7d43386533348a8e3fff728524 |
|
BLAKE2b-256 | e036df3ad28d63575377dc261eaa49593fcabb54228f9348740c6b79d6bb54a2 |
Hashes for llamacpp-0.1.8-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 01cfc12deb4e64bc465cde378242cfa115694caceff4f67c28663cffcf6151ed |
|
MD5 | dcf16f9c0af57884c1f47113eda5df16 |
|
BLAKE2b-256 | 054f8995e106807c1d3a88ba05b694edbd18eb8de92dce1a30a289be84f2bcc6 |
Hashes for llamacpp-0.1.8-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 403f608583fdfabe05c24be9d7ccf62777e1b9cc214bb1b68af029d4a274c705 |
|
MD5 | aeec297d78ef0651aa09d7af63e66877 |
|
BLAKE2b-256 | 3a4deec0a2ac2fdf2caa53629e3bdb35fdc90ae258e27795925330780cf1c2c6 |
Hashes for llamacpp-0.1.8-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a7bd6d400dca4082675711b2c106b8d10c70cef48bb4826c0b8c50dffad1c03a |
|
MD5 | f3996a5a655898bfd47c727ce2a6fa92 |
|
BLAKE2b-256 | f02e86929b02620d5dbadd30237e3330352601cd210ac38c2297accf3968ead9 |
Hashes for llamacpp-0.1.8-cp310-cp310-macosx_10_9_universal2.macosx_10_9_x86_64.macosx_11_0_arm64.macosx_11_0_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f2063f7fc0f81931d658d4da1c744d53e791c22aa672d996b4297d5d7f9202d9 |
|
MD5 | 8812bbab65369f9b8484108d559cba68 |
|
BLAKE2b-256 | 0320deea86026eddc1179b1fa2993f1c46b38186b8dd6895ed3d3f2b0f47ba94 |
Hashes for llamacpp-0.1.8-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b65ce9155a5084c12095c9c00536ecc0184751180651a114611736b0e7505c90 |
|
MD5 | dfddc2ce0fcc75eabac2e3d5305f4844 |
|
BLAKE2b-256 | 26815430a949a2aafc1c9868784c765a8ab7c67b69e74fd9b6ffeaccf2ef896e |
Hashes for llamacpp-0.1.8-cp39-cp39-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1c2e8e0f7f51d83b24ec50d6b4c01b12c1f6aa5bee39ad3fb90d8c462fbabe7e |
|
MD5 | 1bdc22ea06f7abf372c2f8793d1ab262 |
|
BLAKE2b-256 | fcb65e4e82c42b408c7227c85c3b75981d95e497f8943f14c78bc07b16235691 |
Hashes for llamacpp-0.1.8-cp39-cp39-musllinux_1_1_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3179e4620c89de38dc2f0cba3a7525d7bcb3ba7c0041db4608263fc6362cf89d |
|
MD5 | 0b5eb1b98e54a47f38566e0c7a33ce86 |
|
BLAKE2b-256 | 064fa750f30df5ffb46aa2e519224242e0b6481ee338d74344bc2df22a24a225 |
Hashes for llamacpp-0.1.8-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e149b5ebd301f481af944b0836dc0adbed48b1627c039a3d6c82e6abaf14ffb1 |
|
MD5 | b2d9ae8c5f29838d048c1f61f7ab40e6 |
|
BLAKE2b-256 | 63c0e35f853751aa7070b4d1c8be3af7f03ad5bce9bde5303507b7b33c29dfb5 |
Hashes for llamacpp-0.1.8-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8b058e02afffb3d3b7e8763183604bda75428a7cb780861b047eac1fdd3e5ee5 |
|
MD5 | 65cb88b0267719210b267736643e9203 |
|
BLAKE2b-256 | 105d9f6098e9a711f0799b63b9a590f44cdd0b0ebd5304337b82798051ff2544 |
Hashes for llamacpp-0.1.8-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 672f5842ecfa084f9a4c4b49d8bfea677644ea266d74e97ba074af2604c02aae |
|
MD5 | 27625143df659e927d03057e5a90561c |
|
BLAKE2b-256 | b051d02488e4d73aee81416b0ed3e09ecdc7b8b5d4a836652f3ce10ef7b6904b |
Hashes for llamacpp-0.1.8-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bbe50814abc4b3dcf55fdc20f3d206d0b5067dd23d2f60fd86d4c9515def6be9 |
|
MD5 | 476ef02cd8fbf24859e683acb833f83f |
|
BLAKE2b-256 | dc2ed1a73a41bb79aac4c9667a14afafa9d956d4e09c4ad851b9a62defb3efe2 |
Hashes for llamacpp-0.1.8-cp39-cp39-macosx_10_9_universal2.macosx_10_9_x86_64.macosx_11_0_arm64.macosx_11_0_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | acd42b00d41bc33aab17a9177049ac657b9847e8960df8961409f13d6490a0a6 |
|
MD5 | f9ceaebf5594b189b44438deced90b65 |
|
BLAKE2b-256 | d76c288eae7660bb5978ccfbd7715a36ae5cbbc4ec0219d34cbd24c2c0808fa8 |
Hashes for llamacpp-0.1.8-cp38-cp38-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 31bab27bc01091cc7832e725156708888e955642b55e63ded6157e40fc99b1f8 |
|
MD5 | d3eaec0d355bbb6a6ae47d359e1a27ad |
|
BLAKE2b-256 | 9ddfb263d31d31d54e3e18eb8ec4265ac71ea18593ad408d44277ebee5a2dbd9 |
Hashes for llamacpp-0.1.8-cp38-cp38-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9d288cf4e4dd8f369115493cb65533853fd03b86a7cb5994f046049f18b57c85 |
|
MD5 | cdd3e132b08abdde34e98f6a5a74a70f |
|
BLAKE2b-256 | ed29795758562eb3254629b66ef32269aa99cd68e2fdec04f43becf415f8f282 |
Hashes for llamacpp-0.1.8-cp38-cp38-musllinux_1_1_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6277a3d4972a258ad1e5e2ff3d90ddc085e81d1d55a5f3c2ccd5996c41727f03 |
|
MD5 | df4205404a4bc2ff739522cd7608472e |
|
BLAKE2b-256 | fee6c1c2123b7c3b3a1333a64b62d33de1e768948e69aba74d9984c96ad7c8a4 |
Hashes for llamacpp-0.1.8-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 42c86cd6628fcdca0556c2927e3821b3d31ab44a9a757ed2397bc223173e8a17 |
|
MD5 | 07b0dcb94cc83552a0ce498e9feca7cf |
|
BLAKE2b-256 | d639358466870546f20267b31712728a5c0353845871c09efa305a0f7124bb9b |
Hashes for llamacpp-0.1.8-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 010992f5cb863d038d8cbe6abafe80c9364c3e2ac7b86a77c5a0bf9e2230f9aa |
|
MD5 | 5ce0e3bbaa45bf0896f0110effdfb039 |
|
BLAKE2b-256 | 021d7df3507245de09612f6d2a265f4525a8b0fd721066edd5b48f3bc29fe99f |
Hashes for llamacpp-0.1.8-cp38-cp38-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3325eb0f0a0228afcc7f585a0125761a1ffe8c7f5563505a6dffdd7445eef783 |
|
MD5 | 4de6997cb517c946416af6e57e7d2e4a |
|
BLAKE2b-256 | 3c78e9effbc03f64ace65d3a3ca94b89901b74af56b0fb49395fdf12f4b1b9fa |
Hashes for llamacpp-0.1.8-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f6c59d887dda752ea1868a2497c43ef3544c5a995ccd53e0906ff6591db362c3 |
|
MD5 | a280da7cf0829919e8fa8128c776008e |
|
BLAKE2b-256 | d1ba55567a2eca0696bc290e3d91eab3f18c77093ad072fab8499351d063dbd4 |
Hashes for llamacpp-0.1.8-cp38-cp38-macosx_10_9_universal2.macosx_10_9_x86_64.macosx_11_0_arm64.macosx_11_0_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 44016ef3aa0bd50ef8b43a1eb6d5181d1ea7ed3e3fde5eab1cb2a6071006c788 |
|
MD5 | f5ffb68a7b88aac1974bda403e64d71d |
|
BLAKE2b-256 | c6f703306f03ae2ec46025979305b96ab3b90d2c5e3c732ec16d0faaf191ebee |
Hashes for llamacpp-0.1.8-cp37-cp37m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9211f57350f754403f7aa1bb0fdeda4e3c93c56191a77f702d8155960b15ad58 |
|
MD5 | cb0d97eec6c2b2156b75cb57dfbf5531 |
|
BLAKE2b-256 | a290188d857b258e5846664bb831b49f4942472bc14ca63eb814aa7f6dc8c717 |
Hashes for llamacpp-0.1.8-cp37-cp37m-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0b4c5a3000e79e29ad77ce60f2dca853ed79c05b02e670a7c64e24e3b8713db1 |
|
MD5 | 91473a450e1618debfdd8aebf5fa4721 |
|
BLAKE2b-256 | f39731218a02b48584936e8854c05c99a477b6c8839ae0f23ce59a3ad48e3386 |
Hashes for llamacpp-0.1.8-cp37-cp37m-musllinux_1_1_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b0bc19517c52436634f131e1e938e52b560245fe456639d22364d5f3aaa57ad5 |
|
MD5 | ff4a39178534d00e6f0cb03930de66eb |
|
BLAKE2b-256 | fb4ecaa00a35e78146ba65d2f47a674ca25f735d042bd541e49124fbaa93d060 |
Hashes for llamacpp-0.1.8-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c6697aeac12f0ab79fea4ec21c39bffe8740c9e5dfe15ce2e639385c3a4204a9 |
|
MD5 | 13e3d3a12297b32d69e7f2cd3b3f13ff |
|
BLAKE2b-256 | e3fda1e0e2098012c2ea5cdb1f6c51e216ff5a9aeac66f17d762c7bfc1675e05 |
Hashes for llamacpp-0.1.8-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6ee093898c7639e95efa1d1abd6441b65c85dc99a37a46fff140195972ae2e6e |
|
MD5 | fdca40df6b11df4135be4312fa143d09 |
|
BLAKE2b-256 | c63526d74dd69c4b95a2ff5bea9bf998b1f12d731e7800b35d21785192280182 |
Hashes for llamacpp-0.1.8-cp37-cp37m-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 135f5ff711aa86e03e8a0dde86f9acfe63701ded79cc8f7c3716f4fb6bc1c25d |
|
MD5 | cd36130d98c0e0a39718093e6b524e8c |
|
BLAKE2b-256 | 8e68cb763789ac5758e292026ee38bafe37dac302ca27929915be93193864f32 |