Python bindings for @ggerganov's llama.cpp
Project description
Python bindings for llama.cpp
Install
From PyPI
pip install llamacpp
Build from Source
pip install .
Get the model weights
You will need to obtain the weights for LLaMA yourself. There are a few torrents floating around as well as some huggingface repositories (e.g https://huggingface.co/nyanko7/LLaMA-7B/). Once you have them, copy them into the models folder.
ls ./models
65B 30B 13B 7B tokenizer_checklist.chk tokenizer.model
Convert the weights to GGML format using llamacpp-convert
. Then use llamacpp-quantize
to quantize them into INT4. For example, for the 7B parameter model, run
llamacpp-convert ./models/7B/ 1
llamacpp-quantize ./models/7B/
llamacpp-cli
Note that running llamacpp-convert
requires torch
, sentencepiece
and numpy
to be installed. These packages are not installed by default when your install llamacpp
.
Command line interface
The package installs the command line entry point llamacpp-cli
that points to llamacpp/cli.py
and should provide about the same functionality as the main
program in the original C++ repository. There is also an experimental llamacpp-chat
that is supposed to bring up a chat interface but this is not working correctly yet.
API
Documentation is TBD. But the long and short of it is that there are two interfaces
LlamaInference
- this one is a high level interface that tries to take care of most things for you. The demo script below uses this.LlamaContext
- this is a low level interface to the underlying llama.cpp API. You can use this similar to how the main example inllama.cpp
does uses the C API. This is a rough implementation and currently untested except for compiling successfully.
Demo script
See llamacpp/cli.py
for a detailed example. The simplest demo would be something like the following:
import sys
import llamacpp
def progress_callback(progress):
print("Progress: {:.2f}%".format(progress * 100))
sys.stdout.flush()
params = llamacpp.InferenceParams.default_with_callback(progress_callback)
params.path_model = './models/7B/ggml-model-q4_0.bin'
model = llamacpp.LlamaInference(params)
prompt = "A llama is a"
prompt_tokens = model.tokenize(prompt, True)
model.update_input(prompt_tokens)
model.ingest_all_pending_input()
model.print_system_info()
for i in range(20):
model.eval()
token = model.sample()
text = model.token_to_str(token)
print(text, end="")
# Flush stdout
sys.stdout.flush()
model.print_timings()
ToDo
- Investigate using dynamic versions using setuptools-scm (Example: https://github.com/pypa/setuptools_scm/blob/main/scm_hack_build_backend.py)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for llamacpp-0.1.12-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ce4558f211407db66177aad600547bfbacc7011f61d3fee3c7cab5499e597784 |
|
MD5 | c44e281ced1a1ac3730e4c4382d2a9c5 |
|
BLAKE2b-256 | aa611eb75c4239aeb29aca7b2bd56a64b0cc51123112caef2107a1c251d27d0e |
Hashes for llamacpp-0.1.12-cp311-cp311-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | eb5a63493f171cb25501d930b1be25e2dd2dde536a7a9c2e16437b39599aa2d8 |
|
MD5 | 1d8d420f4361c71ad0a40404108f9cad |
|
BLAKE2b-256 | 26a2179dd5c7db15ef446c656d58bdbfbd77e911a11a49ff6383a3ef38843838 |
Hashes for llamacpp-0.1.12-cp311-cp311-musllinux_1_1_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 27c553d1a793bab8c788696a7327479c263a0a0f10bfc197d5217b0fc4ce7ffa |
|
MD5 | 7709b6a3ff26efbc72f6599e0ad8b3b7 |
|
BLAKE2b-256 | 4dff0ce3e7a90ff933bfa0046ad9efd3442776d8932c44a6dc3a2a9c278f0c57 |
Hashes for llamacpp-0.1.12-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d3cc6ed1503e853d928c92dccec205cc564b4aa09818713c284101c75f31a984 |
|
MD5 | 45b4555f083682ca23bec28845768695 |
|
BLAKE2b-256 | fbd34fe294f2830838aa7ca73216a0839d451ab31c0a7441e3df51efb98866c8 |
Hashes for llamacpp-0.1.12-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ad20c5ef9616f5afc013609feb5cf46644ce171dbbc5d5e97a31049ef48e5b5f |
|
MD5 | d64cd83fb06dd66d9207c3c8b0555a10 |
|
BLAKE2b-256 | 15c414f5c54612caecd1ba1c82f1d85d97fda0d520d5bedb35bf15a49d9e2275 |
Hashes for llamacpp-0.1.12-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 26e54bea9e9faf45b3e1d14b91590348ed872d70a4747be25e45e3ea6c2d5ddf |
|
MD5 | 760bc3dc9470db0651c20a907960e2d6 |
|
BLAKE2b-256 | e677a00073099abee62c00a54552884bcd25563039268c9857ce7004a9838222 |
Hashes for llamacpp-0.1.12-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 80a449fb5e37bdcc851c8c019fe73aeb11e9e54d2103c1cbb4224ac22526f18e |
|
MD5 | e9a464e28e064e3e0451c1a82f196bc9 |
|
BLAKE2b-256 | 1d9456d592d370e16259dcfa555863a54cc1cac8b5af1380c54eb1ea0fbeb802 |
Hashes for llamacpp-0.1.12-cp311-cp311-macosx_10_9_universal2.macosx_10_9_x86_64.macosx_11_0_arm64.macosx_11_0_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f13f8d0dee4c22a786646710c7dc4aacd7cc1f159a9acb6d5dc1f110c75a37c2 |
|
MD5 | c40295b152c09a608ec73738747ebba4 |
|
BLAKE2b-256 | 8513d2ccb4871ea95858d20448a97c0a8069a92c310eb44427f9e4df34dc0473 |
Hashes for llamacpp-0.1.12-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 65f307ef3752def36ec406f4c0826a9d92976dfae87aa144c66ab8cbf9df588f |
|
MD5 | 767d1b8379896344580bc37b6b8e17e4 |
|
BLAKE2b-256 | c17465d5eb03a5ece8d04a4b80c39eb93c02dca2ff1e2d0236af015405c228d4 |
Hashes for llamacpp-0.1.12-cp310-cp310-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 120989973800ea0b12557ad680c46e799f53f80d6859d48dd1da77db30dee935 |
|
MD5 | 6a2dc54e7f8fdd8e058138caad1f6e56 |
|
BLAKE2b-256 | 01b48939b229024e058dd27e9abf2a725ce34ee4b6f8a47c59375aafe0a7b8d6 |
Hashes for llamacpp-0.1.12-cp310-cp310-musllinux_1_1_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b7c6389b352a684257fb64a5af8fe7fe792774be78ae6676869a7df2d47c6b5b |
|
MD5 | 1da7f72c4fcd026189c5a5959d06bfb5 |
|
BLAKE2b-256 | 022455d03ed880a2d1edbc4f714e2e0cf9ad49c0b239efd289214c0b022afe34 |
Hashes for llamacpp-0.1.12-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 135d91e05aa67d17e8f02f2f43efac99a3c8386f0a80eef7ea0169bf2b5ccd19 |
|
MD5 | b66e0f52ad1d653cfab9a09ebae9b74a |
|
BLAKE2b-256 | 22d2389b46491990842ff61414233f1780d2c91dd86727301e4e316a415449c5 |
Hashes for llamacpp-0.1.12-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2f24f5e9de0eb22a209c0c1ae7d893aeecc4973bdf5efb5398819f113cd39609 |
|
MD5 | 41570a514d95d66ba960d20e16a3ae57 |
|
BLAKE2b-256 | 45cfa2b5ddcd6b0d806bd5a9a2c0659765fe9fb07cb831389b8e1300834896e6 |
Hashes for llamacpp-0.1.12-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1d5e91a52bdadb928b0ef2a211da8ca5c469df5e01693d1704f973357eedb894 |
|
MD5 | b5b4cd256818212e57ed8f36e11c0a01 |
|
BLAKE2b-256 | 1d503d687d0aa91afecebd4528fb03acbe492df6ec166837a59ab88c97c98437 |
Hashes for llamacpp-0.1.12-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 65064b6948700128224e639baebcb9cd9e4618c7331ce0a29a7ca39694f3d00f |
|
MD5 | 4016c740a8ee926a282a820ff7dda1e9 |
|
BLAKE2b-256 | 803df195401eb0c891d4dc954c1631627bdc395948fc0462748c22852ca50ce8 |
Hashes for llamacpp-0.1.12-cp310-cp310-macosx_10_9_universal2.macosx_10_9_x86_64.macosx_11_0_arm64.macosx_11_0_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 52694c5c7758547333de38e2f07ff7dd8747e11689f5181a1cdf2600fcb8072d |
|
MD5 | ad44f01309ab0d57cdd5d163f9843222 |
|
BLAKE2b-256 | c1ea193d694a70f3a19a7f4ddf0fe734317a87656b8b51637d9bfb8d656fd721 |
Hashes for llamacpp-0.1.12-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f920227184e170fdba37a397028f622ee50f92df2fc5dc2314d38fd0931cd97d |
|
MD5 | dba62e51ddb1d18a84aa9c66d3d17df4 |
|
BLAKE2b-256 | ea79d6b22c3ba83f2192f51e09f2b6f59b03094b676749abbc0a4ce16708533f |
Hashes for llamacpp-0.1.12-cp39-cp39-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9b1b3dc0863bdecd2abcd9841e151c4f5aab49347200879920039b75b11ddad1 |
|
MD5 | 3188984fa7f61dba1562f31bb7e54068 |
|
BLAKE2b-256 | 79acd777e230b97e3e9019c1d379376d259af8ae5880476011c810264de1b23b |
Hashes for llamacpp-0.1.12-cp39-cp39-musllinux_1_1_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 57ab2b2c594488b997268382d159248283b64041674bcf109ca1ff71e495e61c |
|
MD5 | 43871059ba19dad3f324fa5de09c4414 |
|
BLAKE2b-256 | 075d30c9c860ae274a1969d14eb2c4db70902a35333bac8448ef43a9ea11a782 |
Hashes for llamacpp-0.1.12-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 061beba955d9b6af37e61b13263740bea0b7f66efafac204e34dcbe571aa6339 |
|
MD5 | 12e50a389fabac2d848c0d7e61fc4483 |
|
BLAKE2b-256 | b7497df2d1a7d2e33735c8ebfc63ff8d71d7ab2f639505d6f46f0a184063bc3a |
Hashes for llamacpp-0.1.12-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 628dbc2650c65a479c5a151853015476b3811540e86a0f49c5a55a570950d06e |
|
MD5 | 6e8dd9fb9d3b08e69aea8794c732fe89 |
|
BLAKE2b-256 | 7a62c684eaf90f526fe1d9f063559692462d4fc6a7b9c159d4948b17a6c70fe6 |
Hashes for llamacpp-0.1.12-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 348a6fa65c1ca3e8a76e3f1acafbb0b7284aae4a875762f86c3378817e88497b |
|
MD5 | 5979ecff4e7619d6f9a02d8415e54c55 |
|
BLAKE2b-256 | 64ac0c3b35b47891bf3e958a9677512675872f988b93a88b42b4a7fa9f16dce3 |
Hashes for llamacpp-0.1.12-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a1937fe0b201c2d2208a1cc0e4c1630c7c10bcaa42a45c7cbc377a1a47bbc0aa |
|
MD5 | 8326b4cc59b54aa9fb3e7e29e03931b1 |
|
BLAKE2b-256 | 6b59a6b716cd2767693712bf8d61faf0a0249e6de6203e3ac4ac7e2a3048bf9c |
Hashes for llamacpp-0.1.12-cp39-cp39-macosx_10_9_universal2.macosx_10_9_x86_64.macosx_11_0_arm64.macosx_11_0_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d8487713fa78bff1db2c1608410384b44ba6735704efba45ac4e4c7e1ba551ea |
|
MD5 | e7e705348cad5ec11047eeda9f352955 |
|
BLAKE2b-256 | 4326ca1cd0e3dca993a2d23f6e7d67008e575b745fb98dccd259e8eaeb46acab |
Hashes for llamacpp-0.1.12-cp38-cp38-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | eaa51da32c1049b56cc69dd337f6c42fd71ea6b0d2909f27870c279e7ae00243 |
|
MD5 | b8dbc6c7005f351781c006c8f206b81b |
|
BLAKE2b-256 | 6fa0188969f92acff5d585d8bb7100e24659d9a0b2fa8b472929c89f933dabb0 |
Hashes for llamacpp-0.1.12-cp38-cp38-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 506d0b0faaca40d9f6412874dab93928f396e68053e3c88fa3766adc54a7e740 |
|
MD5 | b662a687b6d3535fe26e35f7a4bc76d5 |
|
BLAKE2b-256 | 41006178dbf2f04cc9743f8e2df6bc39dfd732825dd70b9f27d8fe09c4f13b71 |
Hashes for llamacpp-0.1.12-cp38-cp38-musllinux_1_1_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cc3e8901a079906446c05a342f48708a8deecad1596583960d5b189734909f35 |
|
MD5 | 4e68f4ca326f3e0d6aeacbb3559a9416 |
|
BLAKE2b-256 | 2aff84af03bdc1c819db8b42158df5ad60f71d2a9e9376302d247bd456e3cb31 |
Hashes for llamacpp-0.1.12-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 33865bbf943263b29f8bab7e8c91778f0255d991c8e621c9bd5e6403dc0b57c7 |
|
MD5 | 641ad7cd174f99ad125e64471825745d |
|
BLAKE2b-256 | 26711c445f0dc6cec185219c59f46be952e262fb819f16fc2390a0e098f2d30b |
Hashes for llamacpp-0.1.12-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 67cf70d441ca83c113be2bf091aa976defafe64e3ef4ff725d6e013375cb4636 |
|
MD5 | 3fec37401b8381197a1a866d7deada6c |
|
BLAKE2b-256 | de5e09efd574c83fca252ac60b88f0dc2cb66903465d43b2392737b4f520f5a1 |
Hashes for llamacpp-0.1.12-cp38-cp38-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cd0cd99b64517c99ef353092e655b5bbb628989717d31cacaa0ab1b6b6c7d8a1 |
|
MD5 | bf1fc49ff22a97eb881432f4654cc87d |
|
BLAKE2b-256 | 458a701db532dd8ab88325b6a9cb15e50625db96ba59e6f7559610fb139ce76b |
Hashes for llamacpp-0.1.12-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 098a575976c2f3bc3cdb49d98593799a757ad78ca9cb290de515c990a2c15a02 |
|
MD5 | 761abcd02ac6c570b59515f64e67100f |
|
BLAKE2b-256 | 9d8017fdfa1333feef52ddf99a71cb2b60f6b9b63bc2a8ec051cda966b24ab66 |
Hashes for llamacpp-0.1.12-cp38-cp38-macosx_10_9_universal2.macosx_10_9_x86_64.macosx_11_0_arm64.macosx_11_0_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a9a7d4981bcbdc6d5b2a32c575c2b94e735916609e937104618c7b65d84b05e0 |
|
MD5 | 83ef6abe4a13efb2d72196d4d582b558 |
|
BLAKE2b-256 | 267de75af9704856d120aa5095493ef770bbe7fd237f1881c92c3a1f79be6027 |
Hashes for llamacpp-0.1.12-cp37-cp37m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4965413da21eb8d6fa7255e854bab14eadb90626f9a4773956eec6689a52cb1a |
|
MD5 | 5f99ad1e421c7fea14b3962822f9d6bc |
|
BLAKE2b-256 | e88a703f51ef76ff9516a42edec5ec0c95199b378d4b3477aba14c153768fc91 |
Hashes for llamacpp-0.1.12-cp37-cp37m-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f35fc5045085f21565d053a860fa7bc94e50389c418d467c070b38357097c8a8 |
|
MD5 | e0d217fa933344d3d6ddc0bd3b97da65 |
|
BLAKE2b-256 | d1588a5a1320b79a02fb248df7320beaa6f5c7a9fa2152b6344ce04aa9b7c2c4 |
Hashes for llamacpp-0.1.12-cp37-cp37m-musllinux_1_1_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | aa06f0e688f8de48eb489f1fc3485fe20ffeec2dda23892127720610af7cd562 |
|
MD5 | 8dacec1b8c723897896e8e07af47f1c2 |
|
BLAKE2b-256 | c92b3270f8c3c9a472024d1babc5286127d7abc0205ce45f5aac1062839d47b5 |
Hashes for llamacpp-0.1.12-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2ab7685e62ef2303836273a33c1fc806208eccdb331b2b14dc5a8da11d323822 |
|
MD5 | 4d10bbad901d77cfdfa6aebd84a0ff06 |
|
BLAKE2b-256 | 8effbfa3c702d3055ac6ff42fe4a406c05b13a54f9dbbfc9974782ef9c166e01 |
Hashes for llamacpp-0.1.12-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3e5d25b00177bf6d6418260dca5bd56e83ddaa82feaa767ddebbd3c2b9eb8e7a |
|
MD5 | e1a7c2ce052d705dc0512cbf80713a63 |
|
BLAKE2b-256 | 2ce4af2575573e2c7643cbf0856385d0066b41b521bc1113934ccdfb8192b018 |
Hashes for llamacpp-0.1.12-cp37-cp37m-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 51371bb6aa94c6447949033c1f68f1b2a3ca62cec1bc68f06a6c5c85b733aaf2 |
|
MD5 | 493efe67646ab455485d91e6f9369ec7 |
|
BLAKE2b-256 | 9d19e9ecb9af4879a880ab4ff817c842fca0e4baf7ccddd4043e71de907dc8d6 |