Python bindings for llama.cpp
Project description
PyLLaMACpp
Python bindings for llama.cpp
For those who don't know, llama.cpp
is a port of Facebook's LLaMA model in pure C/C++:
- Without dependencies
- Apple silicon first-class citizen - optimized via ARM NEON
- AVX2 support for x86 architectures
- Mixed F16 / F32 precision
- 4-bit quantization support
- Runs on the CPU
Table of contents
Installation
- The easy way is to install the prebuilt wheels
pip install pyllamacpp
However, the compilation process of llama.cpp
is taking into account the architecture of the target CPU
,
so you might need to build it from source:
pip install git+https://github.com/abdeladim-s/pyllamacpp.git
CLI
You can run the following simple command line interface to test the package once it is installed:
pyllamacpp path/to/ggml/model
pyllamacpp -h
usage: pyllamacpp [-h] [--n_ctx N_CTX] [--n_parts N_PARTS] [--seed SEED] [--f16_kv F16_KV] [--logits_all LOGITS_ALL]
[--vocab_only VOCAB_ONLY] [--use_mlock USE_MLOCK] [--embedding EMBEDDING] [--n_predict N_PREDICT] [--n_threads N_THREADS]
[--repeat_last_n REPEAT_LAST_N] [--top_k TOP_K] [--top_p TOP_P] [--temp TEMP] [--repeat_penalty REPEAT_PENALTY]
[--n_batch N_BATCH]
model
This is like a chatbot, You can start the conversation with `Hi, can you help me ?` Pay attention though that it may hallucinate!
positional arguments:
model The path of the model file
options:
-h, --help show this help message and exit
--n_ctx N_CTX text context
--n_parts N_PARTS
--seed SEED RNG seed
--f16_kv F16_KV use fp16 for KV cache
--logits_all LOGITS_ALL
the llama_eval() call computes all logits, not just the last one
--vocab_only VOCAB_ONLY
only load the vocabulary, no weights
--use_mlock USE_MLOCK
force system to keep model in RAM
--embedding EMBEDDING
embedding mode only
--n_predict N_PREDICT
Number of tokens to predict
--n_threads N_THREADS
Number of threads
--repeat_last_n REPEAT_LAST_N
Last n tokens to penalize
--top_k TOP_K top_k
--top_p TOP_P top_p
--temp TEMP temp
--repeat_penalty REPEAT_PENALTY
repeat_penalty
--n_batch N_BATCH batch size for prompt processing
Tutorial
Quick start
A simple Pythonic
API is built on top of llama.cpp
C/C++ functions. You can call it from Python as follows:
from pyllamacpp.model import Model
model = Model(ggml_model='./models/gpt4all-model.bin')
for token in model.generate("Tell me a joke ?"):
print(token, end='')
Interactive Dialogue
You can set up an interactive dialogue by simply keeping the model
variable alive:
from pyllamacpp.model import Model
model = Model(model_path='/path/to/ggml/model')
while True:
try:
prompt = input("You: ", flush=True)
if prompt == '':
continue
print(f"AI:", end='')
for tok in model.generate(prompt):
print(f"{tok}", end='', flush=True)
print()
except KeyboardInterrupt:
break
Attribute a persona to the language model
The following is an example showing how to "attribute a persona to the language model" :
from pyllamacpp.model import Model
prompt_context = """Act as Bob. Bob is helpful, kind, honest,
and never fails to answer the User's requests immediately and with precision.
User: Nice to meet you Bob!
Bob: Welcome! I'm here to assist you with anything you need. What can I do for you today?
"""
prompt_prefix = "\nUser:"
prompt_suffix = "\nBob:"
model = Model(model_path='/path/to/ggml/model',
prompt_context=prompt_context,
prompt_prefix=prompt_prefix,
prompt_suffix=prompt_suffix)
sequence = ''
stop_word = prompt_prefix.strip()
while True:
try:
prompt = input("You: ")
if prompt == '':
continue
print(f"AI: ", end='')
for token in model.generate(prompt):
if token == '\n':
sequence += token
continue
if len(sequence) != 0:
if stop_word.startswith(sequence.strip()):
sequence += token
if sequence.strip() == stop_word:
sequence = ''
break
else:
continue
else:
print(f"{sequence}", end='', flush=True)
sequence = ''
print(f"{token}", end='', flush=True)
print()
except KeyboardInterrupt:
break
API reference
You can check the API reference documentation for more details.
Supported models
Fully tested with GPT4All model, see PyGPT4All.
But all models supported by llama.cpp
should be supported as well:
Supported models:
Discussions and contributions
If you find any bug, please open an issue.
If you have any feedback, or you want to share how you are using this project, feel free to use the Discussions and open a new topic.
License
This project is licensed under the same license as llama.cpp (MIT License).
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for pyllamacpp-2.1.0-pp39-pypy39_pp73-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b4c7e46a834e3d5deffefba46fb86cd614ea74d8a92697502afdadfdcd480b37 |
|
MD5 | 357cb0baeddab57bb93d00adfd82bf46 |
|
BLAKE2b-256 | bd79dcf7d78f9ed3ef460d6ad535eeb141c7c71a7983bea3eac09d96095c85cc |
Hashes for pyllamacpp-2.1.0-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 88fd74ee75513eed3b8861055ffafe9caab8072243261c5e7d1b0ad4e43028a8 |
|
MD5 | 206fdf52e8b6978354a9824a6a5d23b7 |
|
BLAKE2b-256 | 5709e39d96ee8a575cc9ada025f401ef19164fa07b0c53c97df1aca1c5a4fc6f |
Hashes for pyllamacpp-2.1.0-pp39-pypy39_pp73-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 65a4cc6e49ab303839ef81dd23ea94a77b568c27b7577ff4eb7a26ff21ef067e |
|
MD5 | 0fd4225cc7ec5bc028941f0ee525567d |
|
BLAKE2b-256 | 4d2741768ac3e96e6b5c412544642fb20536186f75b685258e568806ce2d023b |
Hashes for pyllamacpp-2.1.0-pp39-pypy39_pp73-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7ccf188e7c1e3ab478c2387709f44b3e76e7db4c0bc6f6bbbb63ea2d83a973c6 |
|
MD5 | 59e1eb6f4a61a8f960e56944623d38d3 |
|
BLAKE2b-256 | 4c43123c29139278e2d42b7c269b8ea54a7184be926ab2bec7606433e1e4017b |
Hashes for pyllamacpp-2.1.0-pp38-pypy38_pp73-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e3ae2b9ee81c279f030b11efec1a71b3de4f7bdba19591bbaed8261f9b92e208 |
|
MD5 | b7c28fb6dfbc0e134cb7d05d7d739e5c |
|
BLAKE2b-256 | 2d308cd57dc3445c25624889738167ac314afeef8e03b6c3e1934892f16bfc16 |
Hashes for pyllamacpp-2.1.0-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 833e98364852efadfaf552ca30336280825e12140892d5c3c20b345528281168 |
|
MD5 | a6d44a5261995dc21cf8f89a2ffb7b13 |
|
BLAKE2b-256 | 6e71b27e0736af5bd3da426180cad922ecc33ad7b8e3e0e77cfefd4521f24e6e |
Hashes for pyllamacpp-2.1.0-pp38-pypy38_pp73-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d4669331c070c5ccf0cff106f7d1934fb99683712efd5a25a593169f2769e76c |
|
MD5 | 50206102e7fcf5ff592cc38b1849bb76 |
|
BLAKE2b-256 | 6f84a11c9226cd3aabe4588928ff10ff89af2c6d6c85d278be3aacfbbacb41ac |
Hashes for pyllamacpp-2.1.0-pp38-pypy38_pp73-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e69bcd077c6d23be4c565d2917792726b5a784f39c870a97f2e0e65b2e6e97d6 |
|
MD5 | 72a5700e384ab6c260adc24b19a77387 |
|
BLAKE2b-256 | a0acc7155da306c42a7d5df3e0fa8238f3c4892f13cec3797ca8d48b26883aea |
Hashes for pyllamacpp-2.1.0-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 282c5ba58dd4418833f7a78d02c80b48401b7269a9d9cf51334916558a757795 |
|
MD5 | dacbf2edf7b3c9a206c28e73aace19c6 |
|
BLAKE2b-256 | f5c80f8702f88325a3b71f729a9a1a638af46648ca1bd6e6c1b3935759f8be6d |
Hashes for pyllamacpp-2.1.0-cp311-cp311-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1da000a0b42adf9202aaff36b32f579b7d067451945335aa9ed7a3aa88cb9372 |
|
MD5 | 517900387b4b766737bc78de1e845740 |
|
BLAKE2b-256 | 3217a04012109dc223334680ee08588f140f2b3cea07b350c704cca2c81eb2e1 |
Hashes for pyllamacpp-2.1.0-cp311-cp311-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7e9049246e1f6bfbc2b1b84aeedc4cf62380075c7b46f7db03c6e3c68125a4a6 |
|
MD5 | e59ce5c2e9a4e76374546bd6b722396b |
|
BLAKE2b-256 | 63506dfb70bd10aa1c5be493b8c7c5395b5256a55f83e670221e87823efbb953 |
Hashes for pyllamacpp-2.1.0-cp311-cp311-musllinux_1_1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | be5ba0650b55967737900f6b0a29687387bc715a853cc3269370ce95c75d0af9 |
|
MD5 | b42f13451d878da33a78264acf8eb6ea |
|
BLAKE2b-256 | fe81b0ba9c70294ff2f59d39d76e28af647d72e656ccd8f0519bcde111b0b4c2 |
Hashes for pyllamacpp-2.1.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 818c02756c961c1ef4216a92f1b225cc777948d272953a3279f7f0bb7aaa387f |
|
MD5 | a1cae6005136abe3ea8f9e81c8facb1c |
|
BLAKE2b-256 | ba7d16c6b37e1667f4df6b6c02c66c392b764c63302b73409ab3862a9863ef15 |
Hashes for pyllamacpp-2.1.0-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8d9ff9c92ee191adb4ebef97b9fd98c6e3059061300a656bdd1efc0c2759b9ce |
|
MD5 | ba4e9412940301b9fff723f035a7cfe4 |
|
BLAKE2b-256 | 675c09f65598c3ae55007b7f4baa7ddfb7eb5eccc569b661ca0cdf2397c49bd0 |
Hashes for pyllamacpp-2.1.0-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a82d724d11fb6e89ec9cdba6fe834609f5c5cbf1533fea89ba1598d798d5499b |
|
MD5 | 7950e6669cc19a928e896f6cc2701a07 |
|
BLAKE2b-256 | 3550bba8d6865156b084e2a1f0e1071b157e101791834927b61c50358c0e56f1 |
Hashes for pyllamacpp-2.1.0-cp311-cp311-macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a6d25a26cd136c3191325864def7f2c0a6e2ed766e0101c8c7da78a998357438 |
|
MD5 | 557c7e83b697572c5cb3d890e547ef20 |
|
BLAKE2b-256 | 7dd8daf5688fa9cdca5f144922bb0fadf2270421988464bdf12707bfbb1bd450 |
Hashes for pyllamacpp-2.1.0-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a1632437885a89a59e3d52798d6137aed6bcc886ffadc3a154d375f337d96239 |
|
MD5 | 0a95816117be0592da6399218e042b87 |
|
BLAKE2b-256 | 52744cf636beac5d45d48cea67b79550d2de714f13b891d6a4a696df19ac658b |
Hashes for pyllamacpp-2.1.0-cp310-cp310-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 087ec9d84c4bd490b2ea69e69c16d22f1ab64b5c193b87151db151c1658047c9 |
|
MD5 | dc8d9c3d638a6ebf5071242be42a3116 |
|
BLAKE2b-256 | 4b915930d55db89411f482cc1c0db856b9a763d926aca67e93c6393423d43c1c |
Hashes for pyllamacpp-2.1.0-cp310-cp310-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8d95320ba01d602678649d955d83b91b1960e5d4d0b75606d9431385c785891e |
|
MD5 | 144620fe35270ccf07c4ba97753b6db9 |
|
BLAKE2b-256 | 3460ccd97a5257834cfd2d0683cbbb0051c400d249c4d135c33731aad5d68a10 |
Hashes for pyllamacpp-2.1.0-cp310-cp310-musllinux_1_1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5eb3a1189d06f30dbc308c536de539d11a554cacdc97d0a767a355f0ad26fef0 |
|
MD5 | 200cdac2351cf36a15902fdb5e1e4de8 |
|
BLAKE2b-256 | f3c9354045978b658fc538b60920c158357b274cbf3e50a87abcfd3feda59584 |
Hashes for pyllamacpp-2.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 24992d1c16adc57f0917130b9b58830d3d0c96dc87fcfe8b92a24fd38583251d |
|
MD5 | 825651c9fe8102c6f16301f060678d07 |
|
BLAKE2b-256 | 0597f9eb0687baea62f11fbfc43c6e3c25c037e1339d698263bbcd572e5298ce |
Hashes for pyllamacpp-2.1.0-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a2b0824a3c0b40d8aa2db9a5e4ca310dd3a4a34d01d0f764eaf9ef4b29d907bc |
|
MD5 | f1ac955fde7da52ab76f16fd836fad47 |
|
BLAKE2b-256 | f7792de350ce25a0c1cfbf9fdeb41d3b3f1f833ae0ce9d468276e8ec5e47dc00 |
Hashes for pyllamacpp-2.1.0-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 50c2a0a198d0194cb38872e7031e5b2019e8d36aca20ce7d577101a4854529d7 |
|
MD5 | 3bf7769a51e6e88404455beb1f72cc82 |
|
BLAKE2b-256 | c8c42da580b3e923f394e41bb1fce77e02e68e285d61578f49c262f7c2686bad |
Hashes for pyllamacpp-2.1.0-cp310-cp310-macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 29fe2fc0b8f9979099c6b90d12eb18e3cae35bb07cde54fdecba2397ddc20aa4 |
|
MD5 | f41d8c61dfc058d6c9a0a61f0b611863 |
|
BLAKE2b-256 | db525eb99f5916b31cabfb6ecdb72869cb26adde651b3d409934cf8ff78b1d2d |
Hashes for pyllamacpp-2.1.0-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6366b3b2a11101707a0c718a06520a6d309fe161f4910ff3d67184c249b9342e |
|
MD5 | 0716f59b72ec61bd712b14eb5c0ac36c |
|
BLAKE2b-256 | e80dc8c76fc46de2f88785105346dd7978a2a4d9945218f7128b657370a69583 |
Hashes for pyllamacpp-2.1.0-cp39-cp39-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fd5a037e62cee5c10d509b7dd857d69cb9670c258d0f998acb0445e340345531 |
|
MD5 | 4af24295d65e41bead4cd269b96a2a7e |
|
BLAKE2b-256 | c3f5d2da7ff12002936c844b1054a106d861a13086b055d56410acfbb47bf836 |
Hashes for pyllamacpp-2.1.0-cp39-cp39-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a9bba7bfbc0f49935d4b60b04d3bed93f40ff846c019986bb4ab3d5acaa7903d |
|
MD5 | 9f998335418c68009e745275e302d8ff |
|
BLAKE2b-256 | d7c82b825770a269bf525c3c21632c7762d53ba0b47f8b408445f8c3fcc29b20 |
Hashes for pyllamacpp-2.1.0-cp39-cp39-musllinux_1_1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2050f15dc3e0370423f8a70b65509f4a37f38fc6bf6c6eab8a639a85ffdd7978 |
|
MD5 | 177fa45f4c91ec27d2ec1d2bc7f0bbb1 |
|
BLAKE2b-256 | 3b50c24495cfa69c24e5b303b47fe9dc3e9d5df200dbcdcc3e2846df9bfcbc51 |
Hashes for pyllamacpp-2.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7076a132455529e3053fff8fdcaaab7b19073ff27f6e310947a4e14a12ef5673 |
|
MD5 | 06510b78071f5f8d0fc39890f6b7522a |
|
BLAKE2b-256 | 44c276273f1ac507a3731fc533b3d67e7719a73d590426909a5a14a65487776e |
Hashes for pyllamacpp-2.1.0-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0f89cf080fc8a8a7fa0eb934dcdbf511ee57d0657bfedb58cbf688c07d8265d0 |
|
MD5 | f9230c438a74d86df473cef923efbbf8 |
|
BLAKE2b-256 | a5195359d78a26a5868772eeb833dc41ddcb036904b0cfad202710ecbcfdc4da |
Hashes for pyllamacpp-2.1.0-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | be74aece29cd3df6d64c94e6a1103b033eec41aaa34869b39b5f6910f7c617c5 |
|
MD5 | d66cebf370da8bccd81dc366bc6b3729 |
|
BLAKE2b-256 | 68789e2f0599ef9b3bb85066f28f14ea29c862319b5ca8d001e0a02690099a6a |
Hashes for pyllamacpp-2.1.0-cp39-cp39-macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 23379c52469b0520802337db07301e4efef6bf5b6d73fb72a5ca6da0b49202da |
|
MD5 | 9193e926b928a840208d7b5d7d457e53 |
|
BLAKE2b-256 | 58352fca3bb386c540fcfa1252dd90c17e0b90df3ad4b4eedd745668ff94da6c |
Hashes for pyllamacpp-2.1.0-cp38-cp38-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c2761e3bf65cdfb34294f719d28b0df9a6ed5c818819a593d8a1d8ef201fd992 |
|
MD5 | fab06680abc8100c925264e10978eeba |
|
BLAKE2b-256 | dd51aa852f76e33d2a21c7f1c4617d11fe6319e64314dcf31eab6ce205f92a35 |
Hashes for pyllamacpp-2.1.0-cp38-cp38-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9670b7df2a45d95626101a5d406a15e288f06b0b3c6eb341d790edbc21d0d6fa |
|
MD5 | 53fb69469e771c0ec053d3c7dd5f1db9 |
|
BLAKE2b-256 | 4e4f682099b67a4c0792829f800d026c33e5742fa51eccee0e23605b70c891e1 |
Hashes for pyllamacpp-2.1.0-cp38-cp38-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2de89d1fe6de9375d07fd2b1b5484cbbaad33a2b9d8e8cc1ccdd61a7035bfcec |
|
MD5 | cb963f748c6e72ecf4a63c44bccde287 |
|
BLAKE2b-256 | 56c18f47b00227ce16f1af71251f8d127a96f5c662ae02775d2236fd0478fe8d |
Hashes for pyllamacpp-2.1.0-cp38-cp38-musllinux_1_1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c150949d9251b5cebdc6729879dc86683cdf5a55ecc4fb50371453f79bc04a6c |
|
MD5 | 1c6c0867e8568d710c949d33890035dd |
|
BLAKE2b-256 | be0fa9c0f462aea1a25b63f960c71130e8ab672c7b9069230883222fbdc59b44 |
Hashes for pyllamacpp-2.1.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ce439e7415e3ec7af724ed3057b0f6a959f78e2e0fe871c6be3635ec417637d8 |
|
MD5 | de177d8cc9c5cfd31ad8b5f5ffba337a |
|
BLAKE2b-256 | 85ccf6f1b446c3ceff80aaa05d4477db303da1f8fecb943f3ab682ed12641a79 |
Hashes for pyllamacpp-2.1.0-cp38-cp38-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3ef802b4b272089251feea5017c60f5eecbc16b0b809b8fd4e31439a746a380d |
|
MD5 | 4b0011859d65ae7624999e8a62e9a397 |
|
BLAKE2b-256 | b4d90db68144c34a2a7b33da94caa3f7ec9e80f2a594140ce90b095145ff22c1 |
Hashes for pyllamacpp-2.1.0-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 784569171ab40d0d838cc40bcebe11cd2ba6f0edf9735441f4306fec8b1db8d5 |
|
MD5 | 875ff8ca8f191104bdc202b8d1d3050c |
|
BLAKE2b-256 | 4459dc79826d7607f1d29f9f69fdf136d49a1f20ba620bdbb3113d4811de25c8 |
Hashes for pyllamacpp-2.1.0-cp38-cp38-macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9d1a85b17b88212210e4df9bed25d130802ea4d28fed2b84c14135b12b1fb098 |
|
MD5 | cebe2f5890c77698803007765c057361 |
|
BLAKE2b-256 | 057a99c521b99d319ae7054b2019a2791758c648071cf2c01adc05adf1836835 |