Python bindings for llama.cpp
Project description
PyLLaMACpp
Python bindings for llama.cpp
For those who don't know, llama.cpp
is a port of Facebook's LLaMA model in pure C/C++:
- Without dependencies
- Apple silicon first-class citizen - optimized via ARM NEON
- AVX2 support for x86 architectures
- Mixed F16 / F32 precision
- 4-bit quantization support
- Runs on the CPU
Table of contents
Installation
- The easy way is to install the prebuilt wheels
pip install pyllamacpp
However, the compilation process of llama.cpp
is taking into account the architecture of the target CPU
,
so you might need to build it from source:
git clone --recursive https://github.com/nomic-ai/pyllamacpp && cd pyllamacpp
pip install .
CLI
You can run the flowering simple command line interface to test the package once it is installed:
pyllamacpp path/to/ggml/model
pyllamacpp -h
usage: pyllamacpp [-h] [--n_ctx N_CTX] [--n_parts N_PARTS] [--seed SEED] [--f16_kv F16_KV] [--logits_all LOGITS_ALL]
[--vocab_only VOCAB_ONLY] [--use_mlock USE_MLOCK] [--embedding EMBEDDING] [--n_predict N_PREDICT] [--n_threads N_THREADS]
[--repeat_last_n REPEAT_LAST_N] [--top_k TOP_K] [--top_p TOP_P] [--temp TEMP] [--repeat_penalty REPEAT_PENALTY]
[--n_batch N_BATCH]
model
positional arguments:
model The path of the model file
options:
-h, --help show this help message and exit
--n_ctx N_CTX text context
--n_parts N_PARTS
--seed SEED RNG seed
--f16_kv F16_KV use fp16 for KV cache
--logits_all LOGITS_ALL
the llama_eval() call computes all logits, not just the last one
--vocab_only VOCAB_ONLY
only load the vocabulary, no weights
--use_mlock USE_MLOCK
force system to keep model in RAM
--embedding EMBEDDING
embedding mode only
--n_predict N_PREDICT
Number of tokens to predict
--n_threads N_THREADS
Number of threads
--repeat_last_n REPEAT_LAST_N
Last n tokens to penalize
--top_k TOP_K top_k
--top_p TOP_P top_p
--temp TEMP temp
--repeat_penalty REPEAT_PENALTY
repeat_penalty
--n_batch N_BATCH batch size for prompt processing
Tutorial
Quick start
A simple Pythonic
API is built on top of llama.cpp
C/C++ functions. You can call it from Python as follows:
from pyllamacpp.model import Model
model = Model(ggml_model='./models/gpt4all-model.bin')
for token in model.generate("Tell me a joke ?"):
print(token, end='')
Interactive Dialogue
You can set up an interactive dialogue by simply keeping the model
variable alive:
from pyllamacpp.model import Model
model = Model(ggml_model='./models/gpt4all-model.bin')
while True:
try:
prompt = input("You: ", flush=True)
if prompt == '':
continue
print(f"AI:", end='')
for tok in model.generate(prompt):
print(f"{tok}", end='', flush=True)
print()
except KeyboardInterrupt:
break
Different persona
You can customize the prompt_context
to "give the language model a different persona" as follows:
from pyllamacpp.model import Model
prompt_context = """ Act as Bob. Bob is helpful, kind, honest, good at writing, and never fails to answer the User's requests immediately and with precision. To do this, Bob uses a database of information collected from many different sources, including books, journals, online articles, and more.
User: Nice to meet you Bob!
Bob: Welcome! I'm here to assist you with anything you need. What can I do for you today?
"""
prompt_prefix = "\n User:"
prompt_suffix = "\n Bob:"
model = Model(ggml_model=model, n_ctx=512, prompt_context=prompt_context, prompt_prefix=prompt_prefix,
prompt_suffix=prompt_suffix)
while True:
try:
prompt = input("You: ")
if prompt == '':
continue
print(f"Bob:", end='')
for tok in model.generate(prompt):
print(f"{tok}", end='', flush=True)
print()
except KeyboardInterrupt:
break
You can always refer to the short documentation for more details.
Supported models
Fully tested with GPT4All model, see PyGPT4All.
But all models supported by llama.cpp
should be supported as well:
Supported models:
Discussions and contributions
If you find any bug, please open an issue.
If you have any feedback, or you want to share how you are using this project, feel free to use the Discussions and open a new topic.
License
This project is licensed under the same license as llama.cpp (MIT License).
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for pyllamacpp-2.0.0-pp39-pypy39_pp73-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1cfbde2318c3be1efcb0d76f57078c6db591cfb92229842e9f602885cacea169 |
|
MD5 | b4316d99212e4c39296ecebfce6f0974 |
|
BLAKE2b-256 | fbdf4c2ef8eec1a794e41f3f9758a756b527ee210d9d64e44d8e41e5b8d624d2 |
Hashes for pyllamacpp-2.0.0-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 77b0542210d91e8c0e71646315e4ed077ca902df93393dfb1a5b970b250bfbde |
|
MD5 | bf309aec053fa683635d6f54718a0906 |
|
BLAKE2b-256 | c85d761c9a057b0650592cf563227c32a0da42059902dc58e234e9ab85a810e6 |
Hashes for pyllamacpp-2.0.0-pp39-pypy39_pp73-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a0363b6d6f6c571777c88c61bbe82f6f17e0520a9c8ab380318a01c1991cb271 |
|
MD5 | 0710b375d2b5c3e2e910fa00dd4806b7 |
|
BLAKE2b-256 | 43949f1dedcdb9d91a547e2864ae8543336afc537530037acad829210a9b656f |
Hashes for pyllamacpp-2.0.0-pp39-pypy39_pp73-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 152ba4ab9f044113c5e50f02da8e8b8bd0e61069e2c9277fb9f4099e1e5c2ad5 |
|
MD5 | bc28bbc977f17c28a4e4316be21a8299 |
|
BLAKE2b-256 | 4d61d331e13ccc295401db30f031e59cf89605242b10ad017d733ed12896cdc5 |
Hashes for pyllamacpp-2.0.0-pp38-pypy38_pp73-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fe3154794b5e8ef383539563397f15d9e591ed1ebab32af1d7af93f76936d3c8 |
|
MD5 | 47fa48199b4b508a5e9406b726c77a5a |
|
BLAKE2b-256 | 42f06d649d17d48a3c09ed6a388d51f7b0cfc4d0feefe257ea0d19f209ba446e |
Hashes for pyllamacpp-2.0.0-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a17a761a23b705580994009969b67d09fd74378b82183e80ec412f6273795c4b |
|
MD5 | 65aa5b5ef3cdbbecbc73b9b89ad78e2e |
|
BLAKE2b-256 | eff23b426f7e2e97f8fd1a96ebea2c545a79d150c16a0e80d0c6d433de7b7bdc |
Hashes for pyllamacpp-2.0.0-pp38-pypy38_pp73-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f13b51514abf1e7df821182b0136f415a88112f3ab1fe4056f50d08615d0b783 |
|
MD5 | 4e656752b9384de3c2796c0838c99872 |
|
BLAKE2b-256 | 671d10eed9ebb98e6eda685feed4c9f71282340905b87f760ce8e7265fa1e8a1 |
Hashes for pyllamacpp-2.0.0-pp38-pypy38_pp73-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2c528aaa67616f49ce42e1ed7e1fa7ca2707660032d2ed0b5fdb49bdb46605f3 |
|
MD5 | 25c343ac5816e6f4544db5f13f179232 |
|
BLAKE2b-256 | 39bccc5ec9ead972a33a0f85fb2317e40ac19011c37ce934c1ed026809b633e1 |
Hashes for pyllamacpp-2.0.0-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dbcab27ed610415f57ea536ada38b7f25b7a5a1bc26d1d675a0ddddf4eed7e1a |
|
MD5 | 9ecb0bdee0acf1460676d28d8d7d75c3 |
|
BLAKE2b-256 | a9246c51a3c5538d6db27eee293852328639068724f95eb101101681fb4230d2 |
Hashes for pyllamacpp-2.0.0-cp311-cp311-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 19f53272d75368b4bd1243506513c888a7666ce8eb70015237921e4d493d8183 |
|
MD5 | c3005e08e3431d2084a05181ac2810f5 |
|
BLAKE2b-256 | c57c592aa51d0b879c34cd27e7751a16c27dbd871f9bb1579e2d4109edc454c1 |
Hashes for pyllamacpp-2.0.0-cp311-cp311-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 090838897ec7fb08ee7143aeaa2cfc0dcd5bb7d7ca81a2d5d213d0687f2f4f4b |
|
MD5 | 9bf4b1147a2a1670e6e654d2773acb84 |
|
BLAKE2b-256 | 4ef777942ce87daf281c64016e8c4078f4df08ed3c14deb8ef2a7fa5d293ed9a |
Hashes for pyllamacpp-2.0.0-cp311-cp311-musllinux_1_1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5206470eaedc89f7cc9c5052e023fb77be1ac62a80634c61906d22395325eaf2 |
|
MD5 | 8638c36c6616e11702addbd0afe8eec5 |
|
BLAKE2b-256 | e863508fddbc4379beca7f6be41aa22b192e9b01c01c5d0c9babb2a42c2e5b14 |
Hashes for pyllamacpp-2.0.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cece953b9083c458a8efc47f161fe94a56dd184eabb4805cec0f8f36a93c19b6 |
|
MD5 | c7efbb5e9c9f22071f798bed0ce56ba1 |
|
BLAKE2b-256 | f3edd07729e5a5089cd232788166fe1fab68a89da8902bc40ee5e85060c73976 |
Hashes for pyllamacpp-2.0.0-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b837439a8122b3d7846e180ccd42575c7d07feebbe761c7c5987820cd681b749 |
|
MD5 | 77a97900956906f09ad379d8b075d1f6 |
|
BLAKE2b-256 | 2bc5f8c3c4c97d72209646e584869ec8fe241dfbc717763b3b0f9db78f1aa72f |
Hashes for pyllamacpp-2.0.0-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fec6808247e843e2c8ab2b08a9a16d06ed2c71e4ce6b64f8664f10cf51afb0b7 |
|
MD5 | 3460826c4ff71642de9fa0bd61493eeb |
|
BLAKE2b-256 | 74d3cbe11820b90948ca33c9ab7fb25d145549348cfb191a6e721202e9dd3bb9 |
Hashes for pyllamacpp-2.0.0-cp311-cp311-macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6703ef70637792238f6a922fa3ea12a01954db69244f39e8730b0a8aea188228 |
|
MD5 | 0d9328909676512926055eb4fd9c566f |
|
BLAKE2b-256 | 8602ed5e7994b42e1ec427f5929d39f8aff54becef97ffebc5a3e140643a00ea |
Hashes for pyllamacpp-2.0.0-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e005198a1324c0a47ff7d9c1e13697543036918fd01e18fb803a20ce8f2b2650 |
|
MD5 | 51e0b9b26219bf0756bd2020346017f8 |
|
BLAKE2b-256 | 8b5df9127c159d8e5e71d71cb47bc3b819dd6e38dc098610b3a184e30dae7e8b |
Hashes for pyllamacpp-2.0.0-cp310-cp310-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b964838096ce94da2940f471b45e3fcc210c83eb5dd5f9b29f68b33d368d831b |
|
MD5 | 48e27f6ed757dc282c5661656941ea34 |
|
BLAKE2b-256 | c66e5b8b4cfc956bb805e868258c3d9f7a89e8e2248c2f7de7f9072fda910157 |
Hashes for pyllamacpp-2.0.0-cp310-cp310-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 04f292aa445a3d68734fa35b4a06435a308937748b952c0f5e77273708054853 |
|
MD5 | 618b60c299e361c8db8164c7d0d7f592 |
|
BLAKE2b-256 | dc79c2b4f892e2493632e844c9d00b84e08bf2f091d405292516dbe02b7f0f99 |
Hashes for pyllamacpp-2.0.0-cp310-cp310-musllinux_1_1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7e8d13138de7e7634b50348c35816f6bf92847a8992b6085ea5e17c12338a844 |
|
MD5 | 5c0cc5128a38ca3ae443292edc00faa1 |
|
BLAKE2b-256 | 114773011ae0a8e16096b5ebbaf5bf6d0f46e9388de840a93f5a8a1bf6d713e0 |
Hashes for pyllamacpp-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fdba92e71004af90251a8088c6f769b1d5b7714b5d95aaeafa105b94271a76eb |
|
MD5 | dcae92c30721b8959ff309a3bd62481b |
|
BLAKE2b-256 | 66f63526ad2374bffb213f8da8573834f6f2657b2dbdf2487d2e98bd8280224a |
Hashes for pyllamacpp-2.0.0-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a16c152b97ae57920c9912b6671c5a622a029531350a6635673989fee0849897 |
|
MD5 | d5bc85b19c6e912dd49fc071aec6552e |
|
BLAKE2b-256 | 8e8a0ff08d82d2759161ef88c64d4ddc9af55994709e2abf7c5c54096d57f8e9 |
Hashes for pyllamacpp-2.0.0-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 94e3f2153d1b87a44fa32a5c5a34179acfab0dbfb1b6313fe011cbe188f0a069 |
|
MD5 | 2004d573c6c614a7c1f5a522828bb58d |
|
BLAKE2b-256 | 38083a4f10f49e8cd9655eb97c8bf2d203c53b13c890b318f3ee14654ee0e153 |
Hashes for pyllamacpp-2.0.0-cp310-cp310-macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c08bca28c8ec4da6375b5b1fc402c938becfb85599393063ef6abd04682572c7 |
|
MD5 | 742b7fc57e985fb753bebc2d02278b08 |
|
BLAKE2b-256 | 0a7f64e6deccd0cea7ad3c701a8d904c648720190967101fcb5f6ddc15bbc4aa |
Hashes for pyllamacpp-2.0.0-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | eb23a137701aaf0a55e5316cb1436fc2cfc8ec4126e4efb20931304393141a93 |
|
MD5 | 3b4d7abeb42d3ddc84c99946e2e5a8c1 |
|
BLAKE2b-256 | 005f8a9fc8577defb3a56044da5cae6363b62943ca5dbe0365e3b373ef53035b |
Hashes for pyllamacpp-2.0.0-cp39-cp39-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3a5628a077b49c80a102685f28746690571f1525cbb998fb8bb85c262a632b31 |
|
MD5 | 78f9d0b918f030c3b252e3abc7ca1fe8 |
|
BLAKE2b-256 | 28ba043ca659c2ce4218bdbf8d6e529d6ce99c82e22fbd36a55961e9e2c7c397 |
Hashes for pyllamacpp-2.0.0-cp39-cp39-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | eee2caf8415ece4ee951886279ddc7558c333f4e3e39c27e3332f9f1571da19c |
|
MD5 | 97aade583b29264663bd8f28e4febaa7 |
|
BLAKE2b-256 | f98f73bfeb46f4ce05e94061adb8dbbb87aeb60d524e3d1ee22621642279086c |
Hashes for pyllamacpp-2.0.0-cp39-cp39-musllinux_1_1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 81d35815d4d3cebddc4c63167ab1af0be24f6a0f95236e1c4d917b3b347d8c6b |
|
MD5 | c5ccc9fd4d713d470b9351b8a9c58e52 |
|
BLAKE2b-256 | 32dbe3648f2a6694686edce70f69b36036007f8dfa52fc117115b63714a490f6 |
Hashes for pyllamacpp-2.0.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | df0e30e07c83317950fe6fb31079a2977668ac61bfe8777121964565fa19c249 |
|
MD5 | e72442e61d41ec154015b5394125902d |
|
BLAKE2b-256 | b5d7e37cb315cbf58313dbf919838d653a7d63d8112e6383f9069a52cab635b9 |
Hashes for pyllamacpp-2.0.0-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3241a4e50cfa7bdc02bd6a9f7978ce05ab8f3be838d126dc4a7cccca85a78924 |
|
MD5 | 35e18860cc4cf8ee39a0f8c5d6006e60 |
|
BLAKE2b-256 | 5faff67ae84f56b2083cf21bdfe4dac94109a7f2d235f61425affe25bc2f97d4 |
Hashes for pyllamacpp-2.0.0-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cfa0dd14540761e385aba385f6bfe4fb29f7803d201fdca60d7d5fde7b8076c3 |
|
MD5 | 6f9bf7535fcdc4c10bd85983cc9a35e9 |
|
BLAKE2b-256 | 5e00e4b80417ccd3fdaf82122070812049537b92794671092406043bd2c659ba |
Hashes for pyllamacpp-2.0.0-cp39-cp39-macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 07ae1bba635a35d17f1aa1013b2efa26b44a1f9739bfc2054a3775cd4429f17d |
|
MD5 | 11d1a2d9cc3b1e58c31c420631ca31fb |
|
BLAKE2b-256 | 550afc997b591dd2e60e5476b201f0fd24f71908f9ce7b0b8c268388e27e065a |
Hashes for pyllamacpp-2.0.0-cp38-cp38-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 97fe9975dbe700d6e78a310d261ee8ef26410e8bf59b4918f83ac5fe0eb79ef7 |
|
MD5 | 5e008eea490a03b3b90b1a59e0fed57e |
|
BLAKE2b-256 | 653f82a00b8a9d1bc28b97b5bf79ebf9e1e1481d3a7f555f0d16273da61f1497 |
Hashes for pyllamacpp-2.0.0-cp38-cp38-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3bffc687d836461d406e35d82d3ed777c950ffff07773563e1f0854faec3687f |
|
MD5 | 3fbcbe174a0a8ddb26de494d51f6fadb |
|
BLAKE2b-256 | 8d8749ebfb6056b7ad242d62aa689394c22aa48b7f3aa2c8e1357ae9a603b50a |
Hashes for pyllamacpp-2.0.0-cp38-cp38-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ac19d9cd4adf41caffb295301aff3be1ca4c6fcda648a7e4c6f62adf30170b45 |
|
MD5 | d5ee36bed0628cd13cf1b67b22207c12 |
|
BLAKE2b-256 | 923501d560727e89f101ed0f3bb82b883b70c6b4384f0dc322dd8d8069babb44 |
Hashes for pyllamacpp-2.0.0-cp38-cp38-musllinux_1_1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e93f35581bdfc58e64f1f372a806137f3f9a6c495ebe8b411a9ff92c0af179fe |
|
MD5 | e2ce959ce7b78886294b9b40782af300 |
|
BLAKE2b-256 | 87950df7d338a5a33476a7844709ac41236b970664981952ed68901bf4fce3e9 |
Hashes for pyllamacpp-2.0.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cad137247326d9ba438cb1f84c37bca0667929ab6996539e21400bdce6ecc05e |
|
MD5 | f19e214a924ed133cecd8a72ea3bf501 |
|
BLAKE2b-256 | a547e07792a3e305ae06d14249a7fc3307a6d39e71e25e3bec930c4a0420e71a |
Hashes for pyllamacpp-2.0.0-cp38-cp38-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8f5f54202c93e516aa941271559cf80edbe164c26aa1b6451add4a234c56f0d9 |
|
MD5 | 30873d9fc237b0797d27c26aa0da2ba0 |
|
BLAKE2b-256 | c0da9d4dc5030252e63fe375716b79fc6ec3e2bbce4bbe5f97d737c66d717ff2 |
Hashes for pyllamacpp-2.0.0-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 27cb08199fb6278abc548be0ee2e4cbcadda3a0de23a75fe6f6a983b5f6b23d8 |
|
MD5 | 9356e6e65747c554a37fb3733e267af5 |
|
BLAKE2b-256 | 885c8a41c47dab60ed70e234045ab184c70c0e290aa66c9de3e57b188e35f853 |
Hashes for pyllamacpp-2.0.0-cp38-cp38-macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c887ed5bc312cce6a504477d4bc65bc73eb23a4883d1955c88276312343ffc35 |
|
MD5 | c103cea7b6e4203e98c66c26aa8d2f2e |
|
BLAKE2b-256 | 16b332da70040fad40b177f13048031d4d77eeed77c63feb083372fce8568241 |