Python bindings for llama.cpp
Project description
PyLLaMaCpp
Python bindings for llama.cpp + A simple web UI
For those who don't know, llama.cpp is a port of Facebook's LLaMA model in pure C/C++.
The main features are:
- Plain C/C++ implementation without dependencies
- Apple silicon first-class citizen - optimized via ARM NEON
- AVX2 support for x86 architectures
- Mixed F16 / F32 precision
- 4-bit quantization support
- Runs on the CPU
Table of contents
Installation
- The easy way is to use the prebuilt wheels
pip install pyllamacpp
However, the compilation process of llama.cpp
is taking into account the architecture of the target CPU
,
so you might need to build it from source:
git clone --recursive https://github.com/abdeladim-s/pyllamacpp && cd pyllamacpp
pip install .
Usage
Web UI
The package contains a simple web UI to test the bindings:
- Lightweight, and easy to use.
- Only needs Python.
- Has the option to convert the models to
ggml
format. - A code like editor.
- Different options to tweak the llama.cpp parameters.
- Ability to export the generated text.
From the command line, run:
pyllamacpp-webui
That's it!
A web page will be opened on your default browser, otherwise navigate to the links provided by the command.
Python bindings
You can call the package directly from Python, if you want to create your own scripts or APIs:
from pyllamacpp.model import Model
def new_text_callback(text: str):
print(text, end="")
model = Model(ggml_model='./models/ggml-model-f16-q4_0.bin', n_ctx=512)
model.generate("hello, my name is ", n_predict=55, new_text_callback=new_text_callback)
If you don't want to use the callback
, you can get the results from the generate
method once the inference is finished:
generated_text = model.generate("hello, my name is ", n_predict=55)
print(generated_text)
You can always refer to the short documentation for more details.
Discussions and contributions
If you find any bug, please open an issue.
If you have any feedback, or you want to share how you are using this project, feel free to use the Discussions and open a new topic.
License
This project is licensed under the same license as llama.cpp (MIT License).
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for pyllamacpp-1.0.0-pp39-pypy39_pp73-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5204855a0c03ac8448f0b6a3de76f12ab981f2d35bdd895a0e840c55446eba37 |
|
MD5 | 929e8dc9ec6f107cb0729da9f8e0bca6 |
|
BLAKE2b-256 | 87b30ef5011476d56b752059911bbd412ff85e1650ddeea75d86679ccbc89774 |
Hashes for pyllamacpp-1.0.0-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b6e4e1624149977409238d90ad5c3b6977e8fe618e58d8dbe0bd5183de50b6f0 |
|
MD5 | 2937deb263ec34c2bc4dc955a471ef24 |
|
BLAKE2b-256 | adb9d66f2267037c7426428f0237f3499e43775b1c66e62b4b3ec07f9b116b8c |
Hashes for pyllamacpp-1.0.0-pp39-pypy39_pp73-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c57e9266459b675751dbccf4ef1cba2abee32eac1a66dda6b127a71e31255339 |
|
MD5 | 4afcc3267c5a51fc282135ebe95036bd |
|
BLAKE2b-256 | c44f81a2983a381b1ee12570c28484bd81010a4612cc171bd60f9bf521bfc93f |
Hashes for pyllamacpp-1.0.0-pp39-pypy39_pp73-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 09e98c1d1a43d54f13dbdc99232573fab6105f427eb3daadaf23dd21aff79807 |
|
MD5 | 2ac57ade55092087fb6525df12d174b7 |
|
BLAKE2b-256 | 7c6f3d4d6f9bdaa04391e1d91f4db99aadb91e4e51c3590e33a66f892875a4e3 |
Hashes for pyllamacpp-1.0.0-pp38-pypy38_pp73-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 21f6247926b89a45f3c75ca4da4d7310fe7d908dafe444c158f0c20247554bc0 |
|
MD5 | 1969f8839616551114461c6288b0766b |
|
BLAKE2b-256 | a777256ccec71f59036ccbd8147325e250f401f40e1443b977a84edf3ed0d6a9 |
Hashes for pyllamacpp-1.0.0-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | de4ead215a5ea9e683fa47a2c238152f175f7c6089b60b6294187475481b89af |
|
MD5 | 2d4e656b1ca6d3a299d67b880f99c4ac |
|
BLAKE2b-256 | e944c4dc29fbd6e7c1d6eabdbff0918360d299f782f041ba41ad359fa903f311 |
Hashes for pyllamacpp-1.0.0-pp38-pypy38_pp73-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9f6007acd4c40e4ea36d57bf7a16663256c11dc0f2ccc5b770e9cbfd179a0335 |
|
MD5 | 71230bb750f551c7b6cd37d32ccb167c |
|
BLAKE2b-256 | 67e494ef721bb91ea95ab91834a0b9e4129501ebb7569ec77f278463a0847e2f |
Hashes for pyllamacpp-1.0.0-pp38-pypy38_pp73-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b01633a5418736634d786c5e648220f8b5d978562c14bb9b200eca138c11e00f |
|
MD5 | c18c40d9b604dc5faac99e7d6da63b10 |
|
BLAKE2b-256 | 6b8a756a3de2cb2ddb95b51315dcd8a9862d6568eb65989e7a339fdb510764d2 |
Hashes for pyllamacpp-1.0.0-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4d08f10852219e3c49ee7e224735b3e354f6a562c059c21c8b42f3d3f89c9734 |
|
MD5 | 0df876a19f1bc35e5a5e70f898981fa6 |
|
BLAKE2b-256 | 24476c6a234af65d3a4947e2f49d9068d58f5f150c8915989fafb7da80371107 |
Hashes for pyllamacpp-1.0.0-cp311-cp311-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6bead9e78ca098bd0e0909932561b8fd2ee06aeb0530c548ddf165b940945913 |
|
MD5 | 0714d5405766205bf58de83224c9247d |
|
BLAKE2b-256 | 257c4df5f741371ce14caa02162e22b06b785898ac17c802dcb219fd483d84e7 |
Hashes for pyllamacpp-1.0.0-cp311-cp311-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f41c47329e7e3e70467cc1e103a81971ed43546fd49882a1193b3231380b95a8 |
|
MD5 | 16f40384d248535ba6135fbec8c630a3 |
|
BLAKE2b-256 | 84728a2a0bcf53c8d91595194b953f00111c6b6dd02b1122e0084f333b8447b9 |
Hashes for pyllamacpp-1.0.0-cp311-cp311-musllinux_1_1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 31bec9a416c5fd249c2b956353cc45655f89e9597fd893440a3edcc4248a0166 |
|
MD5 | 95bb1a41c6f43444b4884d765a036d0f |
|
BLAKE2b-256 | 0389521c686173486a25242745b6de743b9b9caa38cf1ff05b5a566709edb9bf |
Hashes for pyllamacpp-1.0.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 165c62d75520ac00fd57c9f9b1d70c18311ef4858dcd393e44b9909fa8e409ae |
|
MD5 | e2f39add42fea400b2f588baf635c235 |
|
BLAKE2b-256 | 39150369f4f4679f7fc919ac5640ba460864fcbfebab7e42fb2287d0db493086 |
Hashes for pyllamacpp-1.0.0-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e7b236e8824d36719e5a29a3457dbce49a5ea042f7b6341d83f7d6f92086fdee |
|
MD5 | ea8a220fca7dfe9d4cb567d0abbb6fae |
|
BLAKE2b-256 | bd8194c6058688960745ce6975123a324fc84be674b146fe1889375fb0e2e3f5 |
Hashes for pyllamacpp-1.0.0-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ac0fc2642478933116e42f06c2ff1c7572c0c9b20ff89fa5fd3d1f3f8e64befa |
|
MD5 | 69d2125b001be035c3ac1c8f16e536c4 |
|
BLAKE2b-256 | b3e47aa8de7d6187d019f32082a32610e9c789c27decd203f19652088364a08f |
Hashes for pyllamacpp-1.0.0-cp311-cp311-macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f926b13c0755a30be5cfde5835953fe341456445fe65801d82247bed1e14d44c |
|
MD5 | 9f6386d06a7d388e7929ae6e62358ccc |
|
BLAKE2b-256 | 1629180d4cb07a69ff63fa3b8c10ce512b8b042a4034de887bab65494d0cc800 |
Hashes for pyllamacpp-1.0.0-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3ef8c75b7ae12912ee33a327613cc9a572b3b5b0869f21c1d82a0224604d209e |
|
MD5 | 0d0a7024ebdcfb2887ac77484e1756a0 |
|
BLAKE2b-256 | 52a1ea67b4c86a322622e9bece67a2f6416f8e49d9fc202f8af076e781476f3a |
Hashes for pyllamacpp-1.0.0-cp310-cp310-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 375cf4dc8e94e4bf3f51b9cb1c1f295a5ef74da7037b8ac28264749433aeeb50 |
|
MD5 | 8f3a22303abb411d9fcffff45bc4ab3b |
|
BLAKE2b-256 | 99d584e9a6e4489baf121d0ae1c679498c398ff6ce36926f97a98e8c64596829 |
Hashes for pyllamacpp-1.0.0-cp310-cp310-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 46f5fc91581e729bb161cd60b905f2fd132feeec3205cfeb54c5a4ae2d9b9c3e |
|
MD5 | 20debd22d966cc311bf6b1ec1b927029 |
|
BLAKE2b-256 | 28f98deb497c36e684841b91f83d74584db4ed64cf30be9b98da82f1ea831fd5 |
Hashes for pyllamacpp-1.0.0-cp310-cp310-musllinux_1_1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 24310dfaee4ae66ded7b76e9a1bef28510eaac4b9edb1735dcac5492d4e75501 |
|
MD5 | 74318886a293a28aed3d3cfd8b74f006 |
|
BLAKE2b-256 | e279478d61c6a0a75023f19d17bf95ded30ed291389359951fe8c141b8fed212 |
Hashes for pyllamacpp-1.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c01efccde64e40fd092d31126e688563a9f73c702de75634ab6df4df9c98509a |
|
MD5 | e4695e7b7b232beed8d997bdd74680da |
|
BLAKE2b-256 | e31e75ea5f3c4d70ea32a592ff3de6eb67441c0e75a5684a3687bbe44a906189 |
Hashes for pyllamacpp-1.0.0-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 472a7b7383edfaed35ea8206e597c492ff4a3f7286ac19ce69c9a372ed996ca8 |
|
MD5 | c83f9396adc22fb1ce2ee3d5f002d0f5 |
|
BLAKE2b-256 | 1b7bfbf9a714f1ee7899ddd02af10e83ea2fd9c63ba6b1b004cf05e3c1b702c2 |
Hashes for pyllamacpp-1.0.0-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 43e0955e62e23e32705d045128247b235bd9ca8b8be86de7f22349f63f7c9890 |
|
MD5 | c173d418dbfa29c0016f6a848543818c |
|
BLAKE2b-256 | 878a440416b7552cb5abf20df86a04a00a56e99a0368edc2b8d1e55b0c3e272d |
Hashes for pyllamacpp-1.0.0-cp310-cp310-macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8316615b454739bd752f2d3d5bbc7460accdadc5b3fe1e917fbf8ecfcf593a05 |
|
MD5 | 4d09cc8e13a9e97fa3433b3ca65c98a3 |
|
BLAKE2b-256 | 8f42ccf6c252ab27a33c7c2c5d36f332306f1e54fd268f3b0a378bcad9a07661 |
Hashes for pyllamacpp-1.0.0-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 84c449b12efc9c8f903b986bdcfe363408ce4b19e8cb71f26e543cb934f6150a |
|
MD5 | 2d64c8cf5155d7d6e0ac634ebaeab121 |
|
BLAKE2b-256 | a9ebc5c681bda45e01b08fa523d1a98897d46661a66d0ad64b08ac906e2b1d00 |
Hashes for pyllamacpp-1.0.0-cp39-cp39-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e56f0fd36b02769ccdbe7890a6ff742eac08e8a9b3de36615ab1beeb0e0cefb5 |
|
MD5 | fa1fa110116f1b537198da65b9754ce2 |
|
BLAKE2b-256 | 3db67a6eaa7c11a592f1fdd27d61c271b67d3cb5ead61894d847fc31c0e9b356 |
Hashes for pyllamacpp-1.0.0-cp39-cp39-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 90224abdfa38eec30cf2c62b8cf4dd196ce32240390388b413ad7f06917fe44b |
|
MD5 | 6bc7f32b57532073519ebe199a80613f |
|
BLAKE2b-256 | a681c877c7029dfd70356d0efa45c64d4a6568b96c9faec30df8ff468960c16f |
Hashes for pyllamacpp-1.0.0-cp39-cp39-musllinux_1_1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dfdd0f4f18a91eb290bd45c9ae4077a0f3553a0e5270f5d02d702dd4ac83881d |
|
MD5 | 44bac9e92311b17078e7383ff2f9e5ec |
|
BLAKE2b-256 | 43e8b2ed6d0ec47a7d10945f35ae6e67aa221fa88de45ac8ee53afd7d6e2b425 |
Hashes for pyllamacpp-1.0.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2092f27741cc298baceef9a254fcd6f345e74bce8e45c0fa4967109cd30c097b |
|
MD5 | 84516ec6fae0ab907e16870b16141c95 |
|
BLAKE2b-256 | ba1cc8a9649e4b336093fb806a697cce17cf8a6d75c23fd7c324afb63819c742 |
Hashes for pyllamacpp-1.0.0-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c407bc8dc3b8d4a986c4a999f412407496658baaae10bd39940929952745cf52 |
|
MD5 | 713976a8a96f3d8413ca0044592eb662 |
|
BLAKE2b-256 | c5a2e4c3289235722ee0488a521e78bfef62fd05415421a1c8b64c260737dc0f |
Hashes for pyllamacpp-1.0.0-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 370dd4c364c548908964850f48bbd33ab56274c9e16f7e0053ef30628d24c014 |
|
MD5 | 666e365c0194d9b666d0fea6c3c9599d |
|
BLAKE2b-256 | fdac09337f0bb5923b01224adacc11783d564b712d9b1e322e73ad8ac3f93f56 |
Hashes for pyllamacpp-1.0.0-cp39-cp39-macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 60523ff895d1614fbf0265519d32dc700fca8b09f45f7bf1d52c346c3270d322 |
|
MD5 | 6dd4cf18be9f7875c8bfa1d681af2912 |
|
BLAKE2b-256 | 1feaabcf6c0dc9e1e7a1e33e756acecd06b066563f8c0327a2252e35098e3c72 |
Hashes for pyllamacpp-1.0.0-cp38-cp38-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3e26fd0ba5f4d666f5d38c1d341972b97b09274387127191cd34be4998368bf2 |
|
MD5 | 569e116e81b5aa63a0aa178fa8487228 |
|
BLAKE2b-256 | 45ba09778c991ec86630d40628949b26dc87bf46a8f988d95b697dc59762e4da |
Hashes for pyllamacpp-1.0.0-cp38-cp38-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 327d9f5af2a2276c73af8a3f0357927d3c148945279db48d20a464098790a93c |
|
MD5 | 89050d6854201ded2fafbabe6bd77aea |
|
BLAKE2b-256 | cd6a7cd9dc393b65aca02577983612340c949b552c98c603273dc4c798e1aa12 |
Hashes for pyllamacpp-1.0.0-cp38-cp38-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d26cf96b24192a0fdc2fc01a820177c4a529ee8c9d50ac772fc7a0d3d2d0cd97 |
|
MD5 | c9b829bb9fc76b86f10e83bc5978a405 |
|
BLAKE2b-256 | 676f64dc08d93e30a62ad5d96de0e82c3902ea1a07e76cb572bdb3cebdaf2352 |
Hashes for pyllamacpp-1.0.0-cp38-cp38-musllinux_1_1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 96842c332b5fd7e5ca012e39c38068d1f56e304900f9f930c42e130e39035074 |
|
MD5 | c5677b7646610a52a8c1d848e8947a76 |
|
BLAKE2b-256 | f318ce6e2dbae6c4022554a21afac4c9148426cc2d2676b583f4e4836cb7a6c3 |
Hashes for pyllamacpp-1.0.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2418cb6dca3a1cf2ddc39b8d713f983c9a7d4a63decad3c2112613659dd6a724 |
|
MD5 | bf36e68387b39085c52ca71b53ff05c6 |
|
BLAKE2b-256 | 0fe607f52b304339f37f3afdb85a0521e72f01cacec19ce42c2f74c42b9a7c08 |
Hashes for pyllamacpp-1.0.0-cp38-cp38-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c56eb7394397ea20d37a2e10206fe137f9d29243d78091e77719df1f7ed1663d |
|
MD5 | 5501bfe8797515847571c4bed716e4f2 |
|
BLAKE2b-256 | ee806e65b368d37f69cd605e2e9036406c1992c161e68492cb96fb963e9b30cc |
Hashes for pyllamacpp-1.0.0-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cf44829db27f15b191b5ae3f74d0f5a475138a801eb7bc93721225a3e87c7f4d |
|
MD5 | 00812e70b4cd8b7482c6601eef82ea73 |
|
BLAKE2b-256 | b6008b10c8dc23ff8218606e4c36b062ce9ddb4f01ff1ec36bc4477007a5abcd |
Hashes for pyllamacpp-1.0.0-cp38-cp38-macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bcb22ab2f6fb6f05836f6444672caaf26203debc7b5803131ecdb27a3a2a3722 |
|
MD5 | f9d456eaac562e51f72281e82901adc6 |
|
BLAKE2b-256 | 10f8d99869818f97f0a9ed4e5f602754caf73112a69a122d9bedda867ef29779 |