Python bindings for @ggerganov's llama.cpp
Project description
Python bindings for llama.cpp
Building the Python bindings
macOS
brew install pybind11 # Installs dependency
git submodule init && git submodule update
poetry install
From PyPI
pip install llamacpp
Get the model weights
You will need to obtain the weights for LLaMA yourself. There are a few torrents floating around as well as some huggingface repositories (e.g https://huggingface.co/nyanko7/LLaMA-7B/). Once you have them, copy them into the models folder.
ls ./models
65B 30B 13B 7B tokenizer_checklist.chk tokenizer.model
Convert the weights to GGML format using llamacpp-convert
. Then use llamacpp-quantize
to quantize them into INT4. For example, for the 7B parameter model, run
llamacpp-convert ./models/7B/ 1
llamacpp-quantize ./models/7B/
llamacpp-cli
Command line interface
The package installs the command line entry point llamacpp-cli
that points to llamacpp/cli.py
and should provide about the same functionality as the main
program in the original C++ repository. There is also an experimental llamacpp-chat
that is supposed to bring up a chat interface but this is not working correctly yet.
Demo script
See llamacpp/cli.py
for a detailed example. The simplest demo would be something like the following:
ToDo
- Use poetry to build package
- Add command line entry point for quantize script
- Publish wheel to PyPI
- Add chat interface based on tinygrad
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for llamacpp-0.1.5-cp311-cp311-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 07c0da2914d21166d6542a7b86edd3d8b6f63634e49dd6feaf05a9eb553848c3 |
|
MD5 | 2c21589b166ad107f7f9db8608b460a2 |
|
BLAKE2b-256 | 82950df559c09239040d978dcab6c31be3b55f6b254d881fb10f79e51ea5a4b8 |
Hashes for llamacpp-0.1.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1af7d0b56ca9a7c6e97bc39473b894ef19971ae8c59cf96964900c04a3eb9d11 |
|
MD5 | 0cb0465b44e311d20989d9e64eac7019 |
|
BLAKE2b-256 | 973fa8cde5d9cdc0f8c33c11c9a244a249d8af21e462691bb9248be71bc4c329 |
Hashes for llamacpp-0.1.5-cp311-cp311-macosx_11_0_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 043e0b477d105e63226a25346425bb2749da417d8ac21bf9552978a4bc675699 |
|
MD5 | 1dd1e85d6658f29ff6a6c232602d2371 |
|
BLAKE2b-256 | a0242b37faaa6e90f334e7fa3074d41aa0f26a5737cef0690d50392f6bd19d53 |
Hashes for llamacpp-0.1.5-cp310-cp310-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b17e3d6a31eb237ec3013db010bc09d5c600f630cc8a11f6a5cdbbb711f1b1ae |
|
MD5 | e2c690f46754fd0e092126910a6b6451 |
|
BLAKE2b-256 | 07a8e6a5020c7c952a2eaa7de96a192fac45f8ca07ff48f7e14ca2a6c393f0df |
Hashes for llamacpp-0.1.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d7283ef2c97b0c38f49bc90580ee269c653505be780c105c23bf7e63727a64d6 |
|
MD5 | 56494eb6a0e7981fc3a3c68618193fd9 |
|
BLAKE2b-256 | a74d28f2455824e4fe9bf58ef4eb71f79c303e28cb7db3d6a59b85daaa12c8bf |
Hashes for llamacpp-0.1.5-cp310-cp310-macosx_13_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2d4be52bc26becfa771bb866856a55e51fb9d603113c28b9048d7c056f9d3103 |
|
MD5 | 231f60e9ff674ae780f82ae5b4955dc8 |
|
BLAKE2b-256 | 790dabfeaad59bf4083375f64fd088bbdc9476c5cd40fa92387ca1650afea6c5 |
Hashes for llamacpp-0.1.5-cp310-cp310-macosx_11_0_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 237c33285a64e9c4626f92e88ba596e1868acb9331ab3f28fc840cadb8c4471b |
|
MD5 | 6342af5ac5d29756415f4399887d1167 |
|
BLAKE2b-256 | 2db201f54ab9f090183cfa7c9a324a64b2101f6f7f19169634b90a0098e7e6e0 |
Hashes for llamacpp-0.1.5-cp39-cp39-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 518321ffffe0aaa495a6d8b7957dbdc2db8f154184b744ba4ae50ce77cfccb73 |
|
MD5 | 725dd76afa814b875267510f1505e29f |
|
BLAKE2b-256 | 20db350c16994d600a48548823a4fa28a17e5dcb2b9616b5e1517fe7502c5fdd |
Hashes for llamacpp-0.1.5-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e18f0b4caebbebcdc34ecd257d6327989107dd8734a203374b1f97a1374212a6 |
|
MD5 | 2db207046f046632b40d8466f29b1d0e |
|
BLAKE2b-256 | e4df473373751f019b5ad55b50410eb70349a878f32beabf8cbd88ce04c03abb |
Hashes for llamacpp-0.1.5-cp39-cp39-macosx_11_0_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7d147dadbc6ecce543e7d4964f18d2f2d1ce30ed3bd5e314c05a8cfe8d9f51ab |
|
MD5 | b99fcd759a55179534237c99d5570ea5 |
|
BLAKE2b-256 | ae15c964ae0d8357f73ae8a438ed381ac6669fdc5cd0e4d5cec0b5e99a739b2e |
Hashes for llamacpp-0.1.5-cp38-cp38-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f0f88daeecdc5c7655649bce201c0a9456a3efe685ea17aa6d4f310a01ef4f1e |
|
MD5 | 08a24b56c4e86495cc9b86ce125e61ec |
|
BLAKE2b-256 | fe5d776d24edebc2f784464b5a143e502445e5e13a8229e7a0d38bd009dc5996 |
Hashes for llamacpp-0.1.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 80eeb5fbcfd3890d618e138e8edb4e566505a14e3e956f7f436aadbc2c44b033 |
|
MD5 | 26925f6b6f5e3be96e1ab426512b36e7 |
|
BLAKE2b-256 | d6daca1b566492f1675c5ad21896aa2c102bccf91ef937afc00f43fbf0ee6c67 |
Hashes for llamacpp-0.1.5-cp38-cp38-macosx_11_0_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | aefac4b4fd729528f2549665eaa3caa62031f765f301d6c99e8ab755e676e246 |
|
MD5 | 088c502853791d4641345c17e7fb57ae |
|
BLAKE2b-256 | e855e6dcffe53e4f88737821844d9338696a6fcc965081fd9add2e27abd6bfe8 |
Hashes for llamacpp-0.1.5-cp37-cp37m-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 759278a7027408e5636c00a1bba624fab8975e49f22d0e0a1656f27434f2a733 |
|
MD5 | 6a99ea1b893cb92bd08ae60d4e9df516 |
|
BLAKE2b-256 | 8c8eebd86f6c7495a0c2d7f6457f96c5b52c4b1c61cf413178ee4a5c2e036a5f |
Hashes for llamacpp-0.1.5-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cf13e1fa8ab1a98afcb802236a4abe9c82e8e6f1f2690c14fbaf7c60106a49f7 |
|
MD5 | 250fb132d581580bfe4a4bd338f55501 |
|
BLAKE2b-256 | dcaf787e3fd806c2ff89389e15460b2171ee6aec72f57ec3948ca35eaaa8b530 |
Hashes for llamacpp-0.1.5-cp37-cp37m-macosx_11_0_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8b3241b595b70a072d1f491b7faf977ed50b46c9e5d2e8c40e97406678eafc3e |
|
MD5 | d8d1142f8946e6aa2051eabce5d2d85a |
|
BLAKE2b-256 | 4206c4d379b4840d3af9b5bc42a6487801bc06961e9517b7ee58f2c89f62302b |
Hashes for llamacpp-0.1.5-cp36-cp36m-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9055c45c85b1e91f83e1a834b791719106453efde2e6db91227361599fbdc716 |
|
MD5 | 814db34b85439bc9470d29a647fddfbc |
|
BLAKE2b-256 | a94324e122e3a8aec39379d36275c5a6ba82152b7424a8b4adc423c6dfedc03c |
Hashes for llamacpp-0.1.5-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f5e4bb94ea4a533530c3e5f368bc7d3c67e24ee2a80ab08abb873e7f029141ba |
|
MD5 | 3696b00812eb6191fd6675b85e817b9f |
|
BLAKE2b-256 | ebaa020bab8d06b4a6aeed78dc63791b0ec82bb6d402cd6398cff7ded45ba509 |
Hashes for llamacpp-0.1.5-cp36-cp36m-macosx_10_16_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d0f912fbf7f96bea60830e48ed6f31cd4995ad1da2366e1293d8abb55b70ba88 |
|
MD5 | dcd74a15b981026928a636ce3e5c3c07 |
|
BLAKE2b-256 | 90b0a302820048b4801008657c23606fee471aa491efc6047c4d903db72d7aab |