Python bindings for @ggerganov's llama.cpp
Project description
Python bindings for llama.cpp
Install
From PyPI
pip install llamacpp
Build from Source
pip install .
Get the model weights
You will need to obtain the weights for LLaMA yourself. There are a few torrents floating around as well as some huggingface repositories (e.g https://huggingface.co/nyanko7/LLaMA-7B/). Once you have them, copy them into the models folder.
ls ./models
65B 30B 13B 7B tokenizer_checklist.chk tokenizer.model
Convert the weights to GGML format using llamacpp-convert
. Then use llamacpp-quantize
to quantize them into INT4. For example, for the 7B parameter model, run
llamacpp-convert ./models/7B/ 1
llamacpp-quantize ./models/7B/
llamacpp-cli
Note that running llamacpp-convert
requires torch
, sentencepiece
and numpy
to be installed. These packages are not installed by default when your install llamacpp
.
Command line interface
The package installs the command line entry point llamacpp-cli
that points to llamacpp/cli.py
and should provide about the same functionality as the main
program in the original C++ repository. There is also an experimental llamacpp-chat
that is supposed to bring up a chat interface but this is not working correctly yet.
API
Documentation is TBD. But the long and short of it is that there are two interfaces
LlamaInference
- this one is a high level interface that tries to take care of most things for you. The demo script below uses this.LlamaContext
- this is a low level interface to the underlying llama.cpp API. You can use this similar to how the main example inllama.cpp
does uses the C API. This is a rough implementation and currently untested except for compiling successfully.
Demo script
See llamacpp/cli.py
for a detailed example. The simplest demo would be something like the following:
import sys
import llamacpp
def progress_callback(progress):
print("Progress: {:.2f}%".format(progress * 100))
sys.stdout.flush()
params = llamacpp.InferenceParams.default_with_callback(progress_callback)
params.path_model = './models/7B/ggml-model-q4_0.bin'
model = llamacpp.LlamaInference(params)
prompt = "A llama is a"
prompt_tokens = model.tokenize(prompt, True)
model.update_input(prompt_tokens)
model.ingest_all_pending_input()
model.print_system_info()
for i in range(20):
model.eval()
token = model.sample()
text = model.token_to_str(token)
print(text, end="")
# Flush stdout
sys.stdout.flush()
model.print_timings()
ToDo
- Investigate using dynamic versions using setuptools-scm (Example: https://github.com/pypa/setuptools_scm/blob/main/scm_hack_build_backend.py)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for llamacpp-0.1.11-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9b28031f20787134a52f8ee6715e752d40819cca0d9d9dbf2109d783e9e2c289 |
|
MD5 | 2ec8fd2b154c56fe050dd3ed3ed99d9f |
|
BLAKE2b-256 | e7ca3eb4e680717bf2f6f13e84898283f05e1f7b194abe2f519a48968bf85c27 |
Hashes for llamacpp-0.1.11-cp311-cp311-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ce27cba255b3327e3ccdad256ba57d626bfeefd0db7c558aabf0e5dd8d45c11d |
|
MD5 | 3cad229a51215300d5d363f1f7325883 |
|
BLAKE2b-256 | 1828c0e8d79779e70948cbc1d57d87d3f90b1a2fb0bcfeb8ad9ac13a52a65750 |
Hashes for llamacpp-0.1.11-cp311-cp311-musllinux_1_1_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 95a8891ef78290ffd7e53da54a53ebd2942d68aa7d94f954ee3c2368de164fbd |
|
MD5 | d788bac785e941454961aa3223065b3c |
|
BLAKE2b-256 | c3e147299e8f81d37fd47da5e5625c9fcf54693bc681181cc4961c31321d65e2 |
Hashes for llamacpp-0.1.11-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f0c2f001f0cf0c63d57250c5d8ac4053bbc5b00df31af51cb7fcd9bae9d72c08 |
|
MD5 | ba8cc115f1e56f3d93228f0da00ccf0a |
|
BLAKE2b-256 | 464f380373ec3ce45289767db8b815db7e08d0b38275ac09d1b1046d86f34ca0 |
Hashes for llamacpp-0.1.11-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f2a4b2f26b533975457caa0ca421994096ada22ff9e54c4fbeb3fd8e2725692f |
|
MD5 | 2db999b3ae6ba2e47f0b6811c136d613 |
|
BLAKE2b-256 | 078e435fc20f677047a08d0107155ef6f8f60483aad7bf16ac76fa35f0e248ce |
Hashes for llamacpp-0.1.11-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a2e28a5bb8cbab66368ebcc5d73dcd90558fcf1558c803f544a49fb40691b58f |
|
MD5 | 10b46716c7c66fbb1460d29d5cc1c4a6 |
|
BLAKE2b-256 | 2afaf297b70e4c11500f92470356f695e1e75ed40dd1984b1a7ebff1275bc30a |
Hashes for llamacpp-0.1.11-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dea37912c24a6a94b26557d8b98834df2e916f835dce246188d6e78ae9ec21ea |
|
MD5 | bb2ebd35616f36c7079f85715009a269 |
|
BLAKE2b-256 | 84a301ba42bb768834f8541998937c3f7473421aa18197b387bb4d2139f5b7de |
Hashes for llamacpp-0.1.11-cp311-cp311-macosx_10_9_universal2.macosx_10_9_x86_64.macosx_11_0_arm64.macosx_11_0_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fbef4f6b9d1036ce6fa5c8f44fa0a6631939fd44a6a784d88bbe2d7e703dc8e9 |
|
MD5 | 53ea2d3c98613b31ed21035b485c879d |
|
BLAKE2b-256 | fde19959fca932fc5d6d470c32a14a351eff5c9c8c1d6fcff8693e50d467db4c |
Hashes for llamacpp-0.1.11-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f89a08c72894f3782edb6fcbc70e41e0af2a4e31463f3701a21ac6d0dd4794d0 |
|
MD5 | b02ea644c7fc7b1828f9945cfa78f1b6 |
|
BLAKE2b-256 | bec3a17c3429a55c959a1a403c9f9dd8e76c715e575bfa7f28238eac8f6c62a8 |
Hashes for llamacpp-0.1.11-cp310-cp310-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ea6e598824c520989801862eac7f8b98d2aa7ed6f3a1a169857324fa2806e8e0 |
|
MD5 | 96d93dced27f482481712f964f5f8768 |
|
BLAKE2b-256 | da35fd7536bc7e12dbb2fcb4d9da73431b59d6211ea0162b5480208046880ee9 |
Hashes for llamacpp-0.1.11-cp310-cp310-musllinux_1_1_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c9d30712c73172abbda41d9412ee19856fd9ccd144b9e936aa87bf8c43cb5a6c |
|
MD5 | 9e288f0afeab99cd195892cd8a3564f6 |
|
BLAKE2b-256 | 3e510c7cae5733eebebb5a447f843a60b113dd763f1423cfb02ca60e1daf3f87 |
Hashes for llamacpp-0.1.11-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a08cee46b62009cc7e82f62d028f3b7fe9b8780f670fe1b4c866db151a40cf4e |
|
MD5 | 85a23e2d9de183096353e75c462f87ea |
|
BLAKE2b-256 | 0c338dec901eb5160186e10ac9bd856ed43c4f251449a30d54c36dd3bab9fdcb |
Hashes for llamacpp-0.1.11-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2d5fe4730c701f5e6c5cc8d5ded3c522a2af4911441c9e887be8ebdfb5ae69ca |
|
MD5 | 173f2979f9fd29e1a28827a4265b6d6e |
|
BLAKE2b-256 | 3a895faf4174f233876ed3cfebbff51247989700cf9f19ca013034b902647d4b |
Hashes for llamacpp-0.1.11-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 765a5fda5b53b12a22f6ec465567fcff6faccb20412d58c94343754bcd588939 |
|
MD5 | 6d76c24aa48d427669c5818293a8fa44 |
|
BLAKE2b-256 | 26f9e1312c05d5f52300189cb39d7607be21413c9a0c002dc17d8dcf758caeea |
Hashes for llamacpp-0.1.11-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 85712a155a2bdb224e29973a11bfb2aed816d18808a234d6e6edbc9b108e6137 |
|
MD5 | 5415cd5d2d1e02b2c2089aeb90ce6ee7 |
|
BLAKE2b-256 | df69758c42428d04bb1b3e3216210e4ec191a43a164121e2fc03083b3e190fc2 |
Hashes for llamacpp-0.1.11-cp310-cp310-macosx_10_9_universal2.macosx_10_9_x86_64.macosx_11_0_arm64.macosx_11_0_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 93060e27e9f733d1b49425fa887f00cf982ad3cc9e3fdfe9acca62f5b6351529 |
|
MD5 | 4c32f1fb8cfe417c1ad79db9ccf78d6e |
|
BLAKE2b-256 | 94ed4dbd61932751ddc1f52282471bab55d1c212956446afb0e96c746b7da308 |
Hashes for llamacpp-0.1.11-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1617c4980e6507dd7c9ff48271e02ba3874d7167d6acc3d42a111cc44c799e30 |
|
MD5 | 9403fa040de24c55955ef8fde01dc019 |
|
BLAKE2b-256 | 9dfdb89d57c27c8719f4f9583903c6b7f31d305dd32dfff14899bd4b84cd33a5 |
Hashes for llamacpp-0.1.11-cp39-cp39-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cc6c6468df5d4d7d26f676cd80ffa496895866fce969fdf2326c94162f2b12be |
|
MD5 | 30c8c3c4358420964834b024c59d6178 |
|
BLAKE2b-256 | 6a5401a87c53d374e260c8eea07a64c7a6f944f2096ecbaea7c8355b61a62d49 |
Hashes for llamacpp-0.1.11-cp39-cp39-musllinux_1_1_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e5fa490fe9088168f2a541b8f56d764f30421a7251d1ff2a1ec2db34efac95ff |
|
MD5 | aeb92c6a430d04f69fbb2fe8e08e8e72 |
|
BLAKE2b-256 | 669c9f011e96158bea01279f0ff2c55cfd05b65daff2cdcbf6fa994e017a98b1 |
Hashes for llamacpp-0.1.11-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a2cbb57892e96161f20983b4680ad47518016bfb7bdcc7f7ff57db98b7d62509 |
|
MD5 | 0bafa5abfbcadfd3a490f9d9228edbf8 |
|
BLAKE2b-256 | bf313261426432812428eb7d2607e2182d61b208fb7229408db931fba81ca762 |
Hashes for llamacpp-0.1.11-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 798c00f758f791e67cfc6e0ce5f4aa01ac0054537c29a02faba2c463f8e88e01 |
|
MD5 | de3903a68ac71dc4a7df03a4712e0e75 |
|
BLAKE2b-256 | a5b693a8f39800f4d30d9721932c6e5c7f6f6a2e07961fce4a70ba98cc34163c |
Hashes for llamacpp-0.1.11-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 51bcb3877e5b6fd892f18d144bf94859281f535ce6e34ba4c497effedcec0b55 |
|
MD5 | d8ebf0ff3d9ecd0ca0d51f284063ac02 |
|
BLAKE2b-256 | 1b76248720cf1c08bf7390c93fd29d23c83fa60cc2054f59ee0694d7ff9ac68f |
Hashes for llamacpp-0.1.11-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4c996c62450a4f963e08cb4335c4a3597a199d5247bdbc519e40cb215a6eb832 |
|
MD5 | 2a7c3ebf20b9ac10923382a88cff0a52 |
|
BLAKE2b-256 | 8f2e2a74d80025c88dd919868bcb2c41736849747a37f5158951401af4762c97 |
Hashes for llamacpp-0.1.11-cp39-cp39-macosx_10_9_universal2.macosx_10_9_x86_64.macosx_11_0_arm64.macosx_11_0_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5ed0fb9cce90079164617aff209b0bf9a82cdba5669d7009044403447dea4b43 |
|
MD5 | 1cfed42dbd323ce899135c5747a4e7cb |
|
BLAKE2b-256 | bdfaf131820f49d64f1e3753371f81afd34637d62e2797c5f62ab4863d4f384a |
Hashes for llamacpp-0.1.11-cp38-cp38-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2e97adaae83864c951e032eb4a01ababc5322848ce99d57ddf6128f98c733aae |
|
MD5 | 3e93bd14ee5a09054944b3dd48e3b1e4 |
|
BLAKE2b-256 | 42ec06df8a9aa040816ee0e46378efb3c8c656abe25219a06442cdbf024c9b3f |
Hashes for llamacpp-0.1.11-cp38-cp38-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8e7c5da0942bbdc893586f46751f9f47e75e7e915a1e0020118cf08f6bc7502b |
|
MD5 | c53206ce652937c334dfec700b5bba84 |
|
BLAKE2b-256 | fdeed75158567ece01476051c59d1b4529c4301dccf04b0554a051f8b963e2c0 |
Hashes for llamacpp-0.1.11-cp38-cp38-musllinux_1_1_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 37e628e8e1958c0fa20404548c2eef0901eda7faffd7598a6720e0361a893069 |
|
MD5 | 6f7670c87b362a847f213712ee476a4a |
|
BLAKE2b-256 | db2caf6b483200e22e856528bb8f6f00ca59777f14a699ffe2f63e3b4df9f3fe |
Hashes for llamacpp-0.1.11-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7a9590aa5c6bc7534e3468ad2917464d1b5b220f7759dc5be79e970947834f65 |
|
MD5 | bb6ed1350bd1b057ab3080cafa30d7ba |
|
BLAKE2b-256 | 0eca9e18bc39ce4ac1fe14d8bc3901dd4f293415d9d3be21432c69953f1300b8 |
Hashes for llamacpp-0.1.11-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8553bbcc8101dab95688e7356cb5024cb3bc3e8d4acf55625e21fc3386b61920 |
|
MD5 | cd70441169e488c57adebf1a37ed249a |
|
BLAKE2b-256 | e2a46815a56532232e93611edacbba9c44c5d8a6f6efd74a2508a16d88def243 |
Hashes for llamacpp-0.1.11-cp38-cp38-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 32b6b38142d4d2632733a2092cf307d8056e09de5de2ee2224a74f0f4ff0b8e8 |
|
MD5 | 2875ab942001eeed779b990a2913d664 |
|
BLAKE2b-256 | 3f8f77a975367bca5913f9016ea36b7b5d534b9f051d75702abb19fe03582cd5 |
Hashes for llamacpp-0.1.11-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f66536c977b79d3657fbf0d486e2ee82455276db06974084837b99da1dabf9cd |
|
MD5 | ae98f00a011f3c9da5e919d250eba521 |
|
BLAKE2b-256 | 2a42a098c50d348704b920408ec3c2f4507c0f980d0b876c0b2c052ac8355ad4 |
Hashes for llamacpp-0.1.11-cp38-cp38-macosx_10_9_universal2.macosx_10_9_x86_64.macosx_11_0_arm64.macosx_11_0_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b690b810fa58bf3c2344aef7ab165e1e1304dea83676b4f4b736f4c8d181d113 |
|
MD5 | 07b46eab7e3f6b9676df4601c172fbfd |
|
BLAKE2b-256 | 59c9014c5e11a80f47466c5ebcdd054c6f3f60dbaa6b7e5ceadaf591e9294bcd |
Hashes for llamacpp-0.1.11-cp37-cp37m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 73e74a562201e1d5816b0ce737f2266f8b65a1540fc3a45a6f0c51d000463d7a |
|
MD5 | 984eddea0944bffa167352304e738743 |
|
BLAKE2b-256 | f500848f51206b1ad2cb53d0e4977d7fee35a9ace564e164c164853de0eb85d8 |
Hashes for llamacpp-0.1.11-cp37-cp37m-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5845decffa8958d7fb549c7b7461647bf3aaafd2210072446dfbd838f60814c0 |
|
MD5 | 648a93ff170a50db47be2463bc3d0671 |
|
BLAKE2b-256 | 5dc7b9ca0764aeea25235b51d091d4485218c6948e188e141e22717f74bfd998 |
Hashes for llamacpp-0.1.11-cp37-cp37m-musllinux_1_1_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 739bee8b100aed7009b2a700860c9a627619cadc24928fcbe9ce14dd1a75aa1d |
|
MD5 | b7bfb146439f9cc717742cb362254394 |
|
BLAKE2b-256 | b765bb6679a64e30a484112576a61c9922daa7c2ce3eb7637b9308f43a4b26cf |
Hashes for llamacpp-0.1.11-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 76a46110782347f450a83bcde3cf052df2571d699306a1de5ec3bce35dd75521 |
|
MD5 | d5d6afdea2739d9dbeb461520187e778 |
|
BLAKE2b-256 | 889e3dd203abe74718bc356e3d1a33f51b981e2cdd1f254935029728d3180b28 |
Hashes for llamacpp-0.1.11-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ef1e514a3c5270abb1b39a74e49e23b7e0035a8a7086d1fc676ea0be153ffe72 |
|
MD5 | 0cab01f2153366d7a0e4340588ac00a4 |
|
BLAKE2b-256 | 44e247c52a0a6e5cd8f3bbaf42454fe5b7b6c635029907111565f901b3c747b7 |
Hashes for llamacpp-0.1.11-cp37-cp37m-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c0a3467f26f587fcba7c874ee480e44f3f0213313585cca469154a077e04b6a5 |
|
MD5 | 997521f49f8a89bf67c15dae2a2fc7b3 |
|
BLAKE2b-256 | a2eaea2b690d026497531b85a7056705deb9ee9f5bef46489f7fcc2ed0e3de88 |