Simple plug and play GBNF compiler for llama.cpp
Project description
GBNF Compiler (Python)
Dependency-free GBNF Compiler, plug and play for llama.cpp GBNF.
Simple yet powerful GBNF Compiler, use it like handlebars.js with better response from LLMs.
Why ?
GBNF is very useful to confine the response format from LLMs.
In most of the time, using sentence-based GBNF can produce better result than JSON-based GBNF in most LLMs without fine-tuning, gbnf-compiler
provides the flexibility to construct or parse sentence / JSON / (anything you can think of) GBNF.
Getting Started
pip install gbnf-compiler
How to Use
- Define the LLM Response Template
- Create the Rule
- Send it out, done!
import requests
from gbnf_compiler.sentence import GBNFCompiler
from gbnf_compiler.rules import *
# Define your Prompt
prompt = "What tool will you use to calculate 2^5 ?"
# Define the LLM Response Template
# Each {{}} is a variable with a rule
template = "I choose {{tool}} because {{reason}}"
# Define the Rule - "tools" for variable ("tool")
tools = multiple_choice('tool', ['calculator', 'web-search', 'web-browse'])
# Create the GBNF Compiler
# Single Sentence is a default Grammar Rule which ends with '.'
c = GBNFCompiler(template, { 'tool': tools, 'reason': SINGLE_SENTENCE })
print(c.grammar())
# Try a dummy result
text = "I choose calculator because it is the most efficient and accurate way to calculate 2^5."
result = c.parse(text)
print(result)
"""
Result:
{'tool': 'calculator', 'reason': 'it is the most efficient and accurate way to calculate 2^5.'}
"""
# Example: Send it out to local llama.cpp
def template(role: str, prompt: str):
return """[INST] <<SYS>>
{role}
<</SYS>>
{prompt}
[/INST]""".format(role=role, prompt=prompt)
data_json = {
"prompt": template("", prompt), "temperature": 0.0,
"n_predict": 512, "top_p": 0.2, "top_k": 10,
"stream": False, "grammar": c.grammar() }
resp = requests.post(
url="http://127.0.0.1:9999/completion",
headers={"Content-Type": "application/json"},
json=data_json,
)
result = resp.json()["content"]
print(result)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file gbnf_compiler-0.1.1.tar.gz
.
File metadata
- Download URL: gbnf_compiler-0.1.1.tar.gz
- Upload date:
- Size: 4.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.11.5
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | ac0c190cf271c9ab7c135395b3142a3e718859060969487ab7996770e8f08791 |
|
MD5 | 34ce993f19b5b868eaad4a5ea2c97881 |
|
BLAKE2b-256 | 2a311d9e8563da692f645c256e389aabca190f5d5a7c9add62b6962416c77473 |
File details
Details for the file gbnf_compiler-0.1.1-py3-none-any.whl
.
File metadata
- Download URL: gbnf_compiler-0.1.1-py3-none-any.whl
- Upload date:
- Size: 5.4 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.11.5
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1068bdb411c091d0778ba1297f212ba6716844a973ce200d4d9ea97dd89550f4 |
|
MD5 | f9466562760c764cefc3e31308428a59 |
|
BLAKE2b-256 | 498ebace191cc0ef6ea58ab4db2d1835750306dc60f20e732eb0a575c09ec9a1 |