Skip to main content

Simple plug and play GBNF compiler for llama.cpp

Project description

GBNF Compiler (Python)

Dependency-free GBNF Compiler, plug and play for llama.cpp GBNF.

Simple yet powerful GBNF Compiler, use it like handlebars.js with better response from LLMs.

Why ?

GBNF is very useful to confine the response format from LLMs.

In most of the time, using sentence-based GBNF can produce better result than JSON-based GBNF in most LLMs without fine-tuning, gbnf-compiler provides the flexibility to construct or parse sentence / JSON / (anything you can think of) GBNF.

Getting Started

pip install gbnf-compiler

How to Use

  1. Define the LLM Response Template
  2. Create the Rule
  3. Send it out, done!
import requests
from gbnf_compiler.sentence import GBNFCompiler
from gbnf_compiler.rules import *

# Define your Prompt
prompt = "What tool will you use to calculate 2^5 ?"

# Define the LLM Response Template
# Each {{}} is a variable with a rule
template = "I choose {{tool}} because {{reason}}"

# Define the Rule - "tools" for variable ("tool")
tools = multiple_choice('tool', ['calculator', 'web-search', 'web-browse'])

# Create the GBNF Compiler
# Single Sentence is a default Grammar Rule which ends with '.'
c = GBNFCompiler(template, { 'tool': tools, 'reason': SINGLE_SENTENCE })
print(c.grammar())

# Try a dummy result
text = "I choose calculator because it is the most efficient and accurate way to calculate 2^5."
result = c.parse(text)
print(result)

"""
Result: 
{'tool': 'calculator', 'reason': 'it is the most efficient and accurate way to calculate 2^5.'}
"""

# Example: Send it out to local llama.cpp
def template(role: str, prompt: str):
    return """[INST] <<SYS>>
{role}
<</SYS>>
{prompt}
[/INST]""".format(role=role, prompt=prompt)

data_json = {
    "prompt": template("", prompt), "temperature": 0.0,
    "n_predict": 512, "top_p": 0.2, "top_k": 10,
    "stream": False, "grammar": c.grammar() }

resp = requests.post(
    url="http://127.0.0.1:9999/completion",
    headers={"Content-Type": "application/json"},
    json=data_json,
)
result = resp.json()["content"]
print(result)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

gbnf_compiler-0.1.1.tar.gz (4.7 kB view details)

Uploaded Source

Built Distribution

gbnf_compiler-0.1.1-py3-none-any.whl (5.4 kB view details)

Uploaded Python 3

File details

Details for the file gbnf_compiler-0.1.1.tar.gz.

File metadata

  • Download URL: gbnf_compiler-0.1.1.tar.gz
  • Upload date:
  • Size: 4.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.5

File hashes

Hashes for gbnf_compiler-0.1.1.tar.gz
Algorithm Hash digest
SHA256 ac0c190cf271c9ab7c135395b3142a3e718859060969487ab7996770e8f08791
MD5 34ce993f19b5b868eaad4a5ea2c97881
BLAKE2b-256 2a311d9e8563da692f645c256e389aabca190f5d5a7c9add62b6962416c77473

See more details on using hashes here.

File details

Details for the file gbnf_compiler-0.1.1-py3-none-any.whl.

File metadata

File hashes

Hashes for gbnf_compiler-0.1.1-py3-none-any.whl
Algorithm Hash digest
SHA256 1068bdb411c091d0778ba1297f212ba6716844a973ce200d4d9ea97dd89550f4
MD5 f9466562760c764cefc3e31308428a59
BLAKE2b-256 498ebace191cc0ef6ea58ab4db2d1835750306dc60f20e732eb0a575c09ec9a1

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page