Format LLM language by using LALR(1) grammar. Supports JSON, XML, etc.
Project description
LLM Format
Introduction
LLM Format is a Large-language model formatter that constrain the outputs of language model to follow certain rules. Our design is to make this tool flexible and efficient to adapt to different use cases.
We currently supports vllm and any LALR(1) grammar including JSON format.
Different from other packages including lm-format-enforcer, jsonformer and guidance. This package ensures the generated text from LLM to be sound and complete.
Tutorial
Installation
pip install llmformat
Usage
To enforce a new type of grammar, a grammar file written in EBNF is needed. We provide the JSON example as here.
Once it is written, we can only need one-line code change to enforce the generation.
In vllm, add this option to sampling_param.
sampling_param.logits_processors=[llmformat.llminterface.build_vllm_logits_processor(model, "/root/llmformat/llmformat/json_min.bnf")]
Example
The example of working on Llama2 and vllm can be found here. Note that you may want to change the location of grammar file.
- Add support for customized JSON
- Add more integrations
Known Issues
We use cache to accelerate grammar parsing. The speed will becomes faster as it runns longer.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for llmformat-0.0.2.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f505715c91c10faf9eec8bbed2c7f6c1890984b945d645fce40fb94c31b5ccea |
|
MD5 | a471ef888bf73e57726ee2d9570c70aa |
|
BLAKE2b-256 | faf2dff80cf4179317fb3dc92917f93fd850caeb3a577f482029f44f25e2f597 |