lextrail 0.1.0

A Python library for constraining language model outputs to follow CFG, REGEX and JSON (experimental).

A Python library for constraining language model outputs to follow CFG, REGEX and JSON (experimental).

Features

• Zero dependencies
• Parses all context-free grammars, including ambiguous grammars
• Returns tokens constrained to a specified vocabulary if needed
• Type annotations with mypy
• Includes a Rust implementation

Quick Start

Installation

pip install lextrail

Usage Modes

The library supports two ways to generate constrained text, depending on your use case:

Trail

Use a Trail object when you want to generate the complete next element without vocabulary constraints.

CFG

from lextrail.guide import trail_cfg

example = r"""
    start: expression

    expression: term (("+" | "-") term)

    term: factor (("*" | "/") factor)

    factor: NUMBER

    NUMBER: /-?[0-9]+/
"""

trail = trail_cfg(example)

Regex

from lextrail.guide import trail_rex

example = r"[a-z]+@[a-z]+\.(com|org|net)"

trail = trail_rex(example)

You can also combine both TERMINAL and REGEX expressions using trail_exp.

from lextrail.guide import trail_exp

example = r"/[0-9]\.[0-9]/ "+" /[0-9]\.[0-9]/"

trail = trail_exp(example)

JSON

This is an experimental version. Not intended for production use.

• Currently supported keywords: type, enum, const, properties, required, items, prefixItems, oneOf
• Constraint intersection (e.g., combining prefixItems with items, or const with enum) is not yet implemented

from lextrail.json import trail_json

example = r"""
    {
        "type": "object",
        "properties": {
            "user": {
                "type": "object",
                "properties": {
                    "name": {"type": "string"},
                    "email": {"type": "string"}
                },
                "required": ["email"]
            }
        }
    }
"""

trail = trail_json(example)

Then, run a random simulation.

import random

from lextrail.guide import get_next_values

response, value = [], ""

while values := get_next_values(trail, value):
    value = random.choice(values)
    response.append(value)

print("".join(response))

ASM

Use an ASM object when you need to constrain the next token to a predefined vocabulary.

Example

from lextrail.assemble import asm_cfg

example = r"""
    start: L0

    L0: ("A" | "B")+ L1

    L1: ("C" | "D") L2

    L2: "E" L3*

    L3: /FGH/
"""

asm = asm_cfg(example, ["AD", "EF", "GH"])

If you launch a simulation, then the proposals will be elements of the provided vocabulary.

import random

from lextrail.assemble import get_next_tokens

response, value = [], ""

while values := get_next_tokens(asm, value):
    value = random.choice(values)
    response.append(value)

print("".join(response))

assert response == ["AD", "EF", "GH", ""]

You can do it with any of the formats.

# CFG
from lextrail.assemble import asm_cfg

asm_cfg(.., [..])

# REGEX
from lextrail.assemble import asm_rex

asm_rex(.., [..])

# MIXED
from lextrail.assemble import asm_exp

asm_exp(.., [..])

# JSON
from lextrail.json import asm_json

asm_json(.., [..])

Playground

I've built a playground to showcase the different simulations, you can use either a Trail object or an ASM one.

from lextrail.guide import trail_cfg
from lextrail.playground import run_playground

example = r"""
    start:  expression

    expression: term* (( "+" | "-") term)+

    term: factor* (("*" | "/") factor)+

    factor: NUMBER?

    NUMBER: /[0-1]+/
"""

trail = trail_cfg(example)

run_playground(trail)