Formwork

Runtime semantic constraints and targeted repair for structured LLM output.

Formwork is a Python library for building more reliable structured-output workflows around LLMs.

It lets you declare, in a single schema, which fields are:

• generated by the model
• computed deterministically by your application
• validated by runtime-dependent semantic rules
• repairable without another model call

When an LLM produces structurally valid but semantically invalid data, Formwork does not blindly accept it or necessarily regenerate the entire object.

Instead, it:

1. validates the generated structure,
2. assembles the complete object,
3. evaluates runtime semantic rules,
4. applies deterministic repairs when possible,
5. and falls back to targeted model repair when necessary.

Let the model generate what requires reasoning. Let deterministic code own what must be correct.

---

Why Formwork?

Structured output does not necessarily mean correct output.

An LLM can produce perfectly valid JSON that satisfies a schema while still violating application-specific rules.

For example:

Schema:

    calories: integer
    protein: integer
    allergens: list[str]

Runtime constraints:

    calories must stay within the user's target range
    protein must satisfy the user's minimum
    forbidden allergens must never appear

A JSON Schema or grammar can enforce the structural part.

It cannot, by itself, know all of the runtime context required to evaluate the semantic part.

Formwork separates these responsibilities:

                    Spec + Runtime Context
                              │
                              ▼
                 Resolve deterministic fields
                              │
                              ▼
                    Model-facing schema
                     computed fields removed
                              │
                              ▼
                             LLM
                              │
                              ▼
                    Structural validation
                              │
                              ▼
                       Assemble object
                              │
                              ▼
                    Semantic rule checking
                              │
                ┌─────────────┴─────────────┐
                │                           │
             valid                        invalid
                │                           │
                ▼                           ▼
             Result                  Deterministic repair
                                            │
                                     still invalid?
                                            │
                                            ▼
                                      Targeted repair
                                            │
                                            ▼
                                      Result / Error

The key distinction is:

Structured generation and semantic correctness are different problems.

Formwork is designed for the layer between them.

---

Core idea

Formwork uses a single Spec definition while maintaining two views of that specification:

                     Spec
                      │
          ┌───────────┴───────────┐
          │                       │
      Full object           Model-facing schema
          │                       │
    computed fields          generated fields
    generated fields         chosen fields
    chosen fields
    runtime rules

The model-facing schema intentionally excludes computed fields.

This gives Formwork a simple invariant:

A field the model cannot see is a field the model cannot incorrectly generate.

Field roles are declared through:

computed()
chosen()
generated()

Fields without an explicit role are treated as generated.

---

A complete example

from dataclasses import dataclass
from typing import Annotated, Any

from pydantic import BaseModel, Field

from formwork import Spec, chosen, computed, generate, generated, repair, rule, soft


@dataclass
class Profile:
    kcal_target: int
    protein_min: int
    catalogue: list[dict[str, Any]]

    @property
    def by_id(self) -> dict[str, dict[str, Any]]:
        return {f["id"]: f for f in self.catalogue}


class FoodPick(BaseModel):
    id: str
    grams: Annotated[int, Field(ge=10, le=1000)]


class DayPlan(Spec):
    """Plan one day of eating."""

    # Your application owns these. They never enter the model-facing schema;
    # they are stated to the model as facts to work around.
    kcal_target: Annotated[int, computed(lambda c: c.kcal_target, describe="target calories")]
    protein_min: Annotated[int, computed(lambda c: c.protein_min, describe="minimum protein")]

    # The model picks, but only from the catalogue supplied at runtime.
    foods: Annotated[list[FoodPick], chosen(source="catalogue", key="id")]

    # The model is free.
    note: Annotated[str, generated(describe="One sentence for the user.")]

    def kcal(self, ctx: Profile) -> float:
        return sum(ctx.by_id[f.id]["kcal"] * f.grams / 100 for f in self.foods if f.id in ctx.by_id)

    # An invented id is fixable without asking the model again.
    @rule(
        "Every food id must come from the catalogue.",
        fields=["foods"],
        repair=repair.drop_invalid("foods"),
    )
    def known_foods(self, ctx: Profile) -> list[str]:
        return [f"{f.id!r} is not in the catalogue" for f in self.foods if f.id not in ctx.by_id]

    # This one is a judgement call, so it goes back to the model —
    # but only the `foods` field goes back.
    @rule("Calories must land within 5% of the target.", fields=["foods"])
    def calories(self, ctx: Profile) -> str | None:
        total = self.kcal(ctx)
        if abs(total - ctx.kcal_target) > ctx.kcal_target * 0.05:
            return f"{total:.0f} kcal, target {ctx.kcal_target} +/-5%"
        return None

    @soft(weight=1.0)
    def few_items(self, ctx: Profile) -> float:
        """Shorter shopping lists are preferable, but never a hard failure."""
        return float(max(0, len(self.foods) - 5))


plan, report = generate(DayPlan, ctx, model)

plan satisfies every rule, or a ConstraintError was raised instead.

Given a first response that invents quinoa and misses the calorie target, the run above resolves like this:

model-facing schema : ['foods', 'note']      kcal_target is absent by construction

attempt 0  initial  asked=('foods', 'note')  repaired=('known_foods',)
attempt 1  repair   asked=('foods',)         repaired=-

report.summary()    '2 model call(s); 300 tokens; repaired by known_foods'
plan.note           'High protein day.'      kept from call 1, never regenerated
report.soft_scores  {'few_items': 0.0}

The invented id was dropped locally without a model call. The calorie violation needed the model, so only foods was reopened — note was frozen and survived.

A runnable version of this pattern is in examples/workout.py.

---

Features

1. Explicit field ownership

Not every value should be generated by an LLM.

Some values are better calculated by deterministic application code.

Formwork makes this distinction explicit:

generated
    ↓
the model owns the value

computed
    ↓
deterministic application logic owns the value

chosen
    ↓
the model selects, but only from a closed set
supplied by the runtime context

chosen fields are enforced today by validating the selection against the context and repairing it. Enforcing the closed set at decode time, through a grammar backend, is on the roadmap.

The ownership decision lives in the schema rather than being hidden inside prompt instructions.

---

2. Separate model-facing schema

The complete Spec describes the final object.

The model-facing schema describes only what the model is allowed to generate.

Conceptually:

Full object

{
    user_id
    target_calories
    meal
    protein
}

        │
        │ computed fields removed
        ▼

Model-facing object

{
    meal
    protein
}

This prevents deterministic application state from becoming another value that the LLM has to invent.

Formwork dynamically derives the model-facing schema and preserves relevant Pydantic constraints such as:

Field(ge=1)

These constraints remain useful as inexpensive structural safeguards.

---

Runtime semantic rules

Structural validation answers questions such as:

Is this an integer?
Is this field present?
Is this value allowed by the schema?

Semantic rules answer different questions:

Does this value satisfy the current user's constraints?
Is this combination of fields allowed?
Does this object respect the current runtime context?

Formwork evaluates these rules after the generated data has been assembled into the complete object.

A rule can report one or more violations.

Those violations can then drive deterministic or targeted repair.

---

Two-stage repair

Formwork deliberately prefers deterministic repair over another model call whenever possible.

Stage 1 — deterministic repair

If a rule declares a repair strategy:

LLM output
    ↓
rule violation
    ↓
deterministic repair
    ↓
re-check

No additional model call is required.

This is useful when the correct repair is mechanically knowable.

For example, if an application can safely normalize or recompute a value, there is little reason to ask an LLM to do it again.

---

Stage 2 — targeted model repair

If deterministic repair cannot resolve the violation, Formwork identifies the fields associated with the failed rule.

Instead of regenerating the entire object:

full object
    ↓
regenerate everything

Formwork can narrow the repair request:

field A → frozen
field B → frozen
field C → regenerate
field D → frozen

Only the fields involved in the violated constraints are reopened.

Rules should declare their affected fields using:

fields=[...]

If a rule does not declare affected fields, Formwork falls back to full regeneration rather than silently ignoring the violation.

---

Execution model

Formwork's core workflow is implemented by Session.

Session is intentionally sans-IO.

It produces requests through next_request() and receives model responses through feed().

Conceptually:

session = Session(DayPlan, ctx)

while (request := session.next_request()) is not None:
    raw, usage = however_you_like(request)
    session.feed(raw, usage)

plan, report = session.finish()

next_request() returns None when the run is settled or out of attempts; finish() then returns the object or raises. Anyone with an unusual setup — a queue, a batch API, a human in the loop — can drive the loop this way.

The higher-level:

generate()
agenerate()

functions drive this state machine for synchronous and asynchronous execution.

The important design principle is:

The retry and repair state machine belongs to Session, not to individual provider drivers.

This keeps synchronous and asynchronous execution paths consistent.

---

Validation model

Formwork distinguishes between two major types of failure.

Structural failure

The model output does not satisfy the model-facing schema.

For example:

{
  "age": "twenty-two"
}

when the schema requires:

age: integer

This results in a structural retry.

---

Semantic failure

The output is structurally valid but violates an application-specific rule.

For example:

{
  "age": 22,
  "calories": 5000
}

The JSON is valid.

The types may be valid.

But the application may require the calorie value to stay within a runtime-defined range.

Formwork therefore performs:

schema validation
        ↓
assemble object
        ↓
semantic rule validation
        ↓
repair if necessary

---

Example

Imagine an application asking an LLM to generate a structured nutrition plan.

The application already knows the user's calorie target.

The model should generate the meals and protein distribution.

A conceptual specification could therefore be:

NutritionPlan
│
├── target_calories   → computed
├── meals              → generated
└── protein            → generated

The model-facing schema becomes:

{
    meals
    protein
}

rather than asking the model to generate:

target_calories

After generation:

computed values
       +
model output
       ↓
complete object
       ↓
semantic rules

A rule might detect:

protein < required minimum

If that violation has a deterministic repair, Formwork applies it locally.

Otherwise, the relevant generated fields are reopened and the model is asked to repair only those fields.

---

Rules and violations

A rule can return:

None
True
False
str
Violation
an iterable of violations

Formwork normalizes these results into Violation objects.

This gives rule implementations a lightweight interface while maintaining one consistent internal representation.

Violation messages are important because they may be included in a targeted repair request.

Prefer concrete messages such as:

"ex_42 is not available in the library"

over vague messages such as:

"invalid value"

The model needs enough information to understand what must change.

---

Reliability invariant

Formwork is designed around a strong output invariant:

generate() never returns an object that still violates the declared semantic constraints.

The public result is therefore:

valid object
    OR
ConstraintError

rather than:

possibly-valid object

This invariant is also tested adversarially across 200 seeds in the test suite.

---

Why not just use JSON Schema or grammar constraints?

Grammar-based structured generation is extremely useful.

Libraries such as Outlines and XGrammar primarily address the problem of constraining the structure of generated output.

Formwork addresses a different layer:

                Structured generation
                         │
                "Is the structure valid?"
                         │
                         ▼
                     Formwork
                         │
                "Is this object valid
                 in the current runtime
                 context?"

These approaches are complementary.

Formwork is not intended to replace grammar-based structured generation.

It provides a runtime semantic validation and repair layer around structured generation.

---

Why not just retry the LLM?

A naive retry loop usually looks like:

generate
   ↓
invalid
   ↓
generate everything again
   ↓
invalid
   ↓
generate everything again

This can:

1. waste model calls and tokens,
2. regenerate fields that were already correct,
3. make application-level guarantees dependent on repeated probabilistic generation.

Formwork instead uses:

generate
   ↓
validate
   ↓
deterministic repair when possible
   ↓
targeted repair when necessary

The goal is not to make the LLM perfect.

The goal is to make the system around the LLM reliable.

---

Benchmark

Formwork includes an initial benchmark comparing several execution strategies:

single          one attempt, model restates the constraints
own-only        one attempt, field ownership on
naive-retry     regenerate everything on failure, model restates constraints
own-retry       regenerate everything on failure, field ownership on
own-targeted    reopen only the implicated fields
formwork        the above, plus deterministic repairs

The arms form a chain in which each consecutive pair differs by exactly one mechanism, so a difference can be attributed rather than merely observed. This is enforced by tests/test_bench_fairness.py, not by good intentions.

The benchmark used:

Model:            gemini-3.5-flash-lite
Temperature:      0.7
Maximum attempts: 3

Tasks:            nutrition, shifts, workout
Difficulties:     easy, medium, hard
Total runs:       54
Total tokens:     30,237

Each benchmark cell currently contains a single run, so every rate below is a count out of nine cells.

Therefore, these results should be treated as pilot/directional results rather than statistically conclusive evidence.

Reproduce with:

.venv/bin/python -m bench.run --runs 1 --model gemini-3.5-flash-lite

Overall result

Strategy	Final success	valid@1	Avg. tokens	Avg. calls
single	67%	67%	482	1.00
own-only	67%	67%	427	1.00
naive-retry	78%	67%	686	1.56
own-retry	89%	78%	576	1.44
own-targeted	78%	78%	600	1.44
formwork	100%	67%	589	1.44

Each percentage is 6, 7, 8 or 9 successes out of 9 cells. At that sample size a 95% Wilson interval on 67% spans roughly 35–88%, so the intervals of every arm in this table overlap heavily.

valid@1 is not comparable across arms. A deterministic repair counts against it by design, and deterministic repairs are enabled only in the formwork arm. Its lower valid@1 reflects repairs firing, not worse model output. Compare that column only among the five arms above it.

The most relevant baseline comparison is naive retry:

naive-retry → formwork

Final success:
78% → 100%

Tokens:
686 → 589

In this pilot benchmark, Formwork achieved a 22 percentage-point increase in final success while using approximately 14% fewer tokens than naive retry.

This is an end-to-end comparison, not an attribution: the two arms differ in every mechanism at once. The per-mechanism breakdown is below, and it does not credit all four mechanisms equally.

These numbers are promising but should not be interpreted as general performance guarantees.

---

Mechanism ablations

Each ablation below isolates one mechanism by comparing two arms that differ in that mechanism alone. Any pair differing in two mechanisms cannot attribute a result to either one.

Field ownership, one attempt

single → own-only

Success:  67% → 67%
Tokens:   482 → 427

Field ownership alone did not change final success here, but reduced tokens by about 11% — the computed fields no longer have to be generated.

Field ownership, with retry

naive-retry → own-retry

Success:  78% → 89%
Tokens:   686 → 576

With a retry available, ownership improved success and cut tokens.

Targeted repair

own-retry → own-targeted

Success:  89% → 78%
Tokens:   576 → 600

Targeted repair did not help in this pilot; it scored one cell worse and cost slightly more. With nine cells per arm, a single cell is 11 percentage points, so this difference is well inside the noise of a one-run-per-cell experiment. It is reported as measured rather than omitted. Whether reopening only the implicated fields pays for itself is exactly the question a larger benchmark needs to answer.

Deterministic repair

own-targeted → formwork

Success:  78% → 100%
Tokens:   600 → 589

Enabling deterministic repairs raised final success while slightly reducing tokens, which is the expected shape: a violation fixed locally costs no model call at all.

---

Constraint-heavy nutrition task

The largest difference appeared in the nutrition task.

On the hard nutrition case:

single         0%
own-only       0%
naive-retry    0%
own-retry      0%
own-targeted   0%
formwork       100%

Formwork reached the valid final result in that run using three model calls and 973 tokens.

However, this result comes from a single run and should not be interpreted as a general 100% success guarantee.

---

Benchmark status

The current benchmark should be considered a pilot benchmark.

Future evaluations should increase the number of independent runs and measure:

• success rate
• first-pass validity
• token usage
• model-call count
• latency
• deterministic repair rate
• targeted repair rate
• failure categories
• confidence intervals
• performance across multiple models
• performance across multiple providers

Performance claims will be expanded as the benchmark becomes more rigorous.

---

Testing

Run the complete test suite with:

.venv/bin/python -m pytest -q

Useful commands:

.venv/bin/python -m pytest tests/test_engine.py -q

.venv/bin/python -m pytest -q -k repair

.venv/bin/ruff check src tests examples bench scripts

.venv/bin/mypy

The suite is offline and takes about a second. Tests that call a real provider are marked live and excluded by default, so filling in .env for the benchmark will not quietly start spending quota on every test run. Opt in deliberately:

.venv/bin/python -m pytest -m live -q     # costs money, needs GEMINI_API_KEY

The project CI targets:

Python 3.11
Python 3.12
Python 3.13

Changes should keep the test suite, linting, type checking, and example workflow clean.

---

Testing your own specifications

Formwork includes fake providers for testing user-defined specifications without depending on a live model.

Available fake-provider strategies include:

Chaos
Recording
Scripted

These are part of the public API rather than merely internal test utilities.

They allow users to exercise validation and repair behavior deterministically.

Chaos is the adversarial one: it returns plausible-but-wrong variations of a baseline response — unknown ids, counts over the limit, numbers out of range, duplicated entries. That is the failure mode that actually bites in production.

---

Providers

One adapter ships today:

pip install "formwork[gemini]"

from formwork.providers.gemini import Gemini

plan, report = generate(DayPlan, ctx, Gemini())   # reads GEMINI_API_KEY

It converts each schema into the OpenAPI subset Gemini's response_schema accepts — inlining $refs and dropping additionalProperties, both of which are rejected otherwise. Anything it drops is still enforced by the engine when the response comes back.

For anything else, the provider interface is a single method:

class MyModel:
    def generate_structured(self, request) -> tuple[dict, Usage]:
        response = client.responses.parse(
            model="...",
            instructions=request.system,
            input=request.prompt,
            text_format=request.schema,        # a Pydantic model
        )
        return response.output_parsed.model_dump(), Usage(...)

Providers transport requests and responses. They do not own retry or repair logic, so a new adapter cannot change the guarantees.

---

Reporting

Every run returns a Report alongside the object:

plan, report = generate(DayPlan, ctx, model)

report.summary()               # '2 model call(s); 300 tokens; repaired by known_foods'
report.model_calls             # 2
report.usage.total_tokens      # 300
report.valid_first_try         # False
report.deterministic_repairs   # ('known_foods',)
report.attempts[1].targeted_fields   # ('foods',)
report.soft_scores             # {'few_items': 0.0}

The point is auditability. "The model said so" is not an answer in a regulated domain; "rule known_foods was checked, failed once, and was repaired deterministically" is.

Note that valid_first_try counts a deterministic repair as a failure. The repair rescued the run, but the output that arrived was still wrong, and the metric would be misleading otherwise.

---

Soft constraints

Not every preference is a pass/fail gate. @soft declares a score to minimise, and candidates=N runs the loop N times and keeps the rule-valid result with the best score:

@soft(weight=1.0)
def few_items(self, ctx) -> float:
    return float(max(0, len(self.foods) - 5))

plan, report = generate(DayPlan, ctx, model, candidates=3)

A poor soft score never fails a run — that is what makes it soft. Candidates multiply cost, so the default is 1.

---

How this compares

	what it constrains
Outlines / XGrammar / llguidance	output format, at decode time. Composes with Formwork rather than competing — grammar for shape, Formwork for meaning.
Instructor / PydanticAI	schema validation plus a retry. The closest neighbours. The differences: no notion of fields your code owns, validation is pass/fail with no soft scores, and the retry regenerates everything instead of the fields that broke.
OR-Tools / python-constraint	can express the constraints, but cannot write the prose or make the judgement call you reached for a model to make.
Formwork	semantic constraints over runtime context, with ownership declared per field and repair targeted at the failure.

---

Installation

Install Formwork from PyPI:

pip install formwork

With the Gemini adapter:

pip install "formwork[gemini]"

For local development:

python -m venv .venv
.venv/bin/pip install -e ".[dev]"

The only runtime dependency is Pydantic 2. Formwork is developed and tested against Python 3.11, 3.12 and 3.13.

---

Design principles

Formwork is built around a small number of principles.

1. Do not ask the model to own deterministic state

If application code can calculate a value reliably, keep it out of model generation.

2. Validate semantics after assembly

Structural validity is not semantic validity.

3. Prefer deterministic repair

If code can safely repair an error, do not spend another model call.

4. Repair the smallest possible surface

If only one field violates a rule, avoid regenerating unrelated fields.

5. Keep the state machine independent from IO

Providers should transport requests and responses.

Session owns the workflow.

6. Never return a known-invalid object

The public generation path should return a valid object or raise an error.

---

Current scope

Formwork currently focuses on:

• structured LLM output
• field ownership
• deterministic computed fields
• runtime semantic rules
• structural validation
• deterministic repair
• targeted model repair
• synchronous and asynchronous generation flows
• provider-independent session orchestration
• deterministic fake providers for testing

The project is intentionally small and focused.

---

Roadmap

Shipped:

• field roles, semantic rules, deterministic and targeted repair
• sans-IO Session, synchronous and asynchronous drivers
• adversarial test doubles
• a Gemini adapter, verified against the live API — including that the provider accepts the narrowed schema a targeted repair generates on the fly

Planned:

• further provider adapters (Anthropic, OpenAI, Ollama)
• grammar-backed chosen fields
• larger and more rigorous benchmarks
• broader provider and model evaluation
• additional repair strategies
• improved validation and repair observability
• expanded performance evaluation

---

Project philosophy

LLMs are probabilistic systems.

Business rules are not.

A reliable LLM application should therefore avoid putting every decision into a prompt and hoping that the model gets it right.

Instead:

                 LLM
                  │
          generate what
          requires reasoning
                  │
                  ▼
        ┌───────────────────┐
        │      Formwork     │
        │                   │
        │ schema validation │
        │ semantic rules    │
        │ deterministic fix │
        │ targeted repair   │
        └───────────────────┘
                  │
                  ▼
          valid structured
              object

Formwork's goal is not to make LLMs deterministic.

Its goal is to make the boundary between probabilistic generation and deterministic application logic explicit, testable, and repairable.

---

Status

Formwork is an early-stage library under active development.

The current benchmark provides promising initial evidence for the repair architecture, particularly on constraint-heavy tasks.

However, the current experiment size is too small to support broad statistical performance claims.

If you need a mature, broad validation ecosystem today, established libraries may be a better fit.

If you need a focused runtime built specifically around:

• field ownership
• semantic constraints
• deterministic repair
• targeted regeneration

Formwork is designed for that problem.

---

License

MIT. See the LICENSE file for the full text.