text2model 1.1.0

LLM Modeling Copilots for Text-to-Model Translation

---

Text Mode • Text2Zinc Mode • Interactive Mode • Copilots • Installation • Dataset Editor • Evaluation • Leaderboard

---

Text2Model: LLM Modeling Copilots for Text-to-Model Translation

Text2Model is a suite of LLM modeling copilots, datasets, fined-tuned models, demos, interactive editor, and online leaderboard for translating natural language text into formal combinatorial constraint models.

Text2Model uses MiniZinc as the target modeling language which makes our copilots both paradigm- and solver-agnostic. Our copilots generate models that can be solved by any MiniZinc compatible solver including Gecode, Chuffed, OR-Tools, CBC, Gurobi, Cplex, HiGH. This covers a wide range of paradigms including CP, CP-SAT, and MIP. As such, Text2Model can address both combinatorial satisfaction and optimization problems.

Please visit Text2Model for latest publications and resources.

Quick Start

Text2Model supports translating given problem descriptions (Text mode), specific problems from our dataset (Text2Zinc mode), or running interactively using the editor (Editor mode).

Text Mode

# Translate a given problem description
text2model --problem "A country produces fighter jets each year. Some of these jets must be set aside for pilot training instead of combat use. Year 1 production is 10 jets, and Year 2 production is 15 jets. Each training jet can train 5 pilots per year. Training runs for 2 years, starting in Year 1. Determine how many pilots will be trained in total by the end of Year 2."

# Translate a given problem file
text2model --problem my_problem.txt

# Choose the copilot strategy and the model
# Note that `knowledge_graph` is not available for text-mode as it requires pre-built TTL files
text2model --problem my_problem.txt --strategies agents_with_code_validation --model gpt-4o

# Redirect the output to a MiniZinc model
text2model --problem my_problem.txt > my_model.mzn

Text2Zinc Mode

# Translate specific problems from Text2Zinc
text2model --problem-ids 0 1 2 --strategies cot --model gpt-4 --output-dir my_results

# Run multiple strategies on all Text2Zinc problems
text2model --strategies cot --model gpt-4 --output-dir my_results

# Run all strategies
text2model --strategies all --model gpt-4 --output-dir my_results

# Run on specific source of Text2Zinc problems
text2model --source nlp4lp --strategies cot --model gpt-4 --output-dir my_results

# List all available data sources
text2model --list-sources

# List all available --model options (which need OPENAI_API_KEY vs. run locally through Ollama)
text2model --list-models

# Advanced options
text2model --strategies agents --model gpt-4 \
  --output-dir my_results \
  --temperature 0.7 \
  --max-tokens 8192 \
  --sleep-time 2 \
  --include-unverified

# Use a local dataset (e.g. one saved by `text2model --editor`) instead of the
# default skadio/text2zinc HuggingFace dataset
text2model --strategies cot --model gpt-4 --output-dir my_results \
  --dataset-path text2zinc_edited.csv

Interactive Mode

text2model --editor
text2model --editor --dataset-path text2zinc_edited.csv

See Dataset Editor for details.

Copilots

Text2Model offers different copilot strategies, ranging from simple single-call approaches to sophisticated multi-agent systems. Each makes different trade-offs between speed and accuracy.

Strategy	Description
baseline	Direct code generation from problem description. No special prompting. Good for simple problems or establishing a baseline.
cot	Chain-of-Thought prompting with guiding principles. The LLM reasons through the problem step-by-step before generating code.
knowledge_graph	First extracts structured information (entities, relationships) from the problem, then generates code from this intermediate representation.
cot_with_code_validation	Generates code with CoT, then validates and fixes any compilation errors. Good default choice.
cot_with_grammar_validation	Generates code with CoT, then checks against MiniZinc grammar rules.
cot_with_code_and_grammar_validation	Combines CoT generation with both grammar checking and code validation.
agents	Decomposes the task into specialized agents: (1) parameters & variables, (2) constraints, (3) objective, (4) assembler that stitches everything together.
agents_with_code_validation	Agents approach plus a final validation/fix step.
gala	Global Agents for different constraint types (all_different, cumulative, etc.) plus an assembler. See the GALA paper.

Want to use knowledge_graph on your own problems? It requires a pre-built .ttl knowledge-graph file per problem. See text2model/generate_knowledge_graph.py and any example file under text2model/knowledge_graphs/ (e.g. nlp4lp_1.ttl) and do similar — either adapt the script to your problem or hand-write a .ttl following the same structure.

Adding a Custom Copilot

Every strategy in the table above follows the same shape, so plugging in a new one is mechanical:

1. Add prompt(s) under text2model/prompts/ (.txt files with {problem_description}, {input_data}, etc. placeholders — see any existing prompt for the pattern).
2. Write a run_<name>_strategy(client, model, problem, problem_identifier, output_dir) function in text2model/main.py. Use utils.prepare_problem_data, utils.get_effective_input_data, and utils.load_file to build your prompt(s), call utils.call_api(client, model, prompt) to get code back, and stitch/combine calls the way run_agents_strategy does if your copilot needs multiple LLM calls. Finish with utils.save_solution(output_dir, problem_identifier, code).
3. Register it by adding '<name>': run_<name>_strategy to _STRATEGY_MAP in main.py, and add '<name>' to the --strategies argparse choices list (and to the 'all' expansion list if it should run as part of --strategies all).

That's it — your strategy is now available via --strategies <name> in both --problem and batch modes.

Installation

Text2Model requires Python 3.8+ and can be installed from PyPI or by building from source.

Set Your API Key

export OPENAI_API_KEY="your-api-key-here"

Install from PyPI

pip install text2model

Install from Source

git clone https://github.com/skadio/text2model.git
cd text2model
pip install -e .

Dataset Editor

Text2Model ships a GUI editor (built with Flet) for browsing and editing Text2Zinc problems (input.json, data.dzn, model.mzn, output.json), running them through MiniZinc, and chatting with an AI assistant for help rephrasing descriptions or fixing model code.

Launch

text2model --editor

By default the editor opens, in order: a previous editing session (text2zinc_edited.csv in the current directory), otherwise the dataset bundled with the package. Use the "Load from HuggingFace" button inside the editor to instead start from a fresh copy of skadio/text2zinc, or open any other local dataset with "Open CSV..." — or non-interactively:

text2model --editor --dataset-path my_dataset.csv

Save and Benchmark

• Save (or Ctrl+S) quick-saves your edits to text2zinc_edited.csv in the current directory.
• Save As New Dataset... exports to any path you choose. That path is a complete Text2Zinc dataset you can pass to --dataset-path to generate or benchmark against your edits instead of the default HuggingFace dataset:

text2model --strategies cot --model gpt-4 --output-dir my_results --dataset-path my_dataset.csv
python evals/evaluate.py --output-dir my_results --dataset-path my_dataset.csv

Evaluation

After generating models, evaluate their correctness via evals/evaluate.py. This script compiles and runs each generated MiniZinc model against test instances, checking for both execution success and solution correctness.

Prerequisite

Install MiniZinc solver: https://www.minizinc.org/doc-2.5.5/en/installation.html

Evals

# Evaluate all generated code
# `--output-dir` is required. Point it at the directory produced by the batch-mode `text2model` run.
python evals/evaluate.py --output-dir my_results

# Evaluate against a local dataset (e.g. one saved by `text2model --editor`) instead
# of the default skadio/text2zinc HuggingFace dataset
python evals/evaluate.py --output-dir my_results --dataset-path text2zinc_edited.csv

Running the eval generates a JSON file (evals/evaluation_results.json by default, via --output-json) with your accuracy metrics. You can PR that file to the Text2Model Leaderboard on Hugging Face to get your results added to the online leaderboard.

Metrics

Metric	Description
Execution Accuracy	% of models that compile and run without errors
Solution Accuracy	% of models that produce correct solutions
Average Score	Average of execution and solution accuracy

Results are broken down by problem type as satisfaction vs. optimization.

Leaderboard

Text2Model hosts an online leaderboard tracking execution accuracy, solution accuracy, and average score across models and copilot strategies on the Text2Zinc benchmark:

Text2Model Leaderboard (Hugging Face Spaces)

Testing

Default tests (tests/test_main.py, tests/test_utils.py) do not need an API key, network, or MiniZinc. They are pure logic tests with mocked API calls and is run by the CI:

pytest -m "not integration"

Integration tests (tests/test_integration.py) hit real external dependencies and are opt-in only. This is not to be used in CI:

• MiniZinc tests run the real minizinc binary and are skipped unless it is on PATH.
• The OpenAI test makes exactly one real, cheap, token-capped call (gpt-4o-mini, max_tokens=20) and is skipped unless OPENAI_API_KEY is set. It's intentionally not exhaustive to avoid API costs.

To run everything locally (with OPENAI_API_KEY set and MiniZinc installed):

pytest -m ""

Repository Structure

text2model/
├── text2model/                  # Installable Python package
│   ├── prompts/                 # Prompt templates for each strategy
│   │   ├── cot_prompt.txt
│   │   ├── code_validation_prompt.txt
│   │   ├── global_constraint_prompts/
│   │   └── ...
│   ├── knowledge_graphs/        # KG files (.ttl) for knowledge_graph strategy
│   ├── editor/                  # Dataset editor GUI (`text2model --editor`)
│   │   ├── app.py               # Flet app: browse/edit/execute Text2Zinc problems, AI chat assistant
│   │   └── data/text2zinc.csv   # Bundled default dataset the editor opens on first run
│   ├── grammar.mzn              # MiniZinc grammar for validation
│   ├── main.py                  # Copilot strategies and CLI entry point
│   ├── generate_knowledge_graph.py  # Generates KGs for the knowledge_graph strategy
│   └── utils.py                 # Shared utilities (API calls, validation, dataset loading)
├── tests/                       # Unit and integration tests (pytest)
├── evals/                       # Evaluation tooling
│   ├── evaluate.py              # Evaluates generated MiniZinc models
│   ├── evaluation_results.json  # Latest evaluation results (PR-able to the HF leaderboard)
│   └── results/                 # Accuracy metrics and leaderboard from paper runs
├── output/                      # Original outputs per strategy, kept for reproducing paper results
│   └── [model]/[strategy]/      # e.g., gpt-4/cot/problem_1.mzn
├── pyproject.toml               # Package metadata and install config
├── CHANGELOG.txt                # Release history
└── LICENSE                      # Apache License 2.0

Support

Please submit bug reports and feature requests as Issues.

License

Text2Model is licensed under the Apache License 2.0.