text2model 1.0.0

LLM Modeling Copilots for Text-to-Model Translation

---

Text Mode • Text2Zinc Mode • Copilots • Installation • Evaluation • Leaderboard

---

Text2Model: LLM Modeling Copilots for Text-to-Model Translation

Text2Model is a suite of LLM modeling copilots, datasets, fined-tuned models, demos, interactive editor, and online leaderboard for translating natural language text into formal combinatorial constraint models.

Text2Model uses MiniZinc as the target modeling language which makes our copilots both paradigm- and solver-agnostic. Our copilots generate models that can be solved by any MiniZinc compatible solver including Gecode, Chuffed, OR-Tools, CBC, Gurobi, Cplex, HiGH. This covers a wide range of paradigms including CP, CP-SAT, and MIP. As such, Text2Model can address both combinatorial satisfaction and optimization problems.

Please visit Text2Model for latest publications and resources.

Quick Start

Text2Model supports translating given problem descriptions, Text mode, or specific problems from our dataset, Text2Zinc mode.

Text Mode

# Translate a given problem description
text2model --problem "A country produces fighter jets each year. Some of these jets must be set aside for pilot training instead of combat use. Year 1 production is 10 jets, and Year 2 production is 15 jets. Each training jet can train 5 pilots per year. Training runs for 2 years, starting in Year 1. Determine how many pilots will be trained in total by the end of Year 2."

# Translate a given problem file
text2model --problem my_problem.txt

# Choose the copilot strategy and the model
# Note that `knowledge_graph` is not available for text-mode as it requires pre-built TTL files
text2model --problem my_problem.txt --strategies agents_with_code_validation --model gpt-4o

# Redirect the output to a MiniZinc model
text2model --problem my_problem.txt > my_model.mzn

Text2Zinc Mode

# Translate specific problems from Text2Zinc
text2model --problem-ids 0 1 2 --strategies cot --model gpt-4 --output-dir my_results

# Run multiple strategies on all Text2Zinc problems
text2model --strategies cot --model gpt-4 --output-dir my_results

# Run all strategies
text2model --strategies all --model gpt-4 --output-dir my_results

# Run on specific source of Text2Zinc problems
text2model --source nlp4lp --strategies cot --model gpt-4 --output-dir my_results

# List all available data sources
text2model --list-sources

# List all available model options
# --model accepts gpt-4, gpt-4o, gpt-5.2 (OpenAI, requires OPENAI_API_KEY), or phi4 (served locally through Ollama, no API key needed)

# Advanced options
text2model --strategies agents --model gpt-4 \
  --output-dir my_results \
  --temperature 0.7 \
  --max-tokens 8192 \
  --sleep-time 2 \
  --include-unverified

Copilots

Text2Model offers different copilot strategies, ranging from simple single-call approaches to sophisticated multi-agent systems. Each makes different trade-offs between speed and accuracy.

Strategy	Description
baseline	Direct code generation from problem description. No special prompting. Good for simple problems or establishing a baseline.
cot	Chain-of-Thought prompting with guiding principles. The LLM reasons through the problem step-by-step before generating code.
knowledge_graph	First extracts structured information (entities, relationships) from the problem, then generates code from this intermediate representation.
cot_with_code_validation	Generates code with CoT, then validates and fixes any compilation errors. Good default choice.
cot_with_grammar_validation	Generates code with CoT, then checks against MiniZinc grammar rules.
cot_with_code_and_grammar_validation	Combines CoT generation with both grammar checking and code validation.
agents	Decomposes the task into specialized agents: (1) parameters & variables, (2) constraints, (3) objective, (4) assembler that stitches everything together.
agents_with_code_validation	Agents approach plus a final validation/fix step.
gala	Global Agents for different constraint types (all_different, cumulative, etc.) plus an assembler. See the GALA paper.

Installation

Text2Model requires Python 3.8+ and can be installed from PyPI or by building from source.

Set Your API Key

export OPENAI_API_KEY="your-api-key-here"

Install from PyPI

pip install text2model

Install from Source

git clone https://github.com/skadio/text2model.git
cd text2model
pip install -e .

Evaluation

After generating models, evaluate their correctness via evaluate.py. This script compiles and runs each generated MiniZinc model against test instances, checking for both execution success and solution correctness.

Prerequisite

Install MiniZinc solver: https://www.minizinc.org/doc-2.5.5/en/installation.html

Evals

# Evaluate all generated code
# `--output-dir` is required. Point it at the directory produced by the batch-mode `text2model` run.
python evaluate.py --output-dir my_results

Metrics

Metric	Description
Execution Accuracy	% of models that compile and run without errors
Solution Accuracy	% of models that produce correct solutions
Average Score	Average of execution and solution accuracy

Results are broken down by problem type as satisfaction vs. optimization.

Leaderboard

Text2Model hosts an online leaderboard tracking execution accuracy, solution accuracy, and average score across models and copilot strategies on the Text2Zinc benchmark:

Text2Model Leaderboard (Hugging Face Spaces)

Testing

Install test dependencies with pip install -e ".[test]".

Default tests (tests/test_main.py, tests/test_utils.py) do not need an API key, network, or MiniZinc. They are pure logic tests with mocked API calls and is run by the CI:

pytest -m "not integration"

Integration tests (tests/test_integration.py) hit real external dependencies and are opt-in only. This is not to be used in CI:

• MiniZinc tests run the real minizinc binary and are skipped unless it is on PATH.
• The OpenAI test makes exactly one real, cheap, token-capped call (gpt-4o-mini, max_tokens=20) and is skipped unless OPENAI_API_KEY is set. It's intentionally not exhaustive to avoid API costs.

To run everything locally (with OPENAI_API_KEY set and MiniZinc installed):

pytest -m ""

Repository Structure

text2model/
├── text2model/                  # Installable Python package
│   ├── prompts/                 # Prompt templates for each strategy
│   │   ├── cot_prompt.txt
│   │   ├── code_validation_prompt.txt
│   │   ├── global_constraint_prompts/
│   │   └── ...
│   ├── knowledge_graphs/        # KG files (.ttl) for knowledge_graph strategy
│   ├── grammar.mzn              # MiniZinc grammar for validation
│   ├── main.py                  # Copilot strategies and CLI entry point
│   └── utils.py                 # Shared utilities (API calls, validation)
├── tests/                       # Unit and integration tests (pytest)
├── output/                      # Generated models (created automatically)
│   ├── [model]/[strategy]/      # e.g., gpt-4/cot/problem_1.mzn
│   └── evaluation_results/      # Accuracy metrics and leaderboard
├── evaluate.py                  # Evaluates generated MiniZinc models
├── generate_knowledge_graph.py  # Generates KGs for knowledge_graph strategy
├── pyproject.toml               # Package metadata and install config
├── CHANGELOG.txt                # Release history
└── LICENSE                      # Apache License 2.0

Support

Please submit bug reports and feature requests as Issues.

License

Text2Model is licensed under the Apache License 2.0.