sysml-bench

Benchmarking harness for evaluating LLM performance on SysML v2 model comprehension tasks with CLI tool augmentation. Part of Nomograph Labs.

Repository: gitlab.com/nomograph/sysml-bench

What This Is

A Python evaluation framework that measures how different tool configurations affect LLM accuracy on structured systems engineering tasks. The benchmark uses a real SysML v2 model corpus (an Eve Online Mining Frigate design) and tests five models across 40+ experimental conditions with 132 active tasks.

The primary question: does giving an LLM more tools improve its ability to answer questions about a SysML v2 model? The answer is nuanced — it depends on the task type, the model, and whether the agent receives guidance on tool selection.

Key Observations

14 observations from our evaluation. Selected highlights:

Corpus

Eve Online Mining Frigate — a SysML v2 model of a fitted mining ship designed for the Eve Online universe. 19 files, 798 elements, 1,515 relationships. Covers requirements, concerns, stakeholders, logical architecture, COTS modules, interfaces, verification cases, and rollup analysis.

A secondary corpus of 95 files (SysML v2 specification examples) is used for scaling experiments.

Task Categories

Primary corpus (Eve Online Mining Frigate, 19 files)

Multi-corpus discovery (scaling and generalization)

Total: 132 active tasks across 13 task files. An additional 82 archived tasks from earlier experimental phases are in eval/tasks/archive/.

Tool Sets

Scoring

Per-field structured scoring with the following types:

Task score = mean of field scores. Condition score = mean of task scores across N runs.

Reproduction

Prerequisites

• Python ≥3.12
• uv package manager
• nomograph-sysml CLI binary on $PATH
• ANTHROPIC_API_KEY and/or OPENAI_API_KEY environment variables

Setup

git clone https://gitlab.com/nomograph/sysml-bench.git
cd sysml-bench
uv sync

Run an experiment

uv run python -m eval.llm_cli \
    --task-file eval/tasks/eve_discovery.yaml \
    --models claude-sonnet-4-20250514 \
    --tool-set cli_search \
    --runs 3 --max-turns 15 \
    --output results/my-experiment.json

With Docker

docker pull registry.gitlab.com/nomograph/sysml-bench:latest
docker run -e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
    registry.gitlab.com/nomograph/sysml-bench:latest \
    python -m eval.llm_cli \
    --task-file eval/tasks/eve_discovery.yaml \
    --models claude-sonnet-4-20250514 \
    --tool-set cli_search --runs 1 --max-turns 15 \
    --output /results/my-experiment.json

With guided system prompt

uv run python -m eval.llm_cli \
    --task-file eval/tasks/eve_discovery.yaml \
    --models claude-sonnet-4-20250514 \
    --tool-set cli_graph \
    --system-prompt-file eval/prompts/guided_discovery.txt \
    --runs 5 --max-turns 15 \
    --output results/guided-experiment.json

With vector search

uv run python -m eval.llm_cli \
    --task-file eval/tasks/eve_discovery.yaml \
    --models claude-sonnet-4-20250514 \
    --tool-set cli_search --vectors \
    --runs 3 --max-turns 15 \
    --output results/vector-experiment.json

Results

Result JSON files are stored separately in nomograph/sysml-bench-results (private — contains model outputs and cost data). Each result file is a JSON document containing run metadata and per-task scored results with cost, token counts, and per-field scoring breakdowns.

When running experiments, output files are written to a local results/ directory (gitignored in this repo).

Dataset

The benchmark tasks and baseline results are available on HuggingFace:

from datasets import load_dataset
tasks = load_dataset("nomograph/sysml-v2-reasoning-benchmark", split="tasks")
results = load_dataset("nomograph/sysml-v2-reasoning-benchmark", split="results")

See nomograph/sysml-v2-reasoning-benchmark for the full dataset card.

Known Limitations

1. Corpus size: Primary corpus is 19 files. Results may not generalize to production-scale models (1000+ files). Scaling experiments (O10) show dramatic performance drops at 95 files.

2. Model versions: All results are from specific model snapshots (claude-sonnet-4-20250514, gpt-4o-2024-08-06, etc.). Results may differ with future model versions.

3. Stochastic variance: Some tasks show high run-to-run variance (e.g., D12, D13 are bimodal). N=3-5 replication mitigates but does not eliminate this.

4. ListStr scoring: The set-based F1 scorer handles flat string lists only. Compound answer types (list-of-dicts) require decomposition into separate fields.

5. Single domain: All tasks are from a SysML v2 model. The observations may not generalize to other modeling languages or domains.

License

MIT — see LICENSE

Citation

@article{dunn2026sysmlbench,
  title={sysml-bench: Evaluating Tool-Augmented {LLMs} on {SysML} v2 Model Comprehension},
  author={Dunn, Andrew},
  year={2026},
  journal={arXiv preprint},
  url={https://gitlab.com/nomograph/sysml-bench},
  note={Nomograph Labs}
}