lica-gdb-helm

HELM integration for GDB (GraphicDesignBench) — run all 39 GDB benchmarks through Stanford CRFM's HELM framework.

Install

pip install lica-gdb-helm

This installs lica-gdb and crfm-helm as dependencies. For benchmarks with heavier metrics:

pip install "lica-gdb-helm[svg]"      # SVG rendering metrics (LPIPS, SSIM, CLIP)
pip install "lica-gdb-helm[layout]"   # Layout generation metrics (NIMA, HPS, FID)
pip install "lica-gdb-helm[full]"     # Everything

Usage

# Run a single benchmark
helm-run --run-entries gdb:benchmark_id=category-1,model=openai/gpt-4o \
         --suite gdb-eval --max-eval-instances 50

# Run multiple benchmarks
helm-run --run-entries gdb:benchmark_id=svg-1,model=openai/gpt-4o \
                       gdb:benchmark_id=svg-2,model=openai/gpt-4o \
         --suite gdb-eval

# Summarize and view results
helm-summarize --suite gdb-eval
helm-server --suite gdb-eval

Available benchmarks

All 39 GDB benchmarks are available. Pass any benchmark ID:

Options

How it works

This package is a thin adapter (~300 lines) that translates between HELM and GDB types. All benchmark logic — data loading, prompt construction, output parsing, and metric computation — is delegated to the lica-gdb package. Metrics from HELM runs are identical to standalone GDB evaluation.

Development

cd integrations/helm
pip install -e "../../[hub]"   # install lica-gdb in editable mode
pip install -e .               # install lica-gdb-helm in editable mode

# Test with HELM's built-in echo model
helm-run --run-entries gdb:benchmark_id=category-1,model=simple/model1 \
         --suite test --max-eval-instances 5