Astro-Eval

A reusable evaluation framework for language models.

Evaluate models with standardized benchmarks, generation metrics, factuality analysis, error analysis, experiment tracking, model comparison, and reproducible reports.

pip install astro-eval

---

Why Astro-Eval?

Evaluating an LLM should not mean maintaining a collection of one-off scripts.

Astro-Eval provides a reusable evaluation layer that separates model execution from evaluation logic.

                    Model
                      │
                      ▼
                 Astro-Eval
                      │
        ┌─────────────┼─────────────┐
        ▼             ▼             ▼
    Benchmarks     Metrics      Factuality
        │             │             │
        └─────────────┼─────────────┘
                      ▼
               Error Analysis
                      │
                      ▼
              Reports / Tracking

---

Features

• Model Adapters: Decouple model loading and generation implementations from the evaluation loop.
• Reusable Benchmarks: Standardized schema for loading, validating, and iterating domain-specific prompts.
• Generation Metrics: Automatic characterization of output length and processing throughput (tokens/sec).
• Diversity & Repetition: Built-in distinct-N diversity metrics and repetition rate analysis.
• Scientific/Domain Metrics: Specialized metrics such as scientific term checking, prompt overlap, and composite quality scores.
• Factuality Scoring: Detailed factuality extraction, concept coverage, and reference overlap checks.
• Error Analysis: Automated classification of generation errors (PASS, PARTIAL, FAIL).
• Model Comparison: Multi-model comparison leaderboard generator with automated winner determinations.
• Experiment Tracking: Saves reproducible run records containing hyperparameters, configurations, and aggregate scores.
• Markdown Reports: Exports full evaluation reports detailing overview, generation, domain quality, and factuality.

---

Quick Start

1. Installation

pip install astro-eval

2. Run an Evaluation

astro-eval run \
  --model astrogpt \
  --checkpoint checkpoints_50m/best.pt \
  --benchmark astronomy

3. List Available Benchmarks

astro-eval benchmarks list

4. List Registered Model Adapters

astro-eval adapters list

5. Generate a Markdown Report

astro-eval report \
  --input astro_eval_results/best_evaluation.json \
  --output report.md

---

Python Developer API

Integrate evaluation directly into your development or training pipelines:

from astro_eval import (
    Evaluator,
    load_benchmark,
    get_adapter,
)

# 1. Load benchmark prompts
benchmark = load_benchmark("astronomy")

# 2. Get and load model adapter
model = get_adapter("astrogpt")
model.load("checkpoints_50m/best.pt")

# 3. Initialize Evaluator
evaluator = Evaluator(
    model=model,
    benchmark=benchmark,
)

# 4. Run evaluation
results = evaluator.evaluate(
    max_new_tokens=150,
    temperature=0.9,
    top_k=100,
)

# 5. Output summary metrics
print(results.summary())

---

Model Adapters

Astro-Eval separates model-specific code from evaluation logic.

An adapter implements:

• load(checkpoint_path): Loads and compiles the model weight parameters onto the device.
• generate(prompt, max_new_tokens, temperature, top_k): Runs generation inference and returns structured text details.
• evaluate_loss(x, y): Computes forward validation loss over tokens.
• metadata(): Returns dictionary of checkpoint iteration and config.

The framework currently includes:

• AstroGPT: Adapter implementation for the AstroGPT astronomical model.

The adapter architecture allows additional model implementations to be added without changing the evaluation engine.

---

Architecture

src/astro_eval/
├── adapters/
│   ├── base.py
│   ├── registry.py
│   └── astrogpt.py
│
├── benchmarks/
│   ├── astronomy.json
│   ├── astrophysics.json
│   ├── cosmology.json
│   ├── general_science.json
│   ├── physics.json
│   └── prompts.json
│
├── metric/
│   ├── builtin.py
│   ├── defaults.py
│   └── registry.py
│
├── schema/
│   └── result.py
│
├── evaluator.py
├── factuality.py
├── error_analysis.py
├── comparison.py
├── experiments.py
├── report.py
└── cli.py

---

Example Evaluation

Astro-Eval was initially developed and validated using AstroGPT, an astronomy-focused language model.

Example evaluation metrics:

Metric	Result
Validation Loss	3.3323
Distinct-1	0.9179
Distinct-2	0.9952
Distinct-3	1.0000
Repetition Rate	0.0000
Tokens/sec	104.22

The framework also produces factuality and error-analysis outputs for benchmark-level evaluation.

---

Design Philosophy

Astro-Eval is designed around four principles:

1. Model-agnostic evaluation: Run standard benchmarks across any LLM architecture.
2. Reproducible experiments: Track runs with unique IDs to easily compare iterations.
3. Extensible metrics and adapters: Easily register custom metrics or model APIs.
4. Machine-readable results: Save evaluation outputs in structured, versioned JSON schemas.

---

Roadmap

• Model adapter system
• Benchmark system
• Generation metrics
• Factuality evaluation
• Error analysis
• Model comparison
• Experiment tracking
• Markdown reporting
• Python API
• CLI
• More model adapters
• More public benchmarks
• Statistical significance testing
• HTML evaluation dashboards

---

Contributing

Contributions are welcome. You can contribute by:

• Adding a model adapter
• Adding a benchmark
• Adding a metric
• Improving evaluation methods
• Improving documentation
• Reporting bugs

Open an issue or submit a pull request on GitHub.