LMUnit: Fine-grained Evaluation with Natural Language Unit Tests
This repository provides code for evaluation and reproduction of our results in LMUnit: Fine-grained Evaluation with Natural Language Unit Tests.
🚀 Recent Updates
- 2025-07: Open-Sourced LMUnit checkpoints Llama 3.1-70B and Qwen 2.5-72B models. Besides, evaluation framework for reproducing our direct scoring and preference tasks results
📋 Table of Contents
🔧 LMUnit Inference Example
Install LMunit
pip install lmunit
Run Inference Example Script
from lmunit import LMUnit
from vllm import SamplingParams
# Initialize LMUnit with your model
model = LMUnit(
model_path="ContextualAI/LMUnit-llama3.1-70b",
tp_size=4
)
sampling_params = SamplingParams(temperature=0.0, max_tokens=10, logprobs=20)
# Define your evaluation prompt
unit_test = "Does the response specify minimum system logs required for compliance?"
query = "What security validation protocols ensure HIPAA compliance when connecting research databases to clinical systems?"
response = """Customer Concentration Risk Comparison:
Snowflake's Platform segment shows higher concentration risk than Product segment:
Product: 15% (Customer X), 8% (Customer Y), 77% others
Platform: 20% (Customer Z), 5% (Customer W), 75% others
HIPAA Technical Controls:
Data Encryption Requirements:
Regulation: 45 CFR § 164.312(a)(2)(iv)
Validation: All data encrypted at rest/transit using FIPS 140-2 algorithms
Testing: Vulnerability scanning and penetration testing for encryption weaknesses
Additional Compliance Measures:
Risk analysis for security threats
Access controls for PHI authorization
Incident response planning
Required logs: encryption key management, data access, security incidents"""
prompt = f"Query: {query}\n\nResponse: {response}\n\nUnit Test: {unit_test}"
output = model.generate(prompt, sampling_params)
📁 Repository Structure
lmunit/
├── assets/ # Documentation assets and figures
├── eval/ # Evaluation scripts and benchmarks
│ ├── eval.py # Main evaluation script
│ └── reward_bench2.py # Reward benchmarking utilities
├── lmunit/ # Core LMUnit package
│ ├── __init__.py # Package initialization
│ ├── constants.py # Framework constants
│ ├── lmunit.py # Main LMUnit class implementation
│ ├── metrics.py # Evaluation metrics
│ └── tasks.py # Task definitions and utilities
├── requirements/ # Dependencies
├── requirements.txt # Main dependencies
└── dev.txt # Development dependencies
📄 Artifacts
📋 Paper
🤗 HuggingFace Collection
- LMUnit Models Collection - Pre-trained models and evaluation datasets
💾 Checkpoints
| Model | Flask | BiGGen-Bench | Human-Internal | InfoBench | RB | LFQA | RB2 |
|---|---|---|---|---|---|---|---|
| LMUnit-LLaMA-3.1-70B | 72.03 | 67.69 | 93.63 | 89.00 | 91.56 | 76.15 | 80.5 |
| LMUNIT_Qwen2.5-72B | 73.85 | 69.56 | 94.44 | 88.67 | 91.13 | 73.85 | 82.1 |
🚀 Quick Start
Installation
pip install lmunit
Run Evaluation Task
For running an specific task on an LMUnit model
python eval/eval.py --task <task> --model-path <lmunit-model> --tensor-parallel-size <tp-size>
For running rewardbench2 results:
python eval/reward_bench2.py --model-path <lmunit-model> --tensor-parallel-size <tp-size>
Reproduce LMUnit evaluation suite
./scripts/run_all_evaluations.sh <model_path> <tensor_parallel_size> [output_dir]
Citation
@misc{saadfalcon2024lmunitfinegrainedevaluationnatural,
title={LMUnit: Fine-grained Evaluation with Natural Language Unit Tests},
author={Jon Saad-Falcon* and Rajan Vivek* and William Berrios* and Nandita Shankar Naik and Matija Franklin and Bertie Vidgen and Amanpreet Singh and Douwe Kiela and Shikib Mehri},
year={2024},
eprint={2412.13091},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2412.13091},
note={*Equal contribution}
}
Metadata
Release files for lmunit 1.0.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| lmunit-1.0.1.tar.gz | 11.7 kB | Without provenance | Details for lmunit-1.0.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| lmunit-1.0.1-py3-none-any.whl | Python 3 | none | any | Details for lmunit-1.0.1-py3-none-any.whl |
Total release size: 21.1 kB
Release files / lmunit-1.0.1.tar.gz
| Download URL | lmunit-1.0.1.tar.gz |
|---|---|
| Size | 11.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
4a569c622b866b8f3cb210a3746994618e87c92acebad6494245799db6610d9a
|
|
BLAKE2b-256 checksum How to use checksums |
ec9e800a0bf5e65c5380150f9cedb6c873e6b6af1eac9427e42007d77bb5d192
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.10.15
|
Release files / lmunit-1.0.1-py3-none-any.whl
| Download URL | lmunit-1.0.1-py3-none-any.whl |
|---|---|
| Size | 9.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2eda7a6d8abb93e25741aa2dd4c6a72130e4f12f0e50b93c0165e81aeb2b1341
|
|
BLAKE2b-256 checksum How to use checksums |
e796db9c91b4396518aceef05ced4fa14ecae00625afacda2e514abc7a0d238b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.10.15
|
Security for lmunit 1.0.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.