NerfProbe Core

Scientifically-grounded LLM degradation detection.

nerfprobe-core provides the essential detection logic, scorers, and probe definitions used by the NerfProbe CLI and NerfStatus. It is designed for developers who need rigorous, research-backed instruments to measure model quality, consistency, and alignment.

Installation

pip install nerfprobe-core

Features

• 17 Research-Backed Probes: Detection instruments grounded in specific academic papers on model collapse and quantization artifacts.
• 3-Tier Architecture: Organized into Core (Essential), Advanced (Structural), and Optional (Experimental) tiers.
• Universal Scoring: Reusable scorers (JSON schema validation, TTR, Fact verification) independent of the probe execution.
• Type-Safe: Fully typed with Python 3.11+, leveraging Pydantic for configuration and results.

Probes

Core Tier (Essential Signals)

Advanced Tier (Structural Integrity)

Optional Tier (Experimental)

Usage

Basic Probe Execution

import asyncio
from nerfprobe_core import ModelTarget
from nerfprobe_core.probes import MathProbe
from nerfprobe_core.probes.config import MathProbeConfig

# 1. Configure the probe
config = MathProbeConfig(
    prompt="Calculate 15 * 12 + 8.",
    expected_answer="188",
)

# 2. Define the target
target = ModelTarget(provider_id="openai", model_name="gpt-4o")

# 3. Instantiate and run (requires an LLM Gateway)
probe = MathProbe(config)
# result = await probe.run(target, gateway)

# print(result.summary())
# > math_probe: PASS (1.00) in 234ms

Using Scorers Directly

You can use the scoring logic without the full probe infrastructure:

from nerfprobe_core.scorers import JsonScorer

scorer = JsonScorer(strict=True)
valid_json = '{"name": "NerfProbe"}'
invalid_json = '```json{"name": "NerfProbe"}```'

score, metadata = scorer.score(valid_json)
print(f"Score: {score}") # 1.0

score, metadata = scorer.score(invalid_json)
print(f"Score: {score}") # 0.0 (Strict mode rejects markdown blocks)

Contributing

We welcome contributions! Please see CONTRIBUTING.md for details on how to set up the development environment, run tests, and submit PRs.

License

Apache-2.0