gepa-dapo-grn 0.4.0

GEPA-aware DAPO reinforcement learning with optional Global Response Normalization

gepa-dapo-grn

gepa-dapo-grn is a standalone reinforcement learning engine for research workflows. It is GEPA-shaped but GEPA-agnostic, providing DAPO and MaxRL-style optimization, curriculum tracking, safety controls, verifier-first hooks, and optional Global Response Normalization (GRN).

What this library is

• Standalone RL engine with a stable public API under gepa_dapo_grn.*.
• GEPA-shaped but GEPA-agnostic: feedback is structured reward/tag/verifier dictionaries.
• Supports DAPO or MaxRL + curriculum + safety + GRN with conservative defaults and GRN disabled by default (GRNConfig.enabled=False). At runtime, SafetyController.adjust_grn_config(...) (and internally SafetyController._apply_grn_adjustment) can auto-enable GRN when excess risk (max(0, risk_score - risk_tolerance)) exceeds grn_enable_threshold. It is recommended to use risk_tolerance=0.0 (the default) for straightforward behavior. Using a nonzero tolerance can suppress triggering; for example, a risk_score of 0.6 with a risk_tolerance of 0.2 yields an excess risk of 0.4.

Backends

• DAPO: general GEPA-shaped RL with mixed reward dimensions, curriculum, safety control, and optional GRN.
• MaxRL: verifier-heavy training for binary or near-binary correctness settings.

MaxRL is typically best when tasks have robust validators (for example code generation, exact math checks, executable test suites, and other verifier-backed tasks). Non-binary ethics/alignment training generally still benefits from DAPO-style hybrid control.

Minimal backend selection snippet:

from gepa_dapo_grn import (
    DAPOConfig,
    MaxRLConfig,
    TrainerBackendConfig,
    make_trainer,
)

trainer = make_trainer(
    policy=policy,
    optimizer=optimizer,
    backend_config=TrainerBackendConfig(backend="maxrl"),
    maxrl_config=MaxRLConfig(enabled=True, num_samples=4),
)

# DAPO backend example:
dapo_trainer = make_trainer(
    policy=policy,
    optimizer=optimizer,
    backend_config=TrainerBackendConfig(backend="dapo"),
    dapo_config=DAPOConfig(),
)

Practical guidance (v0.3.0)

• Verifier-first: use VerifierResult and GEPAFeedback.verifier for pass/fail, scores, confidence, coverage, and diagnostics.
• Composition curriculum: CurriculumTracker tracks saturation and composition depth; use TaskComposer (or SimpleTextComposer) to generate harder tasks as easier ones saturate.
• Soft gating option: set DAPOConfig(use_soft_gating=True) to smoothly down-weight ratio outliers instead of hard clipping.
• Deception handling policy: do not apply built-in deception penalties. Treat deception-like signals as tags/risk/controller inputs (abstention, calibration, verifier constraints).
• GRN placement guidance: GRN is off by default. Prefer enabling only named policy/value modules via include_modules/exclude_modules, and keep probe/interpretability modules unwrapped unless explicitly included.

Optional Graph-Active-DAPO

The package also includes optional graph-native feedback and Active-GRPO-style adaptive references. These modules can be used independently or combined through GraphActiveDapoTrainer.

• gepa_dapo_grn.graph scores public graph artifacts: claims, evidence links, mechanisms, assumptions, constraints, contradictions, and answer alignment.
• gepa_dapo_grn.active_grpo tracks per-prompt references, chooses imitate/mixed/reinforce modes, and promotes policy candidates only through strict external-verifier and safety gates.
• Hidden chain-of-thought is not stored or rewarded. The graph layer uses public artifacts only: graph JSON, public assumptions, claims, evidence, tests, answer summaries, and verifier outputs.
• Reference promotion is disabled by default and cannot rely on model self-score alone when external verification is required.

See docs/graph_active_dapo.md and the examples/graph_*demo.py files for minimal usage.

Install

pip install gepa-dapo-grn

Optional extras:

• gepa-dapo-grn[hf] adds HuggingFace integration helpers.
• gepa-dapo-grn[dev] installs test and formatting tools.

Minimal example (CPU-safe)

from gepa_dapo_grn import DAPOTrainer, DAPOConfig, GEPAFeedback, RewardMixerConfig

fb = GEPAFeedback(
    rewards={"truth": 1.0, "helpfulness": 0.5},
    tags={"risk_score": 0.1},
    verifier={"verifier_pass": 1.0, "verifier_confidence": 0.9},
    meta={"task_id": "demo"},
    abstained=False,
)

Building and installing a local wheel safely

If you build multiple versions locally, avoid pip install dist/*.whl because pip will try to install all matching wheel files (which can include multiple versions of this same package and fail with ResolutionImpossible).

Use this sequence instead:

rm -rf build dist *.egg-info
python3 -m pip install --upgrade build tomli
python3 -m build
python3 scripts/install_local_wheel.py --prune-other-versions

This follows pip's suggested fix to remove conflicting versions before install: by default, scripts/install_local_wheel.py only locates and installs the current version wheel and does not delete files in dist/. Cleanup is opt-in: --remove-version <version> deletes wheel files for the specified version(s), and --prune-other-versions removes other non-current wheel files so pip receives exactly one path.

Publishing to PyPI safely

A common cause of InvalidDistribution during twine upload dist/* is stale or non-package artifacts left in dist/. Use a clean build and upload only the current version artifacts:

rm -rf build dist *.egg-info
python3 -m pip install --upgrade build twine tomli
# Build isolation uses pyproject build-system pins (setuptools<77) for twine compatibility.
python3 -m build
PROJECT_VERSION=$(python - <<'PY2'
from pathlib import Path
try:
    import tomllib as toml
except ImportError:
    import tomli as toml
import re
from packaging.version import Version, InvalidVersion

data = toml.loads(Path('pyproject.toml').read_text(encoding='utf-8'))
project = data.get('project', {})
version = project.get('version')
if isinstance(version, str) and version.strip():
    print(version.strip())
else:
    # Dynamic version fallback: read built artifact name from dist.
    # Expect files like gepa_dapo_grn-0.3.0-py3-none-any.whl
    wheel_names = list(Path('dist').glob('gepa_dapo_grn-*.whl'))
    if not wheel_names:
        raise SystemExit("No wheel found in dist for dynamic version resolution")
    
    def get_version(path):
        match = re.match(r"gepa_dapo_grn-([^-]+)-", path.name)
        if not match:
            raise SystemExit(f"Unable to parse version from wheel name: {path.name}")
        try:
            return Version(match.group(1))
        except InvalidVersion:
            raise SystemExit(f"Invalid semantic version in wheel name: {path.name}")

    newest_wheel = max(wheel_names, key=get_version)
    match = re.match(r"gepa_dapo_grn-([^-]+)-", newest_wheel.name)
    print(match.group(1))
PY2
)
python scripts/validate_dist_metadata.py \
  dist/gepa_dapo_grn-${PROJECT_VERSION}.tar.gz \
  dist/gepa_dapo_grn-${PROJECT_VERSION}-*.whl
twine check dist/gepa_dapo_grn-${PROJECT_VERSION}.tar.gz dist/gepa_dapo_grn-${PROJECT_VERSION}-*.whl
twine upload dist/gepa_dapo_grn-${PROJECT_VERSION}.tar.gz dist/gepa_dapo_grn-${PROJECT_VERSION}-*.whl

This avoids uploading unrelated files and ensures both Name and Version metadata come from the freshly built distributions only.

If twine check reports supported metadata versions only up to 2.2 while your wheel has a newer Metadata-Version (for example 2.4 from newer setuptools), upgrade your upload tooling first:

python -m pip install --upgrade twine pkginfo

Public API

Public API is defined by __init__.py exports. Anything not exported there is considered internal and may change without notice.

Versioning policy

This project follows semantic versioning:

• 0.x.y while interfaces are still evolving
• bump minor for interface changes
• bump patch for bugfixes only

See CHANGELOG.md for release notes.

License

MIT (see LICENSE).