entropy-loop-core 0.9.0

A Failure Compiler for AI agents: turn failed outputs into lessons, evaluation summaries, and replayable regression cases.

Entropy Loop Core

A Failure Compiler for AI agents. Turn failed agent outputs into regression cases and replay them before the same bug ships again.

Get started · Example · CLI · Architecture · Releases · Contributing

Star the repo if you want to follow the Failure Compiler roadmap.

Get started

pip install entropy-loop-core
entropy-loop replay-demo

Works on Windows, macOS, and Linux with Python 3.10+.

Development setup

Use a virtual environment when working on the repository.

macOS / Linux

git clone https://github.com/koreaelonmusk/entropy-loop-core.git
cd entropy-loop-core
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"
pytest

Windows PowerShell

git clone https://github.com/koreaelonmusk/entropy-loop-core.git
cd entropy-loop-core
py -m venv .venv
.\.venv\Scripts\Activate.ps1
py -m pip install --upgrade pip
py -m pip install -e ".[dev]"
pytest

Why

AI agents often fail the same way twice.

Entropy Loop Core makes failures reusable: capture the failed output, classify it, compile a lesson, generate a regression case, and replay it — before the same bug ships again.

Task
→ AgentOutput
→ VerificationResult
→ FailureTrace
→ Lesson
→ RegressionCase
→ RegressionSuite
→ Replay
→ Report

The core is deterministic: no LLM calls, no network calls, no hidden state.

Example

Turn a failure into a regression case, then replay it against a fixed agent:

from entropy_loop_core import (
    AgentOutput,
    FailureTrace,
    RegressionRunner,
    RegressionSuite,
    RetryContext,
    Task,
    VerificationPolicy,
    Verifier,
    generate_regression_case,
)

# A verifier built from a policy: non-empty output that contains "status".
verifier = Verifier.from_policy(
    VerificationPolicy(require_non_empty=True, required_terms=["status"])
)

# A past failure (the agent omitted "status") becomes a regression case.
task = Task(id="job-1", instruction="report the job status")
bad = AgentOutput(content="done")
case = generate_regression_case(
    FailureTrace(
        task=task,
        output=bad,
        verification_result=verifier.verify(bad),
        attempt=1,
    )
)


# Replay the case against a corrected agent.
def fixed_agent(task: Task, ctx: RetryContext) -> AgentOutput:
    return AgentOutput(content="status: ok")


report = RegressionRunner().run_suite(
    RegressionSuite(name="job", cases=[case]), fixed_agent, verifier
)
print(report.passed, report.total_cases, report.success_rate)  # 1 1 100.0

Full worked example: examples/json_agent_guard.py.

CLI

entropy-loop replay-demo   # generate a regression case, then replay it as a suite
entropy-loop memory-demo   # compact repeated failure lessons with a MemoryPolicy
entropy-loop pack-demo     # build, save, load, and run a regression pack
entropy-loop agent-demo    # refresh a pack from an agent, then run it
entropy-loop triage-demo   # diff a baseline run against a current run
entropy-loop ci-demo       # write a CI evidence bundle from a triage
entropy-loop demo          # run the loop: verify → trace → learn → retry → regress
entropy-loop doctor        # health-check the install

memory-demo shows how repeated failure lessons can be compacted with a deterministic MemoryPolicy — see docs/memory-policy.md.

Run a regression pack in CI

Turn captured failures into a portable pack and gate your build on it:

entropy-loop run-pack examples/json_agent_guard.pack.json

run-pack exits non-zero when a known agent regression reappears (0 = pass, 1 = failure, 2 = bad input), making replayable failure checks usable in CI. To gate on your agent's current output, refresh the pack from an explicit local command first (no shell, no secret injection):

entropy-loop refresh-pack input.pack.json output.pack.json -- python3 my_agent.py
entropy-loop run-pack output.pack.json

See docs/regression-packs.md, docs/agent-adapters.md, and docs/github-actions.md.

Explain what changed

Don't just fail CI — diff the current run against a baseline and fail only on newly introduced regressions:

entropy-loop compare-reports reports/baseline.json reports/current.json \
  --markdown-report reports/triage.md \
  --fail-on new-failures

compare-reports classifies each case as newly failing, fixed, persistent, or missing, and exits 1 only when the policy trips (0 = pass, 1 = policy fails, 2 = bad input). See docs/regression-triage.md.

Use it in GitHub Actions

- name: Compare Entropy Loop reports
  uses: koreaelonmusk/entropy-loop-core@v0.9.0
  with:
    baseline-report: baselines/entropy-loop.json
    current-report: reports/current.json
    fail-on: new-failures
    evidence-dir: reports/entropy-loop-evidence
    junit-report: reports/entropy-loop-junit.xml
    write-step-summary: true

This writes a local CI evidence bundle and can append a summary to the GitHub Actions step summary. The optional junit-report emits a deterministic JUnit XML file that GitHub Actions, GitLab CI, Jenkins, CircleCI, and other test reporters can consume. It does not call the GitHub API, comment on PRs, upload artifacts, or require GITHUB_TOKEN. See docs/ci-evidence.md.

When pinned to a semver tag (e.g. @v0.8.0) with no package-version, the Action installs the matching PyPI version (entropy-loop-core==0.8.0). On a branch ref like main it installs the latest; set package-version for reproducible CI.

What it is / what it is not

It is

• a deterministic failure compiler,
• a structured failure-trace layer,
• a regression replay primitive,
• a small AI-agent reliability tool.

It is not

• a full agent framework,
• model training,
• model-as-judge by default,
• a correctness guarantee,
• a cloud platform.

Architecture

• Verifier applies ordered, deterministic rules and classifies failures.
• EntropyLoop runs an agent, verifies, traces the failure, compiles a lesson, and retries.
• LessonGenerator turns a FailureTrace into a reusable Lesson.
• generate_regression_case pins a failure as a repeatable check.
• RegressionRunner replays a RegressionSuite and returns a report.

Deeper reading: architecture · reliability model · research influences · recording the demo.

Public / private boundary

Open-source the primitive, not the private advantage.

This repository contains only generic reliability primitives — no business logic, proprietary prompts, customer data, secrets, external AI API calls, or network calls. See docs/public-private-boundary.md.

Releases

• v0.9.0 — CI-native reporter outputs (JUnit XML) (current)
• v0.8.1 — Action runner hardening (self-test)
• v0.8.0 — GitHub Action / CI evidence bundle
• v0.7.0 — regression triage / baseline diff
• v0.6.0 — agent adapter / live pack refresh
• v0.5.0 — regression packs / CI gate
• v0.4.0 — memory policy / lesson compaction
• v0.3.1 — packaging readiness
• v0.3.0 — replay
• v0.2.0 — classification + evaluation
• v0.1.0 — the first Failure Compiler loop

Details in CHANGELOG.md.

Roadmap

• Next (directional) — persistence adapters, richer reports, and broader failure-memory recall.

Full plan in docs/roadmap.md.

Contributing

Contributions are welcome. Keep the core small, readable, and deterministic, and respect the public/private boundary. See CONTRIBUTING.md and CODE_OF_CONDUCT.md.

ruff check .    # lint
ruff format .   # format
pytest          # tests

License

Released under the Apache-2.0 license.