keeploop 0.3.0

English | Tiếng Việt | 简体中文

Your AI proposes the change. A number decides if it stays.

Quick start · Examples · How it works · FAQ · Full guide

_{The session lines are real output of kl next / kl try on the offline demo (fastpath.txt); the speed numbers are measured (docs/proof/bench-latency-*.json). Video for X: keeploop-x.mp4. Guides: AI chat · AI code.}

keeploop turns the AI you already use into a small team, with a CEO that works beside you.

• Easy. Paste one file into a chat app, or one prompt into your coding agent. You talk in plain words; you never learn commands.
• It asks first, then sizes and staffs the team. The CEO (your AI) asks what "better" means, how to measure it, what score is good enough and what a wrong change would cost you. A draft gets a light loop of a few cycles; work people rely on gets the rigorous one (kl level). Then it names a team for the job: a cheap, fast model proposes, your main model builds, the strongest model checks the real diff in a fresh context. Mistakes become lessons it does not repeat, at every level.
• Fast. Two commands a cycle in a coding agent (kl next, kl try); one paste each way for a chat AI that cannot see your repo.
• Saves tokens. With profile: fast the Engineer writes its own card and the ideas you listed need no Analyst call: −33% tokens and −26% time per cycle in our offline benchmark.
• Keeps only real gains. A change is committed only if it beats the best score by more than the evaluation's own measured noise (best + 2σ). Everything else is reset, and the reason is written down (measured).

Works with ChatGPT, Gemini, Claude, Perplexity, Claude Code, Codex, Cursor, Copilot and Gemini CLI. keeploop itself is not an AI: it is a small Python tool that measures and refuses, with no dependencies, no telemetry and no API keys.

Only have a chat app (ChatGPT, Claude, Gemini, no code)? Use keeploop-lite: one file you paste into the chat, the same keep-or-discard loop for a post, a CV or a plan (Tiếng Việt, 简体中文, example). It is fast by default: say run 3 cycles to try up to five changes in one reply, and save to get the ledger to keep. Short on space (a free plan, Perplexity)? Use KEEPLOOP-LITE-MINI.md. Save either file once as standing instructions (a ChatGPT Project or custom GPT, a Gemini Gem, a Perplexity Space, a Claude Project) instead of pasting it into every chat. Your AI can run code? Use the full version below.

_{Step by step in a chat app. Full-size video (1080p): the guide.}

Quick start

(a) Hand it to your AI. If it can run commands, paste this and fill in the last line:

Clone https://github.com/johnha912/keeploop next to my project (not inside it),
run `python prepare.py` in the clone, then read its AGENTS.md and act as the CEO
it describes. What I want better: <for example "make my test suite faster">

You never type kl and there are no flags to learn. The first time, your AI asks in plain words what "better" means and how to measure it, then proposes an evaluate.py (a small script that prints that number) for you to approve before anything changes. In Claude Code, /keeploop 3 inside the clone runs three cycles.

_{Step by step with a coding AI. Full-size video (1080p): the guide.}

(b) Try it offline. Needs Python 3.10+ and git; no AI, no API key, a few seconds.

git clone https://github.com/johnha912/keeploop.git && cd keeploop
python prepare.py
python run.py --mock --iterations 5

cycle 3: discard  metric=0.942  best=0.992000  tokens=2820
cycle 4: crash  metric=None  best=0.992000  tokens=3510  lesson=L-001
run complete: 3 keeps, 1 discards, 1 crashes, 1 lessons, 17259 tokens. Best metric: 1.000000

The mock is a scripted stand-in for a real AI. It shows how the loop works, not how good a real model is. Every keep is a real commit: git -C .keeploop/workspace log --oneline.

(c) Install it as a command. kl without a clone, from PyPI:

pipx install keeploop      # or: pip install keeploop (in a venv of its own)
mkdir my-loop && cd my-loop
kl init                    # copies AGENTS.md, program.md, team.yaml, packs/, .claude/ here
kl begin                   # the demo baseline

kl init puts the files you edit in your own folder and never overwrites one. Then open the folder in your AI as in (a), and follow AGENTS.md to point it at your repo. kl run is the headless mode. Keep the clone if you want to read or change keeploop itself.

One command from your own repo instead, with an evaluate.py written for you:

cd my-repo
kl quickstart tests        # or: command | prompt | buildtime (writes ../my-repo-keeploop)
cd ../my-repo-keeploop
kl doctor                  # runs evaluate.py 5 times: how noisy is it, what gain can it see?
kl begin

Started in a chat with keeploop-lite? kl import-lite ledger.md continues the session in the CLI. Its best version becomes the workspace, and its scorecard becomes an evaluate.py.

(d) Share a run. kl report --html run.html turns the ledger into one page, with no model call. The page has a summary line and a chart: kept changes filled, discarded hollow, the running best and its noise band. Example: the proof run's chart.

Examples

Set up any example as a sandbox with kl init --example <name> <new short folder> (installed kl) or python tools/make_sandbox.py examples/<name> <new short folder> (clone) (details).

Why trust the keeps

keeploop is the first installable tool found in our scan of autoresearch-style loops (November 2026) that turns evaluation noise into a commit law: a change is kept only if it beats the best score by more than the evaluation's own measured noise (best + 2σ), with the falsification rule fixed in advance and a fresh-context Auditor reading the real diff. Evaluation noise is not our discovery; research has discussed it before. What is new is a tool that turns it into the rule that decides every commit.

What that buys, measured. We ran a naive keep-if-better loop and keeploop on the same 20 proposals and the same noisy evaluation, a random 1,000 of 10,000 dev cases per call. A 5,000-case holdout scored the result, and the loop never saw it. Each was run on 25 seeds fixed before the run (docs/proof/PROOF.md, python tools/proof_run.py):

The price is real. keeploop throws away some true gains that land inside the noise. On 9 of the 25 seeds its holdout ended lower than the naive loop's, including seed 1, the run fixed before the sweep as the headline. kl doctor measures your evaluation's noise before you start, and tells you when to reduce it.

Why keeploop

An agent saying "done" is not a measurement.

Also in the guide: lessons so a mistake is not repeated, a trend scout, and one AGENTS.md that Codex, Cursor, Copilot and others read.

How it works

Built on Andrej Karpathy's autoresearch idea: a fixed metric, a keep-or-discard rule, and git as the memory of what worked.

flowchart LR
    A[Analyst<br/>writes one card] --> E[Engineer<br/>makes the change]
    E --> G{Code gates<br/>fence · protected<br/>secrets · size}
    G -- pass --> V[evaluate.py<br/>one number]
    V --> U[Auditor<br/>fresh eyes,<br/>real diff]
    U --> C{beats best<br/>+ noise?}
    C -- yes --> K[git commit]
    C -- no --> D[git reset]
    G -- fail --> L[(lesson)]
    D -. why .-> L
    L -. read next time .-> A

An Analyst writes one card, an Engineer makes exactly that change, code gates and evaluate.py check it, and a fresh Auditor reads the diff. Only kl keep commits. The full cycle, the critic round, lessons, the trend scout and speed tuning are in the guide.

Your own work: packs

A pack gives the team the craft rules and review checklist for one field.

New field? python kl.py pack new NAME scaffolds one. Setup for your own repo, team.yaml, vendors and the headless overnight mode (python run.py) are in docs/GUIDE.md. Why the experimental ones are experimental: guide.

Safety

• keeploop never executes text from a model or a web page. Only the Engineer gets edit rights; changed files are scanned for credentials, and token-like environment variables are removed before any vendor CLI or evaluation starts.
• While code you did not write runs, every file in the project folder outside the workspace is content-hashed. Any change stops the run.
• The fence detects changes; it is not a sandbox. Run real vendors in a container or VM if that matters to you.

Details: docs/GUIDE.md and SECURITY.md.

FAQ

More questions: docs/GUIDE.md.

Docs

Credits

keeploop stands on autoresearch by Andrej Karpathy: a fixed evaluation the agent cannot touch, an append-only results.tsv, a keep/discard rule and git as memory. keeploop turns it into an org chart any AI can run, and adds roles, cross-vendor audit, code gates, lessons and a trend scout. No code from autoresearch is copied. If you like this repo, star his first.

The banner is set in Google Sans Flex.

If keeploop stopped a bad change from landing, a ⭐ helps other people find it. Bugs and ideas: open an issue.

License

MIT.