Bello

An autonomous coding loop with independent verification.
You write the task in a file and leave. Bello keeps the coder inside a disposable sandbox while a separate supervisor, running on a fresh context, reviews risky actions, catches drift, and handles recovery. On ProgramBench, averaged over nine matched runs with GPT-5.5 and GPT-5.6 Sol, the highest-effort setup raised completion by 36.4% relative to Raw Codex.

Contents

• TL;DR
• Motivation
• How Bello solves tasks
• Relationship to existing LLM research
• Choose your supervision depth
• Results
• Requirements
• Install
• Quick start
• Configuration
• Command reference
• License

TL;DR

You hand a coding agent a long task, close the laptop, and come back to a confident report that the work is done. Often it is not. On the benchmark tasks in Results, Codex on its own averaged a completion score of 44.87% across nine runs, which means more than half of what those tasks asked for was missing from the code it finished with. Now and then an agent also does something you cannot undo, such as dropping a database.

Bello runs the Codex coder for you and does not take its word for the result. The coder works inside a disposable sandbox. A second model watches the run from the outside, blocks dangerous commands, and pulls the coder back when it drifts away from the task.

The deeper settings add two more roles, which you turn on in Configuration. An independent reviewer reads the task, the code, and the diff, runs its own checks, and keeps sending the work back until the result holds up. An attacker then goes after the finished code without seeing how it was written, looking for cases nobody tested, such as invalid input or two features breaking each other. Confirmed findings go back to the coder. Choose your supervision depth covers the three settings we recommend and how to build your own.

Set Bello up as a Codex plugin or as a standalone command in Install, then hand it a task file the way Quick start shows. Bello runs until the work is finished and writes .supervisor/FINAL_REPORT.md with the status, the changed files, the checks that ran, and the risks that are left.

The cheapest setting costs about what Codex costs on its own, and the deepest scored 36.4% higher than Codex on the benchmark runs.

Motivation

Modern language models can write code, analyze documents, and solve hard problems, but a model still produces its answer one step at a time. On a long, multi-stage task, the same model has to hold the requirements, plan the work, carry it out, judge its own progress, notice its own mistakes, and decide when the result can be called finished.

Doing all of that inside one model is unreliable. As the context grows, model quality drops quickly and hallucination becomes more likely [1] [2] [3] [4]. Compressing the history reduces the context problem, but compression can drop a rule, a decision, or a prohibition that still applies. A confident report from the model is also not evidence that the task was actually completed.

Bello moves the orchestration, the state, and the control of complex work outside the language model.

The coder still plans its own work and derives the requirements from the task, because a language model is good at exactly that. What Bello keeps outside the model is everything around it: which role runs when, what state survives a restart, what the coder is allowed to do, and who decides that the work is finished. With completion review and the adversary enabled, Bello runs a repeatable loop in which a solution is written, reviewed, attacked, corrected, and accepted only after an independent check confirms it.

How Bello solves tasks

The run starts by building the first complete solution. The coder in the isolated sandbox reads the task, modifies the project, runs checks, and produces a working prototype. While the coder works, a runtime supervisor with a fresh context watches the execution, blocks risky actions before they happen, and steers the coder back on track when it detects drift, repeated mistakes, or unsafe behavior. The supervisor can also deny an action or restart a failing generation, and a restart keeps the coder's current workspace, so the run stays autonomous and still under control.

When completion review is enabled and a review opportunity remains, the result then goes to an independent completion review. The reviewer does not continue development, and it does not accept the coder's report as evidence. It reconstructs the mandatory requirements of the task on its own and checks:

• whether the required behavior has been implemented;
• whether the checks support the claimed result;
• whether any modes or edge cases remain untested;
• whether any regressions have been introduced;
• whether fresh validation was performed after the latest substantial changes.

When the reviewer finds a problem, the work goes back to the coder. After the fix, the reviewer runs a full review again while the budget still holds a review opportunity, because a local change can affect other parts of the system. Once the budget is spent, Bello moves on to the adversary, or finishes the run when the adversary is off.

Bello starts the adversary when the reviewer accepts the result, or when the review budget before the attack runs out. With max-reviews-before-adversary set to 0 the adversary runs on the first solution, without any review before it. The adversary tries to break the result. It explores invalid inputs, unexpected action sequences, interactions between features, boundary states, and assumptions that the coder and the reviewer may have overlooked.

The adversary works without the development history of the solution. It judges the final artifact rather than the author's explanation. Its report goes to a separate report controller, which checks every finding, keeps the confirmed ones, rejects the incorrect ones, and downgrades the doubtful ones to observations. The coder then receives the surviving findings together with all observations.

If a run ends unexpectedly after the coder has started working, for example because of a usage limit, a provider error, or an interrupted process, Bello preserves the coder's current workspace under .supervisor/, including changes that were never validated. To keep that recovery state available on the next run, leave Start over disabled (start-over: false, the default). If a security policy interrupted the run, restart it with --start-over=false. Enabling Start over discards previous recovery data.

With every stage enabled, Bello therefore implements the following cycle:

build a solution → independently review completeness → fix defects → perform adversarial testing → reassess → accept the result.

Relationship to existing LLM research

Bello separates iterative repair from acceptance. Is Self-Repair a Silver Bullet for Code Generation? found that cost-adjusted self-repair gains were often modest, variable, or absent, and that they increased substantially when feedback came from a stronger model or a human. CRITIC provides the complementary result that correction is more reliable when it is grounded in observable feedback from external tools. Bello therefore lets the coder execute tests and repair the artifact, but does not let the authoring trajectory certify completion. Acceptance is decided by a fresh reviewer that does not modify the artifact and does not treat the coder's report as evidence. The reviewer reads the specification, the artifact, and the diff, and it obtains its own behavioral evidence by selectively rerunning checks against the result. The diff makes that evidence harder to stage, because weakened assertions, skipped cases, substituted mocks, and deleted tests all appear as changes even when the suite reports green. StackEval found that reference answers consistently improved LLM code-judging accuracy, and it detected no statistically significant self-preference when such references were supplied. The finding supports review anchored in evidence, although StackEval's one-shot setting does not establish that a fresh reviewer is an independent correctness oracle. In Bello, the use of a fresh context separates the acceptance decision from the coder's trajectory, and validation remains necessary.

The adversarial stage addresses weaknesses in both fixed and model-generated tests. EvalPlus showed that the original HumanEval suites accepted substantial amounts of functionally incorrect code. Revisit Self-Debugging with Self-Generated Tests for Code Generation found that self-generated tests can produce biased and misleading repair signals. Taken together, the studies above and the 2026 preprint AdverMCTS provide the closest evidence for Bello's attacker role. In AdverMCTS, targeted corner cases reduced pseudo-correctness caused by sparse static tests, in a setting of programming problems. Bello accordingly separates implementation, counterexample generation, and acceptance. The adversary searches beyond the existing suite, but its tests are candidate evidence rather than ground truth. A separate report controller checks each finding and drops the ones it cannot confirm before the coder sees the report, acceptance stays with the completion reviewer whenever review rounds are scheduled after the attack, and relevant edits invalidate earlier acceptance evidence.

The AgentCoder preprint is the closest prior architecture. It separates a programmer, an implementation-independent test designer, and a test executor, and its ablations support separating test construction from code generation. Its evaluation is limited to function-level synthesis, and it treats a task as complete when the generated tests pass. It therefore supports Bello's role separation without covering long-running runtime supervision, a separate completion gate, a separate handler for adversary findings, or restart state. The additional controls in Bello target failures identified by MAST across more than 1,600 multi-agent traces, including role violations, history loss, task derailment, premature termination, and absent or incorrect verification. Bello maps them to fixed role contracts, durable handoffs, live drift detection, explicit stage transitions, and a separate final acceptance decision. Multi-agent specialization is prior art, and Bello's architectural claim concerns the governance and evidence requirements imposed around the roles.

Finally, CaMeL demonstrates a prompt-injection defense in which trusted control flow and security policy are enforced by a protective system layer rather than delegated to model compliance. Bello applies the same principle through isolated execution, mediated actions, live runtime supervision, and fail-closed approvals, without claiming CaMeL's capability model or formal guarantees.

Choose your supervision depth

Bello can be used as a light safety layer or as a full quality pipeline. In the effort levels below, C is an independent completion review and A is an adversarial pass. Runtime supervision stays active at every effort level.

runtime-only, for everyday work

The coder works as usual while a supervisor with a clean context watches the live trajectory. The supervisor stops abrupt, irreversible actions, such as dropping a database or cancelling a paid subscription, and it redirects a coder that has drifted away from the task. On average it matches Raw Codex on both time and cost, and on individual tasks it is sometimes faster and cheaper, because a coder that is kept on track does less useless work.

Use runtime-only as the default for any task. It removes most of the risk that the coder starts hallucinating and doing damage, and the quality gain is largest when the task is written the way people normally write tasks at work: long, messy, and full of requirements added in passing. On our three custom tasks of that kind, runtime-only scored about 9% higher than Raw Codex, because it catches drift and hallucination early. The ProgramBench tasks are short, so they understate the effect, and there the mean gain was about 2%.

C+A, for heavy overnight work

This effort level adds up to one independent completion-review round followed by an adversarial attempt to break the result, and it keeps every runtime-only protection. On ProgramBench it raised macro completion from 53.53% to 67.67%, which is 69% of the improvement delivered by the full 4C+A+2C setup in our shorter-run comparison. The gain costs time and money. The three-task run took about 2.5 times longer than Raw Codex, and it cost roughly 1.8 to 2.3 times more.

The absolute numbers stay small. One ProgramBench task consumed about 0.3% to 0.4% of a weekly Codex limit under Raw Codex, and up to 1.2% under C+A, so a weekly quota still covers dozens of runs.

Use C+A when you want to hand over a hard task at the end of the day and need serious quality with a real review behind it. The run finishes overnight, and the result is clearly better than what Raw Codex produces on the same task.

4C+A+2C, for maximum quality

This effort level allows up to four completion-review rounds to refine the implementation before the adversary probes its assumptions, and up to two further rounds to resolve what the attack uncovers. We built it to see how high Bello can score on a benchmark with every stage enabled, and the measured completion was the highest of the three effort levels. Bello scored higher in all nine matched runs. Averaged over them it improved completion by 36.4% over Raw Codex, and in the GPT-5.6 Sol ultra comparison by 38.30%.

This effort level is significantly more expensive than Raw Codex and takes much longer, so it is worth choosing deliberately. It fits a genuinely awkward task with many cases and nuances, where quality is the priority and cost is not a constraint. Plan for a long run, because in our ultra runs a single task took between 7 hours 39 minutes and 19 hours 25 minutes.

Configure these effort levels with bello config. For runtime-only, set completion-review and adversary to false. For C+A, enable both and set max-reviews-before-adversary, max-adversary-runs, and max-reviews-after-adversary to 1, 1, and 0. For 4C+A+2C, use 4, 1, and 2. The fields, defaults, and one-run CLI overrides are documented in the Configuration section.

Custom effort levels

Bello has no built-in list of modes to pick from. The three effort levels above are configuration recipes, and the budget fields are independent numbers, so you can choose whatever effort level your task needs: a single C with the adversary off, 2C+A, C+A+C, 4C+2A, and so on. Set max-reviews-before-adversary for the review rounds before the first attack, max-adversary-runs for the number of adversary passes, and max-reviews-after-adversary for the review rounds that follow each pass. Both review budgets also accept Unlimited, which removes the cap and lets the loop keep going until the reviewer accepts the result.

The numbers are upper limits rather than a fixed sequence. A run can end before it uses them, because the reviewer can accept early and the run completes once nothing further is scheduled, so a name like 4C+2A describes the most the run may do rather than what it will do. The adversary also requires completion-review to be enabled, and setting max-reviews-before-adversary to 0 is allowed, which sends the first solution straight to the attack.

Results

1. runtime-only, low-cost protection

On the three ProgramBench tasks, Solar, Samtools, and Rumdl, runtime-only improved average completion by approximately 2% over Raw Codex. The larger benefit is risk control. A fresh supervisor can catch a dangerous action or a bad trajectory before it becomes an unrecoverable final result, without the cost of scheduled completion-review and adversary rounds.

We also tested runtime-only on three custom tasks designed to resemble ordinary work rather than polished benchmark prompts. Their briefs are deliberately incomplete, awkward, and uneven, the way a task is often described by a normal colleague at work.

Figure R1. Comparable 0 to 100 evaluator scores for Marl, Slab, and Pinch. The scores are separate task-specific measures, not components of a pooled benchmark.

The linked folder contains the complete task briefs, tests, evaluator outputs, and result artifacts.

2. C+A, a shorter balance of quality and cost

With GPT-5.6 Sol at ultra, C+A raised the unweighted macro completion score from 53.53% to 67.67%: +14.14 percentage points (+26.41% relative). The three-task runtime was 07:08:06, compared with 02:48:55 for Raw Codex. The corresponding rows are available in the C+A run-level data. The corresponding Bello solutions are available in the C+A solution artifacts folder.

Figure C1. ProgramBench completion and runtime for the three matched GPT-5.6 Sol ultra task configurations.

3. 4C+A+2C, maximum effort

Key findings

• Across all three tasks and all model and effort settings, Bello achieved the higher completion score in 9 of 9 matched configurations. The overall unweighted mean increased from 44.87% to 61.21%: +16.33 percentage points (+36.40% relative).
• With GPT-5.6 Sol, Bello achieved the higher completion score in 6 of 6 matched configurations. The unweighted mean increased from 48.92% to 67.04%: +18.13 percentage points (+37.06% relative).
• In the complete GPT-5.6 Sol ultra comparison, every task improved by 18.17 to 24.59 points, and the macro average increased from 53.53% to 74.03%.
• With GPT-5.5 xhigh, Bello scored higher on all three tasks, and the macro average increased from 36.79% to 49.53%: +12.74 percentage points (+34.64% relative).

Evaluation protocol

We evaluated Bello on three ProgramBench tasks: Solar, Samtools, and Rumdl. Raw Codex and Bello were observed on every task with GPT-5.6 Sol in both ultra and xhigh modes and with GPT-5.5 in xhigh mode. We report the completion score recorded in the completion_pct field and time from the runtime field of the run-level data. Completion scores are rounded to the nearest hundredth of a percentage point. Runtime was not held constant, so the comparison is not compute matched. The final solution patches for all nine reported Bello runs, together with SHA-256 checksums, are available in the public evaluation artifacts folder.

GPT-5.6 Sol

ultra

Bold completion values indicate the higher observed score within each matched row.

Across the three matched ultra runs, Bello increased completion by 18.17 to 24.59 percentage points on every task. The unweighted macro average rose from 53.53% to 74.03%, a gain of 20.50 points (38.30% relative).

Figure 1a. Bello-minus-Raw completion differences for the three GPT-5.6 Sol ultra configurations. Every point lies to the right of zero, and the diamond shows the unweighted mean difference (+20.50 points). Uncertainty intervals are not shown because each configuration has one observation.

xhigh

Bold completion values indicate the higher observed score within each matched row.

All three xhigh tasks improved. The gains ranged from 13.55 to 19.89 percentage points, and the unweighted macro average increased from 44.30% to 60.06% (+15.75 points, +35.56% relative).

Figure 1b. Bello-minus-Raw completion differences for the three GPT-5.6 Sol xhigh configurations. Every point lies to the right of zero, and the diamond shows the unweighted mean difference (+15.75 points). Uncertainty intervals are not shown because each configuration has one observation.

GPT-5.5

xhigh

Bold completion values indicate the higher observed score within each matched row.

Bello's score was higher on all three tasks. The task-level differences ranged from 4.69 to 23.93 percentage points, and the unweighted macro average increased from 36.79% to 49.53%, a gain of 12.74 points (34.64% relative).

Cross-task completion summary

Figure 2. Cross-task completion summary on a common 0% to 100% scale. Panels (a), (b), and (c) show the matched GPT-5.6 Sol ultra, GPT-5.6 Sol xhigh, and GPT-5.5 xhigh comparisons. The unweighted macro differences are +20.50, +15.75, and +12.74 percentage points, respectively.

Task-level configuration profiles

The following panels compare all three complete three-task configurations: GPT-5.5 xhigh, GPT-5.6 Sol xhigh, and GPT-5.6 Sol ultra. Each panel contains exactly six bars, one for Raw Codex and one for Bello in each model and effort setting, ordered by increasing completion score. Bello precedes Raw Codex when scores are tied. Ordering is descriptive and does not imply compute equivalence.

Figure 3a. Solar completion scores for the six configurations, sorted from lowest to highest. The two formerly tied values are shown at their available precision: Codex GPT-5.6 Sol ultra at 53.13% and Bello GPT-5.5 xhigh at 53.39%.

Figure 3b. Samtools completion scores for the six configurations, sorted from lowest to highest.

Figure 3c. Rumdl completion scores for the six configurations, sorted from lowest to highest.

Requirements

• Codex CLI installed and authenticated. Bello drives codex app-server, and your Codex account provides the models.
• Python 3.11+ and git.
• macOS or Linux.

Verify your environment at any time with bello doctor.

Install

Option A: Codex plugin (recommended if you work inside Codex):

pipx install bello
codex plugin marketplace add AlexeyKulaev/Bello-codex-marketplace --ref main
codex plugin add bello@bello-marketplace

Then open Codex in your project folder and ask it to run Bello on your task file. The plugin checks for updates and launches the run for you.

Option B: standalone CLI

pipx install bello
bello doctor

Bello checks for updates at startup and offers to install them. Run bello update to update explicitly.

Quick start

cd your-project
echo "Build a CLI tool that ..." > task.md
bello --task task.md

Bello starts the coder, supervises the run, and writes .supervisor/FINAL_REPORT.md when it finishes. The report lists the status, the changed files, the validations that were run, and the remaining risks.

While a run is active you can type into the terminal, and your message is routed to the supervisor rather than the coder:

Everything the run does is written to inspectable files under .supervisor/ in your project: PROGRESS.md (what has happened), DECISIONS.md (standing decisions), HANDOFF.md (restart context), events.jsonl (full event stream), and FINAL_REPORT.md (the result).

Configuration

Open the interactive editor from your project folder:

bello config

It creates and edits .supervisor/config.json. Every value is saved as you press Enter, and future runs in this folder use these settings automatically.

For a new project the editor starts with completion-review and adversary turned off, which is the runtime-only setup, and it only shows settings that can affect the selected pipeline. Turning on completion-review reveals the completion reviewer and review budget. Turning on adversary then reveals the adversary model and the complete C+A setup.

For each visible role, select GPT-5.6 and then choose Sol, Terra, or Luna in the variant row. Sol and Terra support reasoning effort from low through ultra, and Luna supports low through max. Active primary roles default to GPT-5.6 Sol at xhigh, and cheap runtime triage uses Luna.

CLI flags override their corresponding saved settings for one run and never rewrite the project config. Settings without a CLI flag, including cheap runtime and review budgets, are changed through bello config.

Command reference

bello                 # run the configured task in the current folder
bello --task TASK.md  # run a specific task file
bello config          # open the interactive config editor
bello doctor          # check Python, git, Codex, auth, app-server support
bello update          # update Bello to the latest version
bello update --check --json  # machine-readable update status
bello --version       # installed version, latest version, update status

Run flags (each overrides the saved config for one run):

Environment variables: BELLO_SKIP_UPDATE_CHECK=1 skips the startup update check, BELLO_PROMPTS_FILE=/path/to/prompts.toml points Bello at an alternative prompt file for experiments, and BELLO_CONFIG_ANIMATIONS=0 disables motion in the interactive config editor.

License

Bello is released under the MIT License. See LICENSE.

Contributions require signing the project CLA. A bot will prompt you on your first pull request, and you only sign once.