qikly

The problem: Your AI writes both the code and its tests. How do you know the tests are really valid?

The solution: two agents. One turns the acceptance criteria into tests. The other writes the code and never sees the acceptance criteria.

Who it is for: a developer or team pointing an AI coding agent at a self-contained Python module that transforms data, for example an ETL step, a merge, a calculation or a validation routine, who does not want to trust a green suite when the same agent wrote both the code and the tests. It suits one module at a time: when a test fails, only the files that failure names are loaded, so runs stay small and quick. Large multi-file repositories are a different problem. See What it is for.

Imagine a student who writes the exam paper, writes the answer key, and then sits the exam. They pass. Obviously they pass, and nobody would accept that as evidence the student knows the material.

That is what happens when one model is given a specification containing the acceptance criteria and asked to produce both the implementation and the suite that checks it. It writes tests its own code will pass. Everything goes green, and the green means nothing.

qikly takes the answer key away from the student.

It generates a test suite from the acceptance criteria in a specification, then writes an implementation and repairs it against that suite until every test passes or a retry budget runs out. The agent doing the writing and repairing never sees the criteria. Every failure, every piece of reasoning and every diff is recorded.

The part that makes the result mean something: the coding agent never sees acceptance_criteria. It gets the specification with that section stripped out, the same vague brief a developer works from, while test generation gets it in full. When a test fails, the agent sees the failure message and never the rule it broke. Without that asymmetry both sides read the same spec identically and every test passes first try, which proves nothing.

flowchart TD
    SPEC["<b>Full specification</b><br/>task.yaml<br/>requirements + interface<br/>acceptance_criteria"]
    REQ["requirements<br/>+ interface"]
    AC["acceptance_criteria"]
    CODE["<b>Coding agent</b><br/>writes the implementation<br/>FIX then PATCH on failure"]
    TEST["<b>Test-writing agent</b><br/>writes the suite"]
    IMPL["Implementation"]
    SUITE["<b>pytest suite</b><br/>Tests for:<br/>1 integration, 2 system,<br/>then 3 unit"]
    RUN{"Run the suite"}
    FAIL["<b>Failure errors</b> only<br/>no criteria, no test source"]
    OUT["Converged<br/><b>outputs:</b> code + suite<br/>+ audit trail"]
    STALL["Did not converge<br/><b>failure errors and audit trail</b><br/>exits non-zero, ships nothing"]

    SPEC --> REQ
    SPEC --> AC
    AC -. "never reaches" .-x CODE
    REQ --> CODE
    REQ --> TEST
    AC --> TEST
    CODE --> IMPL
    TEST --> SUITE
    IMPL --> RUN
    SUITE --> RUN
    RUN -- pass --> OUT
    RUN -- fail --> FAIL
    FAIL -- "repair loop:<br/>FIX, then PATCH" --> CODE
    FAIL -- "retry budget spent" --> STALL
    IMPL -. "unit stage only:<br/>written last, from the code" .-> TEST

    classDef codeView fill:#f3e8ff,stroke:#7e22ce,color:#4c1d95
    classDef standardView fill:#d9ebea,stroke:#0e6a70,color:#0b3d40
    classDef converged fill:#dcfce7,stroke:#15803d,color:#14532d
    classDef stalled fill:#fdf0d5,stroke:#b45309,color:#78350f
    class CODE,IMPL,FAIL codeView
    class AC,TEST,SUITE standardView
    class OUT converged
    class STALL stalled
    linkStyle 10 stroke:#15803d,stroke-width:2px
    linkStyle 11,12 stroke:#7e22ce,stroke-width:2px
    linkStyle 13 stroke:#b45309,stroke-width:2px

Purple is what the coding agent can see. Teal is what the standard is written from. They never touch. A run that never converges is still worth having: it exits non-zero, names the blocking tests, and keeps the same complete record. The purple arrows are the repair loop, and that is where almost all of a run happens: a failing suite sends the agent the failure text and nothing else, it produces a FIX and a PATCH, and the suite runs again, until the stage passes or the retry budget runs out. It never sees the rule it broke, so it cannot write code shaped to a criterion it was shown. tests/test_withholding.py fails the build if any call site lets one through.

A run works through three stages, integration then system then unit:

1. Integration and system tests are generated first, from the spec alone, before any code exists. They cannot see an implementation because there is not one yet.
2. The coding agent writes the implementation, from the spec minus the criteria.
3. pytest runs. On failure the model produces a FIX (failure summary, root cause, plan, and the files it intends to touch) and then a PATCH (a unified diff of only those files), applied all or nothing. Repeat until the stage passes or the budget is spent.
4. Unit tests are generated last, once real code exists for them to name. This is the only stage allowed to see the implementation.
5. Clearing a stage re-runs the earlier ones, so a later fix cannot silently break something that already passed.

Every arrow back into FIX carries the pytest error text and nothing else. Unit tests come last because they are the only ones that need to name real functions, which makes them the only stage allowed to read the implementation. Re-running the earlier stages after each success is what stops a later repair quietly breaking something that already passed.

Two ways to get a test suite, and what each can prove

Both are useful and they answer different questions. qikly is not purely one or the other: integration and system tests are written from the criteria before any code exists, the unit stage is written last from the code that just passed them, and --refine-criteria reads a converged implementation to propose criteria the first draft missed. Each of those reads the code on purpose, and none of them can question it.

What makes this different

The tests come from the standard, not from the code. This is the one that matters most. Every other AI test generator in this space writes its tests from an implementation that already exists, so it can only describe what the code already does. That makes an excellent regression harness, and it cannot tell you the code is wrong. qikly writes the integration and system suites from the acceptance criteria before any implementation exists, so the standard cannot have been shaped by the thing it judges.

The withholding is a mechanism you can watch. Not a prompt asking a model to ignore a section, and not a convention someone has to remember. One command prints what each side is given and the difference between them, offline and free:

qikly --explain <MY_TASK>     # e.g. qikly --explain MERGE_SALES

Eleven criteria go to test generation. Twelve lines are removed before the coding agent sees the same file. tests/test_withholding.py fails the build if any call site ever lets one through, including one added next year by someone who has never read this. It is a property of the code, and it takes thirty seconds to check.

What you get is an executable suite you keep. The output is pytest files and JUnit XML. Read them, run them, put them in CI, and when one fails in six months it fails for a reason you can inspect and argue with. A suite is a durable asset in a way a model's verdict is not: a verdict cannot be re-run against tomorrow's commit.

It helps you write the standard, not just check against it. --init and --scaffold turn existing code into a task, --criteria-from lifts criteria out of a ticket you already wrote, --generate-criteria drafts a first bar from requirements alone, and --check-criteria looks for two statements anywhere in the specification that no implementation could satisfy at once, including two acceptance criteria that disagree with each other.

Every run is reproducible, and the whole trail is kept. A run records the provider, the model, the settings and the version that produced it, next to every failing test, every FIX with its stated root cause, and every PATCH as a diff. You can read back exactly why a line of code exists: which assertion forced it, what the model concluded, and what it changed. The record survives a run that never converges, which is when you most want it.

"Why not just use two different models?"

It is the first thing most people ask, and it does help a little. It does not reach the underlying issue, because both models still read the same criteria and so both still write to them: the code is still built to satisfy the standard it is about to be judged by, and changing who types it does not change what they were shown. It is also a habit rather than a mechanism, and nothing checks the two stayed different.

The two compose nicely, incidentally, since qikly picks a provider and model per agent role. You can withhold and use two models.

"Why not just add a reviewer agent?"

The newer version of the same question, and the one worth answering carefully, because independent verification steps are now shipping in mainstream coding agents: a second agent, often from a different model family, reviews what the first one produced.

It helps, and it does not reach this. A reviewer given the same specification has read the same acceptance criteria, and resolves the same ambiguity the same way. It will catch a mistake that is visible from that context: an inconsistency, a requirement plainly skipped, an obvious bug. It cannot catch the case this tool is built for: a line that could be read two ways, read once, with both the code and the standard written from that single reading. Nobody is wrong, so nothing looks wrong.

The problem was never that nothing was checking. It is that everything checking had already seen the answer key. Withholding is what makes the check structural rather than one more opinion drawn from the same context, and it is enforced by a test rather than by an arrangement someone has to remember to keep.

There is a second difference, and it outlasts the run: a reviewer emits a verdict, and this emits a pytest suite you still have in six months.

Design rationale, and the harder problem of where acceptance_criteria comes from in the first place: docs/design_1_case_study.md, the first of three parts.

What it is for

Built for self-contained Python modules that transform data, not for a large existing repository: ETL, merges, calculations, validation. That is the layer where a wrong answer looks like a right answer, and where a test written from the rule is the only thing that catches it.

Where it does not fit today: an existing large repository. PATCH prompts load only the files a FIX names, and while a large file is now excerpted rather than loaded whole, there is no cross-file index. See Where it fits today.

How to use the tools in this project

Four ways in, and the table under Quick start says which command each one needs:

1. Verify code you did not write. Supply an implementation through seed: and the suite is written from your acceptance criteria by an agent that never reads that code. A suite generated from the same context as the code is a model agreeing with itself.

   The limit is worth saying plainly: qikly cannot know what the author of supplied code saw. Withholding is a property of a run qikly performed, not of a file you hand it. If the same person or model wrote that code with the criteria open, this gives you an independent suite, not an independent author. What it does give, always, is a suite that was not derived from the code, which is the half a code-derived generator cannot give you at all.

2. Start from a spec. No code yet: get a first implementation and the suite that justifies it, in outputs/, never in your source tree.

3. Bring your own tests. Seed any stage and the loop becomes a repair procedure rather than a generator.

4. Run a catalog unattended. Non-zero exit on any non-convergence, so a scheduler or CI job can run many specs and keep the reports.

Try it without spending anything

Three commands that make no model call, need no API key, and cost nothing.

qikly --explain MERGE_SALES   # what each side is shown, and the difference
qikly --validate              # check your task files: YAML, criteria, fixtures
qikly --explain MERGE_SALES --json

--explain is the one worth running first. It prints the acceptance criteria that test generation receives, then the same task file as the coding agent receives it, then the diff: on MERGE_SALES, eleven criteria and twelve lines removed. It builds those strings through the same function a real run uses, so it shows the mechanism rather than a description of it.

--validate reads your task files and nothing else: that they parse, that acceptance_criteria is a list rather than one long string, that fixture paths resolve, and that criteria name values instead of adjectives. It is also available as a pre-commit hook, qikly-validate, deliberately the free check rather than the paid one.

Quick start

Requires Python 3.10+ and GNU patch on PATH. On Windows it ships with Git under usr\bin\patch.exe, which the tool finds on its own. On macOS you have to install it: the system patch is Apple's BSD one, which rejects the options qikly sends, so no generated diff will apply.

brew install gpatch     # macOS only

qikly looks for gpatch before patch, so nothing else is needed afterwards and your system patch is left alone.

pip install qikly
export GEMINI_API_KEY=...     # or API_KEY, or your provider's own variable
qikly --demo

On Windows, in PowerShell, where export is not a command:

pip install qikly
$env:GEMINI_API_KEY = "..."
qikly --demo

Other providers, and how to set a key so it survives a new terminal, are in docs/PROVIDER_KEY_SETUP.md.

You probably do not have to write the task file by hand

The criteria usually exist already, in a feature page or a ticket, and the interface exists in the code. qikly reads both.

# a markdown page, a ticket export, or a .feature file
qikly --criteria-from feature.md --task-id MY_TASK

# straight from Jira: needs JIRA_BASE_URL, JIRA_EMAIL, JIRA_API_TOKEN
qikly --criteria-from-jira PROJ-412 --task-id MY_TASK

# both halves at once: criteria from the page, interface from the module
qikly --scaffold src/metrics/band.py --from-doc feature.md

Bullet lists, a headed Acceptance Criteria section and Gherkin Scenario: blocks are all understood. Your page stays the source of truth and nobody retypes anything.

One section is never filled for you: requirements. The coding agent reads it, and a feature page usually restates its own acceptance criteria in the prose above them, so lifting requirements across would hand the criteria to the one agent that must never see them. qikly --validate warns if what you write there restates a criterion.

Which command depends on which parts you already have

A task file is one YAML file with three parts, and the split above is a split between them:

1. requirements what the code must do, in the words a person would use. The coding agent reads this.
2. interface the contract, and a description rather than code: the function signatures and the dotted path where the module will live. Both agents read it, and neither is handed an implementation to read from it. When the integration and system tests are written there is not one yet.
3. acceptance_criteria what counts as correct, each one checkable and naming its boundary value. Only test generation reads this.

"Spec" below means 1 and 2 together, which is what the coding agent is given. A tick means you already have that part.

One thing the three parts do not say, and it matters: test generation never reads the implementation either. Integration and system tests are written before any code exists, from the specification alone. The unit stage is the single exception, written last from the code that just cleared the earlier stages, because unit tests have to name real functions.

<MY_TASKS> is one task_id or several separated by commas. A task_id is a filename under inputs_private/config/tasks/ without the .yaml: --tasks CALC_TAX, --tasks CALC_TAX,MERGE_SALES, or omit it to run every task found. <MY_TASK>, singular, takes exactly one.

QIKLY_MAX_CALLS=200 qikly stops at a call limit rather than a bill.

--scaffold reads the module path and the real signatures of every public function straight out of the file, because they are already there. It leaves requirements and acceptance_criteria for you, and that is deliberate: criteria derived from an implementation can only describe what that implementation already does, and a bar that agrees with the code by construction is the exact failure this tool exists to prevent.

--demo runs one task end to end in a throwaway demo/<timestamp>/ directory and prints what it built and where. It writes nothing outside that directory, so a first run leaves everything else untouched. About 30 seconds.

That is a real run on gemini-3.5-flash-lite, 38 seconds, not sped up.

From a clone instead:

pip install -r requirements.txt
python run.py --demo

run.py is a shim around src/qikly/cli.py, the same entry point the installed qikly command calls, so a clone and an install run identical code.

Running

python run.py                            # every task found
python run.py --tasks ETL_ADDRESS        # one
python run.py --tasks ETL_ADDRESS,ETL_EMAIL
python run.py --demo --tasks MERGE_STOCK # isolated, any task

Each task runs in its own process, concurrently, with console output prefixed [task_id]. Exit code is non-zero if any task did not fully converge.

Expect some runs to stall, by design. Roughly 8 runs in 10 finish with the code passing every integration and system test. Roughly 6 in 10 pass everything including unit tests. Nearly the whole gap between those two figures is the unit stage.

Those are round numbers because they were measured three times: a 427-run sweep, a 140-run sweep sixteen days later on the same tasks and settings, and a 400-run sweep after correcting the benchmark itself, when eight of the ten tasks turned out to be carrying acceptance criteria that no input row could trigger. All three landed inside each other's intervals. All three used gemini-3.5-flash-lite, a small cheap model chosen to make repeated sweeps affordable, so treat them as a floor. Three sweeps agreeing is worth more than any one of them's decimal places, so the decimal places are not quoted.

A run that exhausts its budget exits non-zero, names the tests that blocked it, and keeps the full record. It never reports success on code its own tests reject.

Ten example tasks ship with the tool across four domains, listed in docs/design_3_mechanism.md.

Measuring rather than producing. One run is an artifact, not a rate: the same task with the same seed converges on some runs and not others. To claim how often anything converges, repeat the sweep and read the interval:

python -m qikly.orchestrator.run_all --repeat 10

That writes an aggregate report with confidence intervals and groups the non-converging runs by what they got stuck on. Details in docs/design_3_mechanism.md.

When a run does not converge

A stall is a normal outcome, not a broken tool: the run exits non-zero, names the blocking tests, keeps the whole record, and ships nothing.

The first thing to try is a stronger model, which moves convergence more than any setting in this file and costs one environment variable. After that, in order of how often each is the answer: read the timeline report, look for the same patch repeating (a criterion fighting the model's priors), check for a collection error (nothing ran at all), and run --check-criteria, --validate and propose_fixtures.

Each of those, with the signature to look for and the fix: docs/TROUBLESHOOTING.md.

Output

Everything is namespaced by task_id so concurrent runs never collide:

Running it on your own data

Four steps: make the two directories, drop your fixture data in, write the task file, run it. Nothing is written into the package, and nothing is written into your source tree.

qikly --init                    # creates inputs_private/ and a starter task
qikly --tasks MY_TASK

docs/USING_YOUR_OWN_DATA.md has the rest: the task file field by field, getting the split between requirements and acceptance_criteria right (the gap between them is the whole mechanism), lifting criteria out of a ticket you already wrote, seeding your own implementation or test suites, where each file is read from, and proposing the fixture rows a criterion needs before any test can reach it.

Configuration and LLM provider

Gemini by default; OpenAI and Anthropic are supported. One provider per run.

export GEMINI_API_KEY=...
qikly --demo

docs/CONFIGURATION.md has every environment variable, the settings.yaml keys, timeouts, how much determinism each provider actually gives you, and how to notice an SDK changing under you. Keys, PowerShell, CI and what a wrong key looks like: docs/PROVIDER_KEY_SETUP.md.

Version check

On startup the CLI makes one request to pypi.org and prints a single line if a newer release exists. Set QIKLY_NO_VERSION_CHECK=1 to turn it off.

It never changes a run: every failure path is silent, the timeout is 1.5 seconds, it happens once per invocation rather than once per task, and it reads only the public PyPI and GitHub release indexes you already rely on to install software.

Where it fits today

Stated plainly, because the fit matters more than the feature list.

It targets self-contained Python modules. PATCH prompts load only the files a FIX names. A file over roughly 16,000 characters is no longer loaded whole: it is parsed, the definitions the FIX and the failure name are reproduced in full, everything else collapses to a one-line signature, and elided ranges are marked so a diff still applies. Raise the threshold with QIKLY_MAX_FILE_CHARS. What that is not is a repository story: there is no cross-file index, and excerpting reduces size rather than bounding it. A module with four hundred functions still yields four hundred signature lines, so 57k characters becomes 30k, which is smaller and still large. The retrieval is a lookup rather than a similarity search, because the FIX and the failing test already name the symbols.

GNU patch is required, not any patch. Apple and BSD ship an implementation that rejects --fuzz, which is what lets a diff with slightly wrong line numbers apply at all. qikly now tells you when it finds one, instead of reporting it as an ordinary failed hunk while every remaining attempt in the run regenerates diffs that could never land.

Patches apply without asking, by default. They land in the disposable outputs/agent_src/code/ tree with a timestamped backup, never in your working copy, and an unattended loop that stopped to ask would not be unattended. --review-patches puts a person in the path, --dry-run generates every patch and applies none. A declined patch is kept and logged, because a rejection is evidence no test can produce.

Budget exhaustion is a normal outcome, not a rare edge case. A run that cannot satisfy its own suite exits non-zero, names the blocking tests, and ships nothing. That is the property worth having. Three of the four causes are defects in the specification rather than in the model, so most stalls are fixed by editing text: see the stall taxonomy.

Two stall signatures, and they mean opposite things. The same patch repeating byte for byte is the model correctly disagreeing with an artificial rule. No retry budget fixes that: widen the criterion, or move the decision into requirements where the agent can read it as a given rather than as an error to correct. A different patch each attempt, never resolving the same test, is an ordinary bug that the failure text alone does not localise, and a stronger model is the first thing to try. Compare successive diffs under outputs/logs/patches/<task>/<timestamp>/ to tell which one you have. Stalls concentrate in the unit stage, which is also the only stage written with sight of the code.

Test generation can miss a criterion you wrote. A correct, hand-written criterion can end up with no test asserting it, so the coding agent is never forced to satisfy it. Root-caused to the test-generation prompt asking for coverage of "properties the criteria say must hold" without requiring every criterion to be covered, so long lists got sampled down. The prompt fix reduces the gap rather than closing it. Treat as a live gap, and --explain plus the generated suite are how you check it on your own task.

Criteria refinement can tighten a bar, not discover one. The reviewer reacts to what an implementation actually does, so it is good at finding validation gaps in behaviour the code already attempts and cannot surface a behaviour the requirements never asked for. How much it sharpens a bar is an open question.

One provider per run, and no cross-run trend reporting yet: the metrics report covers a single run.

Features

Split three ways, because a single list mixed things that are not alike: what the loop guarantees whatever you type, what you type, and what you are left holding afterwards.

What the loop does on its own

Commands

What you keep

Use it in CI

Listed on the GitHub Marketplace as Qikly Test Generation.

The Action runs one task and keeps what came out: the generated pytest suite, the code that satisfies it, and a JUnit XML file that CI dashboards and test-management tools already read.

- uses: actions/checkout@v5

- id: qikly
  uses: gal-a/qikly@v0
  with:
    task: CALC_TAX
  env:
    GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}

Two working templates are in this repository, meant to be copied out rather than run here: qikly-example.yml runs on demand, and qikly-pr-example.yml comments the result on a pull request.

Three things worth knowing before you wire it up.

It does not fail your build by default. A run that does not converge is a normal outcome, not a broken pipeline, and a tool whose first impression is a red X on someone's main branch does not get a second look. Set fail-on-stall: true when you want it enforcing rather than reporting.

Every trigger spends money. Both templates are workflow_dispatch on purpose. Move to on: pull_request once you know what a run costs you, and scope it by paths: to the task files so an unrelated commit does not pay for a run.

Two ways to pin, and the choice is yours. @v0 is a moving alias that this project repoints at every release, so you receive fixes without receiving a breaking change. @v0.3.4 is an exact pin that never moves, so nothing changes under you and nothing reaches you either. The templates use @v0 because most people want the fixes; use the exact form if your policy requires it.

Use it from your coding agent

qikly runs as an MCP server, so an agent in Claude Code, Codex CLI or Cursor can start a run and read the result without you leaving the conversation.

pip install "qikly[mcp]"
claude mcp add qikly -- qikly-mcp

In VS Code, the Install qikly MCP badge at the top of this page writes the configuration for you. It writes the configuration and nothing else, so the pip install above still comes first.

If the server does not start, one command fixes both ways the badge fails. In your project folder:

python -m qikly --mcp-config

and paste what it prints over the badge's entry. The badge names a bare qikly-mcp, which pip on Windows often installs somewhere that is not on PATH, and the host starts the server in the folder your editor has open, which is often not your project. The printed config names your Python and your project folder in full, so neither can go wrong. --mcp-config claude prints the mcpServers shape that Claude Code and Cursor use. docs/mcp.md has the details.

Four tools: qikly_run, qikly_status, qikly_validate, qikly_scaffold. A run takes minutes to hours and no host will hold a tool call open that long, so qikly_run returns a run id straight away and qikly_status is how the agent finds out what happened. The run is detached, so you can close the editor and ask again tomorrow.

No qikly tool returns your acceptance criteria, on success or on failure. Your agent sees which tests failed and the pytest output. It does not see the rule it broke, which is the same position a human developer is in when CI goes red, and the reason the code it writes next aims at the requirement rather than at the test.

One thing that is your side of the line: the generated tests under outputs/tests/ are written from your criteria, so keep them out of your agent's reach. docs/mcp.md has the config for that and the rest of the setup.

Further reading

Two reference pages, for looking things up rather than reading through: running it on your own data and configuration and provider.

The design write-up is in three parts, and each stands on its own.

Where this came from

The separation this tool enforces is ordinary practice in safety-critical engineering, where verification is required to be independent of implementation as part of a V&V methodology for testing. I worked in that setting before building this toolset.

Built by Gal Arav.

License

Apache License 2.0, see LICENSE.