qikly.

Your AI writes both the code and its tests. How do you know the tests are really valid?

The solution: two agents. One turns the acceptance criteria into tests. The other writes the code and never sees the acceptance criteria.

A student who writes the exam paper, writes the answer key and then sits the exam will pass. That is what happens when one model is given the acceptance criteria and asked to produce both the implementation and the suite that checks it. Everything goes green, and the green means nothing. qikly takes the answer key away from the student.

$ pip install qikly $ export GEMINI_API_KEY=... $ qikly --demo
Source on GitHub How it works, with one repair end to end

The idea

A passing test suite is only evidence if the author could not read the answer key

Give a model the acceptance criteria and ask it to write both the code and the tests, and it will write tests that pass. That is not verification, it is a model agreeing with itself. The only way the agreement means anything is if the two halves are kept apart.

How the task file is split, and how the loop turns One task file feeds two agents. The test generation agent sees all three sections. The coding agent sees only the first two, never the acceptance criteria. Both meet at pytest. A failing run returns to the coding agent carrying the error text alone. Once the integration and system stages pass, the test generation agent writes the unit stage, and that is the only point at which it is allowed to read the implementation. one task file 1 requirements 2 interface (signatures) 3 acceptance_criteria test generation agent sees 1, 2 and 3, not the code coding agent sees 1 and 2, never 3 the suite integration, system, then unit the code pytest pass converged code + suite fail: the error text, never the rule integration and system pass: now write the unit stage, code visible
The main flow consists of two loops. The dashed orange one is the repair cycle: a failing run returns the error text and nothing else, so the coding agent cannot write code shaped to a rule it was shown. The dashed teal one is the staged expansion: integration and system tests are written before any code exists, and only once they pass is the unit stage written, which is the single point where test generation may read the implementation.
Test generation sees

The requirements, the input and output contract, and every acceptance criterion in full. It writes integration, system and unit tests against the standard.

The coding agent sees

The same specification with the criteria section removed, plus the text of whatever test just failed. The same vague brief a developer usually works from.

You might argue, "just use one model for the code and another for the tests." That helps a little. It does not fix the underlying issue.

Both models still read the same criteria, so both still write to them. The code is still built to satisfy the standard it is about to be judged by. Changing who types it does not change what they were shown.

It is also a habit rather than a mechanism. Nothing checks that the two models stayed different, and a config change a year from now undoes it with no test to notice.

Withholding the acceptance criteria works differently. They are cut out of the task file before the coding agent is handed it, so there is no channel left for them to arrive through, whichever model is writing. qikly --explain prints what each side received, so you can see the cut rather than trust it. You can also do both: qikly picks a model per agent role.

Getting started

One command, about thirty seconds, writes nothing outside its own folder

pip install qikly
export GEMINI_API_KEY=...
qikly --demo

The demo runs one task end to end in a throwaway directory and prints where the code, the tests, and the full record of every FIX and PATCH landed. Bring your own model key; the provider is configurable per agent and per test stage.

Three words, so the commands below make sense

A task file is one YAML file and it holds everything. It has three parts, and the split above is a split between them:

#1 requirements
What the code must do, in the words a person would use. The coding agent reads this.
#2 interface
The contract, and a description rather than code: the function signatures, and the dotted path where the module will live. Both agents read it. Neither is given an implementation to read from it, because when integration and system tests are written there is not one yet.
#3 acceptance_criteria
What counts as correct, each one checkable and naming its boundary value. Only test generation reads this.

"Spec" in this page means the first two together, which is what the coding agent is given. The criteria are the standard, and they are the part held back.

One thing the three parts do not say, and it matters: test generation never reads the implementation either. Integration and system tests are written before any code exists, from the specification alone. The unit stage is the single exception, written last from the code that just cleared the earlier stages, because unit tests have to name real functions.

Which command depends on which parts you already have

A tick means you already have it. Everything without a tick is either written for you or is what you sit down to write.

Where you are starting #1 #2 #3 Run What happens
Before anything else: see what is withheld qikly --explain <MY_TASK>
e.g. qikly --explain CALC_TAX		 Prints a task file twice, once as each agent receives it, and the difference between them. No API key, no model call, about a second. You get: the acceptance criteria on one side and the same file with them cut out on the other, which is the claim everything else here rests on. Shown in full below this table.
Just looking qikly --demo A bundled task end to end in a throwaway folder. Needs an API key, because this one really does call a model. Thirty seconds, You get: a working implementation, three test suites, and the full record of every FIX and PATCH, in a directory you can delete. under a cent.
Code someone else wrote, and you want that code verified Y qikly --scaffold <MY_MODULE>.py
then uncomment seed.implementation		 Scaffold reads the real signatures out of the file you point it at and fills in #2 for you. #1 and #3 stay yours to write: criteria read out of an implementation can only describe what that implementation already does, which is a bar it passes by construction. You get: two task files. One tests the code you already have; the other writes a fresh implementation of the same interface. Keep whichever matches the job and delete the other. Then a suite built from a standard your code was never measured against, and a list of what it fails.
You know what it must do, not yet how to check it Y qikly --init Creates the directory layout and one starter task to edit. Its criteria show the habit that matters most: name the value, not the quality. "100 is accepted and 101 is rejected" forces a test at the boundary; "amounts must be reasonable" does not. You get: a task file to fill in, with your fixtures where a run will look for them.
Same, but you want a first draft of the bar YY qikly --tasks <MY_TASKS> --generate-criteria Drafts #3 from #1 alone, then runs. You get: a first draft of the bar written into your task file for you to read and correct, plus the implementation and the suites. Run --compare-criteria on a task where you did write the criteria, to see what a draft would have missed.
The rules are written, but in a ticket YYY qikly --criteria-from ticket.md --task-id <MY_TASK> Reads #3 out of a ticket you saved to a file: a bullet list, an "Acceptance Criteria" heading, or Gherkin scenarios. Only that section is read, so pasting a whole ticket does not turn its description into part of the bar, and prose is never split into rules nobody wrote. You get: acceptance criteria you already had, in the task file, ready to read over before the first run.
You have written all three YYY qikly --tasks <MY_TASKS> Everything you wrote is used, and nothing is drafted on your behalf. You get: an implementation, integration, system and unit suites, a convergence report, and a run summary recording the model and settings that produced them.
You have all three but doubt they agree YYY qikly --check-criteria --tasks <MY_TASKS> One model call asking whether any implementation could satisfy 1 and 3 at once. Advisory, and exits non-zero on a contradiction so a pipeline can gate on it. You get: a list of the requirement and criterion pairs that cannot both hold, before spending a stage budget on them.
A previous run stopped before finishing YYY qikly --tasks <MY_TASKS> --resume Generating the tests and the first implementation already cost model calls, and they are still on disk. This keeps them and picks up where it stopped rather than paying twice. It names what it is reusing before it starts, so you can see what you are about to pay for. You get: the same outputs as a full run, without paying for the parts already built.

<MY_TASKS> is one task_id or several separated by commas. A task_id is a filename under inputs_private/config/tasks/ without the .yaml: --tasks CALC_TAX, --tasks CALC_TAX,MERGE_SALES, or omit it to run every task found. <MY_TASK>, singular, takes exactly one.

Checkable, not asserted

See the acceptance criteria being removed, in one command

The first row of the table above, in full. This is the claim the rest of the page rests on, so it is worth watching rather than believing. 

qikly --explain CALC_TAX

It prints the task file twice: once as the test generation agent receives it, once as the coding agent does, and the difference between them. On that task, eleven acceptance criteria go to one side and twelve lines are cut before the other side sees the file. No API key, no model call, about a second.

Behind it, qikly's own test suite carries tests/test_withholding.py, which fails the build if any call site ever lets a criterion through to the coding agent. The withholding is not a promise about how a model was prompted, it is a property of the code. A tool built to stop you trusting a green checkmark should not ask you to trust its own.

~60% converged end to end, unit tests included measured three times, across 967 runs
~80% passed integration and system tests small cheap model, so read it as a floor
0 runs reported success on code their own tests rejected across every run measured

Those are round numbers on purpose. A 427-run sweep gave 59% and 80%; a second sweep of 140 runs sixteen days later gave 67% and 87%, well inside the first one’s intervals. Then the benchmark itself turned out to be wrong: in eight of the ten tasks some acceptance criteria could not be triggered by any input row, so part of the bar was not lower, it was absent. Thirty-one rows were added and 400 runs were taken again, giving 64% and 83%. The prediction was that convergence would fall once the bar was genuinely enforceable. It did not move. Three sweeps agreeing, one of them against a corrected benchmark, is worth more than any one of their decimal places.

A run that cannot satisfy its own suite exits non-zero, names the blocking tests, and ships nothing. That is the property worth having: the failure mode is a refusal, not a false pass.

One run is an artifact, not a rate. The same task with the same seed converges on some runs and not others, which is why a figure here comes from repetition and names the model and date it was taken on.

The separation this tool enforces is ordinary practice in safety-critical engineering, where verification is required to be independent of implementation as part of a V&V methodology for testing. This library's author worked in that setting before building this toolset.