qikly

Functional tests for backend Python

Independent test generation for AI-written code

The coder never sees the answer key. Its tests come from criteria it was never shown, so they can disagree with it. Two code panels kept apart: calc.py written by the coding agent, and test_calc.py written by the test agent, which asserts a tax rate limit the coding agent was never given. The suite passes 9 of 9.

This was run by two independent AI agents:

  1. The Test Agent builds the test suite: it was told, from the spec's acceptance criteria, that the rate may not exceed 100.
  2. The Coding Agent never saw the acceptance criteria and discovered that rate limit from a failing test: it then added or rate > 100 by itself, so the 9 of 9 is code eventually passing the test suite after applying fixes in an automated repair loop.
$ pip install qikly
Source on GitHub

Start

Create a test suite for your own code

See it work first: qikly --demo runs the whole loop on a bundled task, nothing to fill in: tests written before any code, a coding agent that never sees the criteria, and a repair. About thirty seconds, well under a cent with a Gemini, OpenAI or Anthropic key. Watch one run

Run these from your project's root folder. Each takes one module or a whole folder, and every step takes the same path.

  1. Step 1 Score the tests you have qikly --score your_module.pyorqikly --score your_project_folder

    Plants faults in a copy of your code, one at a time, and turns each one your tests did not notice into a question about behaviour nothing checks. Nothing of yours is changed. Where your tests can be

    free, no key
    What it printed for pricing.py, whose two tests both pass
    [pricing.py] 10 of 13 planted faults missed (77%), 3 caught.
[pricing.py] 8 questions your tests leave open, in the same report.
  2. Step 2 Answer the questions in the task file qikly --scaffold your_module.pyorqikly --scaffold your_project_folder

    Writes the task file, inputs_private/config/tasks/YOUR_MODULE_VERIFY.yaml, one per module step 1 asked about for a folder, with everything qikly can read from your code filled in. What is left is step 1's questions, each with what your code does there now, found by running it, numbered, and a line saying what the module is for. Answer each TODO with an option's number, or write your own; a TODO you leave is skipped. This file is the source of truth: change it any time. qikly --validate with the same path checks it, free. A module and a folder, answered

    free, no key
    The third question in the file it wrote, answered
    #?     if rate < 0 or rate > 100:
#? Your tests never try rate at exactly 100.
#? What should apply_discount(100, 100) do?
#?   1 = apply_discount(100, 100) returns VALUE
#?       (as rate 99 does now)
#?   2 = apply_discount(100, 100) raises ValueError
#?       (as rate 101 does now)
- 2

    qikly runs your code beside the value in question, never at it: the answer is yours

  3. Step 3 Get a fresh test suite written from your answers qikly --tasks your_module.pyorqikly --tasks your_project_folder

    Set a key first, such as GEMINI_API_KEY, or one for Anthropic or OpenAI. A test agent writes tests from your answers before any code is shown to it, and a separate coding agent, which never sees them, repairs a copy of your code until they pass. Your files are never edited. Then qikly --take-tests with the same path copies the new tests into your own tests folder, importing your code, and runs them on it: a test that fails there is one to compare with your answer.

    needs an API key
    The new tests on your own code, and the fix the run made in its copy
    Run on your own code: 3 failed, 11 passed.
These fail on your code and were written
from your criteria:
  test_apply_discount_excessive_rate_raises

-    if rate < 0 or rate > 100:
+    if rate < 0 or rate >= 100:

    - and + qikly's repair, made in a copy of your code: not your edit

Where your tests can be. Anywhere in your project, such as a tests/ folder or beside your code, in files named test_*.py that import your module. Step 1 finds them; to name them instead, add --score-tests test_folder/.

Other ways in

Read more

What it is for

The behaviour of backend logic, one module at a time

It tests

What a Python module does with its inputs: data transforms and ETL steps, calculations, validation rules, merges and business logic. Integration, system and unit tests, in pytest, for small to mid-sized repositories.

It does not test

User interfaces, performance, load, security or any other non-functional property, and it does not run end-to-end tests across services.

It fits most naturally where verification already has to be independent of the people who build the thing: Automotive, Medical Devices, Fintech and Defence.

How it works

The coding agent never sees the acceptance criteria

One task file holds the requirements, the interface and the acceptance criteria. The test agent reads all three and writes the integration and system tests before any code exists. The coding agent gets the same file with the criteria cut out, and works from failing tests until the suite passes.

How the task file is split, and how the loop turns One task file feeds two agents. The test generation agent sees all three sections. The coding agent sees only the first two, never the acceptance criteria. Both meet at pytest. A failing run returns to the coding agent with the failing test and its error, never the criteria. Once the integration and system stages pass, the test generation agent writes the unit stage, and that is the only point at which it is allowed to read the implementation. one task file 1 requirements 2 interface (signatures) 3 acceptance_criteria test generation agent sees 1, 2 and 3, not the code coding agent sees 1 and 2, never 3 the suite integration, system, then unit the code pytest pass converged code + suite fail: the failing test, never the criteria integration and system pass: now write the unit stage, code visible
The main flow consists of two loops. The dashed purple one is the repair cycle: a failing run returns the failing test and its error, never the criteria, so the coding agent can only learn the bar one failed test at a time. The dashed teal one is the staged expansion: integration and system tests are written before any code exists, and only once they pass is the unit stage written, which is the single point where test generation may read the implementation.

Two obvious objections

"Just use one model for the code and another for the tests." That helps a little. It does not fix the underlying issue: both models still read the same criteria, so the code is still built to the standard it is about to be judged by. Changing who types it does not change what they were shown. qikly removes the criteria from the coding agent's prompt, qikly --explain shows the cut, and a test fails the build if any call site lets a criterion through.

"Doesn't a failing test give the criteria away?" It shows the coding agent that one test, its source and its error, and the test often states the rule it checks. It cannot change a test already written: the integration and system suites exist before any code does, so what the agent learns from a failure can shape the code, never the bar.

See it run end-to-end on a bundled task

One command, about thirty seconds, nothing written outside its folder

A qikly --demo run: the acceptance criteria are withheld from the coding agent, three test suites are generated, a test fails, a patch is applied, and the suite goes green.
One qikly --demo run, unedited. The criteria are withheld, the suites are generated, a test fails, a patch lands, and it goes green. 38 seconds on gemini-3.5-flash-lite, not sped up.

The separation this tool enforces is ordinary practice in safety-critical engineering, where verification is required to be independent of implementation as part of a V&V (Verification & Validation) methodology for testing. This library's author worked in that setting on General Motors' autonomous vehicle program before building this toolset.