test-organizer-pro 0.0.4

Test Organizer Pro - Find redundant pytest tests via coverage + mutation analysis

TOP — Test Organizer Pro

Disclaimer: This is a fully vibe-coded project and is not yet ready for any serious use.

Find redundant and useless pytest tests by combining coverage analysis with mutation testing.

TOP runs each test individually, measures which source lines it covers (minus import-time baseline), then mutates those lines to see which ones the test actually asserts on. The output is a JSON map you can use to identify tests that cover code without validating it.

Install

pip install top
# or with uv
uv add top

Usage

Run TOP from the root of your pytest project:

top analyze . -s src

• . — root of your pytest project (where pytest discovers tests)
• -s / --source — source directory to measure coverage for (repeatable for multiple dirs)
• -o / --output — output file path relative to project (default: .top/analysis.json)
• --stdout — print JSON to stdout instead of writing to a file
• --include-tests — include test files in coverage/assertion output (off by default)
• --html — generate an HTML report in .top/report/

By default, results are written to .top/analysis.json inside the analyzed project. The .top/ directory is created automatically.

Examples

# Analyze a standard src-layout project (writes to .top/analysis.json)
top analyze . -s src

# Analyze with multiple source directories
top analyze . -s src -s lib

# Save results to a custom path
top analyze . -s src -o results.json

# Include test file coverage in output
top analyze . -s src --include-tests

# Generate an HTML report
top analyze . -s src --html

# Print to stdout for piping
top analyze . -s src --stdout

# Pipe through jq to find useless tests (coverage but zero assertions)
top analyze . -s src --stdout | jq 'to_entries[] | select(.value | to_entries | all(.value.asserted == []))'

Running with uvx (no install needed)

uvx top analyze . -s src

Output file

TOP writes its analysis to .top/analysis.json inside the project by default. This file contains a JSON object mapping each test to the source lines it covers and asserts:

{
  "tests/test_example.py::test_foo": {
    "src/module.py": {
      "covered": [10, 11, 15],
      "asserted": [10, 15]
    }
  }
}

• covered — lines the test executes beyond the import-time baseline
• asserted — subset of covered lines where a mutation causes the test to fail

A test with empty asserted lists is useless — it runs code but validates nothing. A test whose asserted lines are a strict subset of another test's is a candidate for removal.

By default, the output only includes source files — lines covered in the test file itself are excluded. Use --include-tests to include them.

The .top/ directory is a good candidate for .gitignore since analysis results are generated and machine-specific.

HTML report

Use --html to generate a browsable HTML report in .top/report/:

top analyze . -s src --html
open .top/report/index.html

The report provides a Codecov-style view of your codebase:

• Index page — lists all source files with a stacked coverage bar (green = asserted, yellow = covered only, gray = baseline)
• File pages — line-by-line source view where each line is color-coded:
  • Green — asserted: at least one test fails when this line is mutated
  • Yellow — covered only: tests execute this line but no mutation is caught
  • Gray — baseline: executed at import time (before any test runs)
  • White — not reached by any test

Hover over the test count column to see which tests cover and assert each line.

Optimize — find and skip redundant tests

After running top analyze, use top optimize to identify tests whose assertions are entirely covered by other tests:

top optimize .

This reads .top/analysis.json and reports which tests are redundant — meaning every line they assert is also asserted by at least one other keeper test.

Auto-skip redundant tests

Use --skip to automatically add skip decorators to redundant test functions:

top optimize . --skip

TOP detects the test framework and applies the appropriate decorator:

• pytest tests (standalone functions or classes without unittest):

  @pytest.mark.skip(reason="Redundant: covered by tests/test_foo.py::test_bar")
  def test_redundant():
      ...
  

• unittest tests (methods in classes, file imports unittest):

  @unittest.skip("Redundant: covered by tests/test_foo.py::TestBar::test_baz")
  def test_redundant(self):
      ...
  

The --skip flag is non-destructive — it only adds decorators, never deletes tests. To undo, remove the @...skip decorators. Tests that already have a skip decorator are left unchanged.

Options

• -i / --input — analysis JSON path relative to project (default: .top/analysis.json)
• -o / --output — write JSON report with keepers and redundant tests
• --skip — add skip decorators to redundant test functions

Examples

# Find redundant tests
top optimize .

# Find and auto-skip redundant tests
top optimize . --skip

# Use a custom analysis file
top optimize . -i results.json

# Save the redundancy report as JSON
top optimize . -o .top/optimize.json

How it works

1. Discover all tests via pytest --collect-only
2. Baseline — for each test file, run a noop test that replicates its imports to capture import-time coverage
3. Coverage — run each test individually with coverage.py, subtract the baseline
4. Mutation — for each net-new covered line, apply AST mutations (operator swaps, constant changes, return value changes) and re-run the test. If a mutation makes the test fail, the line is "asserted"

Requirements

• Python >= 3.12
• Your project must use pytest
• coverage (installed as a dependency of TOP)