vacuous

Finds Python tests that pass no matter what your code does.

A test with no assertions, or one that only checks its own mocks, still shows up green and still counts as coverage. Those are the worst kind to have, because they cost you the thing tests are for — you find out the code is broken from somewhere other than your test suite.

$ vacuous check tests/

  12 vacuous tests in 847 (1.4%)

  tests/test_billing.py:41   no-assertions              certain
                             └─ `test_refund_is_recorded` contains no assertions — it passes unless the code under test raises.
  tests/test_users.py:118    swallowed-failure          certain
                             └─ this assertion in `test_email_is_unique` is caught and discarded by the handler on line 121 — it can never fail the test.
  tests/test_orders.py:92    patched-target-under-test  likely
                             └─ `test_charge_card` replaces `charge_card` with a mock and then only asserts on that mock — the real `charge_card` never runs.

  63 files scanned

It's a static analysis pass over tree-sitter, so it doesn't run your tests and doesn't need your dependencies installed. Most repos take a fraction of a second — flask 244ms, celery 483ms — and sqlalchemy's 12,716 tests take 2.2s. No network, no API keys.

Install

uv tool install vacuous

or pipx install vacuous, or pip install vacuous. There are prebuilt wheels for Linux, macOS and Windows, so this needs no Rust toolchain.

If you'd rather build it yourself:

cargo install vacuous

Needs Rust 1.88 or newer.

Usage

vacuous check [PATH]              # defaults to the current directory
  --min-confidence <LEVEL>        # certain | likely | possible  (default: likely)
  --format <FORMAT>               # pretty | json | sarif        (default: pretty)
  --baseline <FILE>               # defaults to .vacuous-baseline.json if present
  --no-baseline                   # report everything, baseline included

vacuous baseline [PATH]           # record what's there today

Exits 0 when clean, 1 when it finds something, 2 if the tool itself fell over — so CI can tell a real failure apart from a broken run.

Adding it to an existing project

Any codebase of any age will have findings already, and nobody is going to fix two hundred of them before their next commit. Record them once:

$ vacuous baseline
vacuous: recorded 144 findings in .vacuous-baseline.json
vacuous: `vacuous check` will now report only new ones

Commit that file. From then on vacuous check passes, and only fails when someone adds a new test that can't fail.

Entries are keyed on the rule, file and test name, deliberately not the line number, so editing a file doesn't invalidate the baseline. When findings get fixed, vacuous says how many entries are stale and you can re-record.

pre-commit

repos:
  - repo: https://github.com/MahdiAlani/vacuous-pre-commit
    rev: v0.1.1
    hooks:
      - id: vacuous

Installs from the PyPI wheel, so contributors don't need Rust.

CI

- run: vacuous check --format sarif > vacuous.sarif
  continue-on-error: true
- uses: github/codeql-action/upload-sarif@v3
  with:
    sarif_file: vacuous.sarif

Findings then show up as annotations on the pull request. certain maps to error, likely to warning, possible to note; GitHub won't block merges on any of them unless you configure it to.

Checks

Check What it means
no-assertions Nothing in the test body can fail
constant-assertion Every assertion is on literals: assert True, assertEqual(3, 3)
swallowed-failure An except block discards the assertion
unreachable-assertion The assertion sits after a return or raise
patched-target-under-test The test mocks the function it's named for, then only checks the mock

Each finding carries a confidence. certain means it's a structural fact about the code; likely means there's a judgement call involved. Only certain and likely show by default.

no-assertions is graded rather than fixed, because it catches two different things. A body of pass or a lone docstring is a stub and gets certain. A body that calls real code but asserts nothing gets likely, because it might be a deliberate crash-or-hang regression test — numpy has several, and that's the right way to test "this used to segfault". Nothing static can tell those apart, so --min-confidence certain gives you only the indefensible ones.

What it deliberately ignores

Tests without assertions aren't automatically bad, which is the main reason pytest hasn't shipped a flag for this. Getting the exceptions right is most of the work here, so vacuous stays quiet on:

  • Tests that delegate to a helper that asserts, including one defined elsewhere in the file and reached indirectly. Flask's common_object_test(app) is the usual shape.
  • Assertion helpers that don't look like assertions — eq_, is_, ne_ and the rest of the nose/SQLAlchemy family.
  • Anything where a decorator does the asserting, like SQLAlchemy's @profiling.function_call_count() or a @pytest.mark.benchmark handing off to the benchmark fixture. Plain pytest.mark labels don't count.
  • Tests returning a value, which means something else is driving them.
  • Functions nested inside tests. Flask names route handlers test and Click commands testcmd; no runner collects those.
  • assert False, "shouldn't get here". That's constant but always fails, so it's a deliberate marker, not a vacuous test. Ruff's PT015 and B011 cover that case from the other direction.
  • except ValueError: pass around an assertion. An AssertionError escapes it, so nothing is being swallowed.

Most of those came from running it over real suites and finding out it was wrong. It's currently checked against flask, requests, httpx, rich, click, black, scrapy, celery, pydantic, ansible, django-rest-framework and sqlalchemy — about 29,000 tests — with every finding read by hand. Roughly 2% of tests in those projects can't fail.

If you hit a false positive, that's a bug worth filing.

How it works

Walk for files a test runner would collect, respecting .gitignore. Parse each one, pull out the test functions, run every check over them, sort by location. Files are handled in parallel.

The Python-specific parts live behind one trait in src/lang/mod.rs, so the checks themselves don't know what language they're looking at. A check is a function from one parsed test to a list of findings; they're in src/rules/. Every check has a pair of fixtures under tests/fixtures/, one that should be flagged and one that shouldn't.

Related tools

  • mutmut and cosmic-ray answer a harder version of this question by mutating your code and re-running the suite. Far more thorough, and much slower. Worth reaching for when you want a real answer about a specific module.
  • flake8-aaa lints tests for Arrange-Act-Assert structure, which catches some of the same shapes.
  • ruff has a pytest ruleset worth turning on regardless, and covers always-false assertions.
  • flake8-pytest-style for pytest idioms more broadly.

vacuous is meant to sit alongside these, not replace them.

Planned

Config in pyproject.toml, so per-project settings live where the rest of your tooling is configured. Per-line suppression comments.

After that, a verify mode: stub out a function, run the tests, and see whether anything notices. Scoped to a diff it's quick enough to be useful, and it answers directly what this tool can only approximate by reading code.

TypeScript, eventually. The checks don't know they're looking at Python — that lives behind one trait — so it's a new adapter rather than a rewrite.

License

MIT or Apache-2.0, at your option.

Project links

Data verified by PyPI on
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for mahdialani from gravatar.com mahdialani

GitHub Statistics

Data verified by PyPI on 
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

MIT OR Apache-2.0
View SPDX License List

Requires

Python >=3.8

Tags

testing linter pytest test-quality static-analysis

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

vacuous-0.1.2.tar.gz (44.7 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

vacuous-0.1.2-py3-none-win_amd64.whl (1.4 MB view details)

Uploaded Python 3Windows x86-64

vacuous-0.1.2-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.5 MB view details)

Uploaded Python 3manylinux: glibc 2.17+ x86-64

vacuous-0.1.2-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.4 MB view details)

Uploaded Python 3manylinux: glibc 2.17+ ARM64

vacuous-0.1.2-py3-none-macosx_11_0_arm64.whl (1.4 MB view details)

Uploaded Python 3macOS 11.0+ ARM64

vacuous-0.1.2-py3-none-macosx_10_12_x86_64.whl (1.4 MB view details)

Uploaded Python 3macOS 10.12+ x86-64

File details

Details for the file vacuous-0.1.2.tar.gz.

File metadata

  • Download URL: vacuous-0.1.2.tar.gz
  • Upload date:
  • Size: 44.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for vacuous-0.1.2.tar.gz
Algorithm Hash digest
SHA256 d6f55b0fb0edb1270ca693f55090305e4d51b5c1640eca84cf91c319ba167865
MD5 747449b67ddf8612585d1211ced6d468
BLAKE2b-256 e858a421f2326f05f4d03ec2459e3a771993adb09626d5834bd0f95f33276eba

See more details on using hashes here.

Provenance

The following attestation bundles were made for vacuous-0.1.2.tar.gz:

Publisher: release.yml on MahdiAlani/vacuous

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file vacuous-0.1.2-py3-none-win_amd64.whl.

File metadata

  • Download URL: vacuous-0.1.2-py3-none-win_amd64.whl
  • Upload date:
  • Size: 1.4 MB
  • Tags: Python 3, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for vacuous-0.1.2-py3-none-win_amd64.whl
Algorithm Hash digest
SHA256 f4003c7c3e489491c9f3b87d3fb27496dc6588cb10e1f861fd75dd34090954e5
MD5 757316d0e0097ada79019a8162ab5ce8
BLAKE2b-256 75ac40fa76f070e742112128a5959f0c40286a8b9a655814b5cad054b4d8cba6

See more details on using hashes here.

Provenance

The following attestation bundles were made for vacuous-0.1.2-py3-none-win_amd64.whl:

Publisher: release.yml on MahdiAlani/vacuous

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file vacuous-0.1.2-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for vacuous-0.1.2-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 52c001b0fcc81433a398b6d2210c203272f8cf51ad544c92ed7ec65eb2cd5ab9
MD5 56c0f8697c2ebc63ed24367fb253a18c
BLAKE2b-256 2cc043bfb19638019fc1cba0cb03dc6dd7ad170cc6e9349b513bb44d8171a067

See more details on using hashes here.

Provenance

The following attestation bundles were made for vacuous-0.1.2-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release.yml on MahdiAlani/vacuous

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file vacuous-0.1.2-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for vacuous-0.1.2-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 7336e0220354324534d7965be0f35ac38abb90b92e143e3e915cfcf681997c20
MD5 7d59516675544bee645243beddc76012
BLAKE2b-256 608129ae0e750690e63136c0a2f6b815b6b57b986fe93b1fadcfcdcb1432de22

See more details on using hashes here.

Provenance

The following attestation bundles were made for vacuous-0.1.2-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: release.yml on MahdiAlani/vacuous

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file vacuous-0.1.2-py3-none-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for vacuous-0.1.2-py3-none-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 0722cbc3054470a908f229e8c3e09e6ec51bd2f32b42f7ea2750fa7e713c5421
MD5 7a4f3a32053e8d5664df9d4941a6d2e1
BLAKE2b-256 a06a666caf05235c18c946804eff402e5d7ecaf68ea53dfdbd93b8963135a9a4

See more details on using hashes here.

Provenance

The following attestation bundles were made for vacuous-0.1.2-py3-none-macosx_11_0_arm64.whl:

Publisher: release.yml on MahdiAlani/vacuous

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file vacuous-0.1.2-py3-none-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for vacuous-0.1.2-py3-none-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 32faff56764570706190f0ab3f608aca746c0512b07c614973d7c1a47670520b
MD5 3dfa1d1d156a4bbd260fd67f158f91e2
BLAKE2b-256 c47722d3bc4e6d1e3ee670d48de0923e69ad999b6ea6c630e138d898834d576d

See more details on using hashes here.

Provenance

The following attestation bundles were made for vacuous-0.1.2-py3-none-macosx_10_12_x86_64.whl:

Publisher: release.yml on MahdiAlani/vacuous

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page