vacuous

Finds Python tests that pass no matter what your code does.

A test with no assertions, or one that only checks its own mocks, still shows up green and still counts as coverage. Those are the worst kind to have, because they cost you the thing tests are for — you find out the code is broken from somewhere other than your test suite.

$ vacuous check tests/

  12 vacuous tests in 847 (1.4%)

  tests/test_billing.py:41   no-assertions              certain
                             └─ `test_refund_is_recorded` contains no assertions — it passes unless the code under test raises.
  tests/test_users.py:118    swallowed-failure          certain
                             └─ this assertion in `test_email_is_unique` is caught and discarded by the handler on line 121 — it can never fail the test.
  tests/test_orders.py:92    patched-target-under-test  likely
                             └─ `test_charge_card` replaces `charge_card` with a mock and then only asserts on that mock — the real `charge_card` never runs.

  63 files scanned

It's a static analysis pass over tree-sitter, so it doesn't run your tests and doesn't need your dependencies installed. Whole repos take well under a second. No network, no API keys.

Install

cargo install vacuous

Or from source, if you'd rather:

git clone https://github.com/MahdiAlani/vacuous
cd vacuous
cargo build --release

Needs Rust 1.88 or newer. Wheels are on the way, so uv tool install vacuous will work without a Rust toolchain.

Usage

vacuous check [PATH]              # defaults to the current directory
  --min-confidence <LEVEL>        # certain | likely | possible  (default: likely)

Exits 0 when clean, 1 when it finds something, 2 if the tool itself fell over — so CI can tell a real failure apart from a broken run.

Checks

Check What it means
no-assertions Nothing in the test body can fail
constant-assertion Every assertion is on literals: assert True, assertEqual(3, 3)
swallowed-failure An except block discards the assertion
unreachable-assertion The assertion sits after a return or raise
patched-target-under-test The test mocks the function it's named for, then only checks the mock

Each finding carries a confidence. certain means it's a structural fact about the code; likely means there's a judgement call involved. Only certain and likely show by default.

What it deliberately ignores

Tests without assertions aren't automatically bad, which is the main reason pytest hasn't shipped a flag for this. Getting the exceptions right is most of the work here, so vacuous stays quiet on:

  • Tests that delegate to a helper that asserts, including one defined elsewhere in the file and reached indirectly. Flask's common_object_test(app) is the usual shape.
  • Assertion helpers that don't look like assertions — eq_, is_, ne_ and the rest of the nose/SQLAlchemy family.
  • Anything where a decorator does the asserting, like SQLAlchemy's @profiling.function_call_count() or a @pytest.mark.benchmark handing off to the benchmark fixture. Plain pytest.mark labels don't count.
  • Tests returning a value, which means something else is driving them.
  • Functions nested inside tests. Flask names route handlers test and Click commands testcmd; no runner collects those.
  • assert False, "shouldn't get here". That's constant but always fails, so it's a deliberate marker, not a vacuous test. Ruff's PT015 and B011 cover that case from the other direction.
  • except ValueError: pass around an assertion. An AssertionError escapes it, so nothing is being swallowed.

Most of those came from running it over real suites and finding out it was wrong. It's currently checked against flask, requests, httpx, rich, click, black, scrapy, celery, pydantic, ansible, django-rest-framework and sqlalchemy — about 29,000 tests — with every finding read by hand. Roughly 2% of tests in those projects can't fail.

If you hit a false positive, that's a bug worth filing.

How it works

Walk for files a test runner would collect, respecting .gitignore. Parse each one, pull out the test functions, run every check over them, sort by location. Files are handled in parallel.

The Python-specific parts live behind one trait in src/lang/mod.rs, so the checks themselves don't know what language they're looking at. A check is a function from one parsed test to a list of findings; they're in src/rules/. Every check has a pair of fixtures under tests/fixtures/, one that should be flagged and one that shouldn't.

Related tools

  • mutmut and cosmic-ray answer a harder version of this question by mutating your code and re-running the suite. Far more thorough, and much slower. Worth reaching for when you want a real answer about a specific module.
  • flake8-aaa lints tests for Arrange-Act-Assert structure, which catches some of the same shapes.
  • ruff has a pytest ruleset worth turning on regardless, and covers always-false assertions.
  • flake8-pytest-style for pytest idioms more broadly.

vacuous is meant to sit alongside these, not replace them.

Planned

Config in pyproject.toml. A baseline file, so an existing project can fail only on new findings instead of drowning in old ones. JSON and SARIF output. A pre-commit hook. Wheels, so uvx vacuous works.

After that, a verify mode: stub out a function, run the tests, and see whether anything notices. Scoped to a diff it's quick enough to be useful, and it answers the question this tool can only approximate statically.

TypeScript, eventually.

License

MIT or Apache-2.0, at your option.

Project links

Data verified by PyPI on
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.1.2)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for mahdialani from gravatar.com mahdialani

GitHub Statistics

Data verified by PyPI on 
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

MIT OR Apache-2.0
View SPDX License List

Requires

Python >=3.8

Tags

testing linter pytest test-quality static-analysis

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

vacuous-0.1.0.tar.gz (35.6 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

vacuous-0.1.0-py3-none-win_amd64.whl (1.3 MB view details)

Uploaded Python 3Windows x86-64

vacuous-0.1.0-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.4 MB view details)

Uploaded Python 3manylinux: glibc 2.17+ x86-64

vacuous-0.1.0-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.3 MB view details)

Uploaded Python 3manylinux: glibc 2.17+ ARM64

vacuous-0.1.0-py3-none-macosx_11_0_arm64.whl (1.3 MB view details)

Uploaded Python 3macOS 11.0+ ARM64

vacuous-0.1.0-py3-none-macosx_10_12_x86_64.whl (1.3 MB view details)

Uploaded Python 3macOS 10.12+ x86-64

File details

Details for the file vacuous-0.1.0.tar.gz.

File metadata

  • Download URL: vacuous-0.1.0.tar.gz
  • Upload date:
  • Size: 35.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for vacuous-0.1.0.tar.gz
Algorithm Hash digest
SHA256 2f3584a2bea604b6c9d2edc614ccfa9b574d790596fec412e70acd17d9282cf3
MD5 b7ebbd9249700fa35bcd9473174d403b
BLAKE2b-256 2bf69e06a240b28110224626f297bfaca9f628dc12ac693dce1d14db798c07da

See more details on using hashes here.

Provenance

The following attestation bundles were made for vacuous-0.1.0.tar.gz:

Publisher: release.yml on MahdiAlani/vacuous

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file vacuous-0.1.0-py3-none-win_amd64.whl.

File metadata

  • Download URL: vacuous-0.1.0-py3-none-win_amd64.whl
  • Upload date:
  • Size: 1.3 MB
  • Tags: Python 3, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for vacuous-0.1.0-py3-none-win_amd64.whl
Algorithm Hash digest
SHA256 b0e7217a14b94704c08a49470bb11650faff998d53f89caaf0774841de7b90a4
MD5 a31c2858ce77ded2f96b4d3690e4a239
BLAKE2b-256 fd090f69f820793b7cc6bcbf616f20e10e4425b33b6291ce71b0cddd001b0dee

See more details on using hashes here.

Provenance

The following attestation bundles were made for vacuous-0.1.0-py3-none-win_amd64.whl:

Publisher: release.yml on MahdiAlani/vacuous

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file vacuous-0.1.0-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for vacuous-0.1.0-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 837a2a416ae5faccfa010456d3bba431c4f6f94fdd2dbec37572507b24f1c593
MD5 3faa9639374f62d599d0598725766211
BLAKE2b-256 ae2b5abae42e7b82e20223be953d3076013869cc8730730c6bd824566351ef03

See more details on using hashes here.

Provenance

The following attestation bundles were made for vacuous-0.1.0-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release.yml on MahdiAlani/vacuous

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file vacuous-0.1.0-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for vacuous-0.1.0-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 2a17dfd228b3f7d5389d75763b7d6bf7107635d9b9ce5964be451bba41794456
MD5 cbd41eebe7590b0eb9a394c77b3eafaf
BLAKE2b-256 cad2d9d321422e86bc78728389101932ca528ad49f3d90800ecc8ae6435a400e

See more details on using hashes here.

Provenance

The following attestation bundles were made for vacuous-0.1.0-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: release.yml on MahdiAlani/vacuous

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file vacuous-0.1.0-py3-none-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for vacuous-0.1.0-py3-none-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 df813543d671c93627f11f16ad4bec101a66401c59d3961fba26d6dc74ebda52
MD5 db51451d1b84946faa19284cb1b2841d
BLAKE2b-256 d1fa6bd2d98a8d860428462ad7cb8a5eb87ef928094c83f6de7c552da8a9e377

See more details on using hashes here.

Provenance

The following attestation bundles were made for vacuous-0.1.0-py3-none-macosx_11_0_arm64.whl:

Publisher: release.yml on MahdiAlani/vacuous

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file vacuous-0.1.0-py3-none-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for vacuous-0.1.0-py3-none-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 fcd4ca925cd73376cc79b364c98f4723972dd8ab4da959981bd967643b6e9a7f
MD5 50786b0bbdad83a2b11946497abba4a4
BLAKE2b-256 b91a65146357669ef83d8865d20b13c8ba453de2532af54b6f01dacf7b00ae8b

See more details on using hashes here.

Provenance

The following attestation bundles were made for vacuous-0.1.0-py3-none-macosx_10_12_x86_64.whl:

Publisher: release.yml on MahdiAlani/vacuous

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page