vacuous

Finds Python tests that pass no matter what your code does.

A test with no assertions, or one that only checks its own mocks, still shows up green and still counts as coverage. Those are the worst kind to have, because they cost you the thing tests are for — you find out the code is broken from somewhere other than your test suite.

$ vacuous check tests/

  12 vacuous tests in 847 (1.4%)

  tests/test_billing.py:41   no-assertions              certain
                             └─ `test_refund_is_recorded` contains no assertions — it passes unless the code under test raises.
  tests/test_users.py:118    swallowed-failure          certain
                             └─ this assertion in `test_email_is_unique` is caught and discarded by the handler on line 121 — it can never fail the test.
  tests/test_orders.py:92    patched-target-under-test  likely
                             └─ `test_charge_card` replaces `charge_card` with a mock and then only asserts on that mock — the real `charge_card` never runs.

  63 files scanned

It's a static analysis pass over tree-sitter, so it doesn't run your tests and doesn't need your dependencies installed. Whole repos take well under a second. No network, no API keys.

Install

uv tool install vacuous

or pipx install vacuous, or pip install vacuous. There are prebuilt wheels for Linux, macOS and Windows, so this needs no Rust toolchain.

If you'd rather build it yourself:

cargo install vacuous

Needs Rust 1.88 or newer.

Usage

vacuous check [PATH]              # defaults to the current directory
  --min-confidence <LEVEL>        # certain | likely | possible  (default: likely)
  --format <FORMAT>               # pretty | json | sarif        (default: pretty)
  --baseline <FILE>               # defaults to .vacuous-baseline.json if present
  --no-baseline                   # report everything, baseline included

vacuous baseline [PATH]           # record what's there today

Exits 0 when clean, 1 when it finds something, 2 if the tool itself fell over — so CI can tell a real failure apart from a broken run.

Adding it to an existing project

Any codebase of any age will have findings already, and nobody is going to fix two hundred of them before their next commit. Record them once:

$ vacuous baseline
vacuous: recorded 144 findings in .vacuous-baseline.json
vacuous: `vacuous check` will now report only new ones

Commit that file. From then on vacuous check passes, and only fails when someone adds a new test that can't fail.

Entries are keyed on the rule, file and test name, deliberately not the line number, so editing a file doesn't invalidate the baseline. When findings get fixed, vacuous says how many entries are stale and you can re-record.

pre-commit

repos:
  - repo: https://github.com/MahdiAlani/vacuous-pre-commit
    rev: v0.1.0
    hooks:
      - id: vacuous

Installs from the PyPI wheel, so contributors don't need Rust.

CI

- run: vacuous check --format sarif > vacuous.sarif
  continue-on-error: true
- uses: github/codeql-action/upload-sarif@v3
  with:
    sarif_file: vacuous.sarif

Findings then show up as annotations on the pull request. certain maps to error, likely to warning, possible to note; GitHub won't block merges on any of them unless you configure it to.

Checks

Check What it means
no-assertions Nothing in the test body can fail
constant-assertion Every assertion is on literals: assert True, assertEqual(3, 3)
swallowed-failure An except block discards the assertion
unreachable-assertion The assertion sits after a return or raise
patched-target-under-test The test mocks the function it's named for, then only checks the mock

Each finding carries a confidence. certain means it's a structural fact about the code; likely means there's a judgement call involved. Only certain and likely show by default.

What it deliberately ignores

Tests without assertions aren't automatically bad, which is the main reason pytest hasn't shipped a flag for this. Getting the exceptions right is most of the work here, so vacuous stays quiet on:

  • Tests that delegate to a helper that asserts, including one defined elsewhere in the file and reached indirectly. Flask's common_object_test(app) is the usual shape.
  • Assertion helpers that don't look like assertions — eq_, is_, ne_ and the rest of the nose/SQLAlchemy family.
  • Anything where a decorator does the asserting, like SQLAlchemy's @profiling.function_call_count() or a @pytest.mark.benchmark handing off to the benchmark fixture. Plain pytest.mark labels don't count.
  • Tests returning a value, which means something else is driving them.
  • Functions nested inside tests. Flask names route handlers test and Click commands testcmd; no runner collects those.
  • assert False, "shouldn't get here". That's constant but always fails, so it's a deliberate marker, not a vacuous test. Ruff's PT015 and B011 cover that case from the other direction.
  • except ValueError: pass around an assertion. An AssertionError escapes it, so nothing is being swallowed.

Most of those came from running it over real suites and finding out it was wrong. It's currently checked against flask, requests, httpx, rich, click, black, scrapy, celery, pydantic, ansible, django-rest-framework and sqlalchemy — about 29,000 tests — with every finding read by hand. Roughly 2% of tests in those projects can't fail.

If you hit a false positive, that's a bug worth filing.

How it works

Walk for files a test runner would collect, respecting .gitignore. Parse each one, pull out the test functions, run every check over them, sort by location. Files are handled in parallel.

The Python-specific parts live behind one trait in src/lang/mod.rs, so the checks themselves don't know what language they're looking at. A check is a function from one parsed test to a list of findings; they're in src/rules/. Every check has a pair of fixtures under tests/fixtures/, one that should be flagged and one that shouldn't.

Related tools

  • mutmut and cosmic-ray answer a harder version of this question by mutating your code and re-running the suite. Far more thorough, and much slower. Worth reaching for when you want a real answer about a specific module.
  • flake8-aaa lints tests for Arrange-Act-Assert structure, which catches some of the same shapes.
  • ruff has a pytest ruleset worth turning on regardless, and covers always-false assertions.
  • flake8-pytest-style for pytest idioms more broadly.

vacuous is meant to sit alongside these, not replace them.

Planned

Config in pyproject.toml, so per-project settings live where the rest of your tooling is configured. Per-line suppression comments.

After that, a verify mode: stub out a function, run the tests, and see whether anything notices. Scoped to a diff it's quick enough to be useful, and it answers directly what this tool can only approximate by reading code.

TypeScript, eventually. The checks don't know they're looking at Python — that lives behind one trait — so it's a new adapter rather than a rewrite.

License

MIT or Apache-2.0, at your option.

Project links

Data verified by PyPI on
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.1.2)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for mahdialani from gravatar.com mahdialani

GitHub Statistics

Data verified by PyPI on 
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

MIT OR Apache-2.0
View SPDX License List

Requires

Python >=3.8

Tags

testing linter pytest test-quality static-analysis

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

vacuous-0.1.1.tar.gz (41.0 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

vacuous-0.1.1-py3-none-win_amd64.whl (1.4 MB view details)

Uploaded Python 3Windows x86-64

vacuous-0.1.1-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.5 MB view details)

Uploaded Python 3manylinux: glibc 2.17+ x86-64

vacuous-0.1.1-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.4 MB view details)

Uploaded Python 3manylinux: glibc 2.17+ ARM64

vacuous-0.1.1-py3-none-macosx_11_0_arm64.whl (1.3 MB view details)

Uploaded Python 3macOS 11.0+ ARM64

vacuous-0.1.1-py3-none-macosx_10_12_x86_64.whl (1.4 MB view details)

Uploaded Python 3macOS 10.12+ x86-64

File details

Details for the file vacuous-0.1.1.tar.gz.

File metadata

  • Download URL: vacuous-0.1.1.tar.gz
  • Upload date:
  • Size: 41.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for vacuous-0.1.1.tar.gz
Algorithm Hash digest
SHA256 554c165ec09019a26b58e008f40c25e4a046d98adc57161cdadf9e12b94adbb1
MD5 f853a2b5ae455917917d3f772c858d65
BLAKE2b-256 90bc74390e83f0d439fd680d85389bbb42b7d752924355b609f2ba163e7c0fed

See more details on using hashes here.

Provenance

The following attestation bundles were made for vacuous-0.1.1.tar.gz:

Publisher: release.yml on MahdiAlani/vacuous

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file vacuous-0.1.1-py3-none-win_amd64.whl.

File metadata

  • Download URL: vacuous-0.1.1-py3-none-win_amd64.whl
  • Upload date:
  • Size: 1.4 MB
  • Tags: Python 3, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for vacuous-0.1.1-py3-none-win_amd64.whl
Algorithm Hash digest
SHA256 9f3db8fd2dcd18ba2fffed796e50da7feada96617bbfcea078c55e5ddb9e6beb
MD5 86f4dd3d719e81cb50ae30490d5be81e
BLAKE2b-256 aec1b6b117d27ef62bc1fe80c30397e2996528197c251ae3e7e4d8db41bf48b2

See more details on using hashes here.

Provenance

The following attestation bundles were made for vacuous-0.1.1-py3-none-win_amd64.whl:

Publisher: release.yml on MahdiAlani/vacuous

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file vacuous-0.1.1-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for vacuous-0.1.1-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 404c486933c3c4cccb9da8a9ed1ecd3eb38729e16da8052f6b3bab7852e5b18a
MD5 64bfcc3afc4b7d3cd531c995a54a367f
BLAKE2b-256 f5aa030723cf629e0d8677314510681053fb3590aa050bfefc6f1b85438bf6e3

See more details on using hashes here.

Provenance

The following attestation bundles were made for vacuous-0.1.1-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release.yml on MahdiAlani/vacuous

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file vacuous-0.1.1-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for vacuous-0.1.1-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 2c50a125b03c7604fc383b79650671211f46274f8a823e771d0a6780e1ac8176
MD5 a2981fdeda7495c06df02ca58d59d0bf
BLAKE2b-256 fbaca6f98b33681b1bb349fa48983f2550b54529bd405f7823d9a4434ea0dffd

See more details on using hashes here.

Provenance

The following attestation bundles were made for vacuous-0.1.1-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: release.yml on MahdiAlani/vacuous

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file vacuous-0.1.1-py3-none-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for vacuous-0.1.1-py3-none-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 6fa4574c11584ed562b7073c416f65782c9d3bbcb38b8357c77f04b56d0e4c17
MD5 87ea4cdbfd90b13b3044edfa10fb62d8
BLAKE2b-256 d371a55c63d537c561517c36632ff5b0dc9bad3b405412afc79070484f30b785

See more details on using hashes here.

Provenance

The following attestation bundles were made for vacuous-0.1.1-py3-none-macosx_11_0_arm64.whl:

Publisher: release.yml on MahdiAlani/vacuous

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file vacuous-0.1.1-py3-none-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for vacuous-0.1.1-py3-none-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 524ea32691bd06eb9fd72849352dca3c7303d9abd317bac3a22a995f18bb88c2
MD5 7258f79a83fefcdd68255e374c463534
BLAKE2b-256 369b75ec8042fe5eae971b5355aa0ca1295a5e9f5d4c60f8ac3aafbc54a39df5

See more details on using hashes here.

Provenance

The following attestation bundles were made for vacuous-0.1.1-py3-none-macosx_10_12_x86_64.whl:

Publisher: release.yml on MahdiAlani/vacuous

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page