Antigen

Extendable mutation testing framework

What is mutation testing?

Mutation testing provides what coverage tries to, it finds logic that is not covered by your test suite.

It finds such places by applying mutations to your code and running the modified code against your test suite. If the tests succeed with the mutated code, it means the changed expression is likely not covered ny the tests.

In comparison to coverage:

:green_heart: Checks expressions, not lines.

:green_heart: Checks whether the expression is covered, not whether it was executed.

:x: Can find irrelevant mutants (e.g. mutations in logging or performance optimizations or a mutation that does not break the code)

:x: Executes the tests many times and therefore takes much more time.

There are mitigations for these downsides:

:star: We can mutate only lines that have changed in a given PR

:star: We can show the failing mutants via comments/warnings, as opposed to failing the whole CI pipeline.

A much more in-depth explanation about the concept can be found in This blog post by Goran Petrovic

Why use Antigen?

Extendability

In my personal experience, trying to integrate mutation testing into your CI pipeline can be a bit challenging. There are a lot of features you might want to customize to mitigate some of the downsides of mutation testing, or to be able to integrate it to your project and dev environment effectively.

For example, ignoring mutations on logging logic (which depends on your logging framework and conventions), or showing the results on various platforms (e.g. github, bitbucket, gitlab).

Antigen puts extendability as a top priority so that adding mutation testing to your project is feasible.

How?

Mechine friendly

The core of antigen is a pure python package that can be used by scripts.

The actual CLI uses the core package instead of the logic being coupled to it.

Pluginable

Antigen is written as a pipeline, each stage has an interface (e.g. Mutator, MutantFilter, Runner).

Extending the logic is as simple as creating an object or function that matches that (simple) interface.

The Antigen CLI utilises entry points so that antigen plugins can be added just by installing them with pip.

Usage

Antigen is currently in development, the API might change between versions.

antigen = Antigen(
    filters=[PatchFilter.from_git_diff("develop")],
    config=Config(project_root=Path("/home/myuser/myproject/")),
)

for path in antigen.find_files("."):
    for mutation in antigen.gen_mutations(path):
        result = antigen.test_mutation(
            mutation,
            run_command="pytest",
        )
        print(result)

Roadmap

• Add CLI
• Use parso instead of the built-in ast for cross-version mutations.
• Add wrapper class for remote components (i.e. RemoteFilter(hostname), RemoteRunner(hostname)).
• Add Output component (with JunitXMLOutput, GithubOutput, BitbucketOutput builtins)
• Add Cache component (with FileCache, MongoCache builtins)
• Add Sorter component (for selecting the most likely to succeed mutations)