Python mutation testing: test your tests!
Project description
mutatest
: Python mutation testing
Are you confident in your tests? Try out mutatest
and see if your tests will detect small
modifications (mutations) in the code. Surviving mutations represent subtle changes that are
undetectable by your tests. These mutants are potential modifications in source code that continuous
integration checks would miss.
- Features:
Simple command line tool.
Built on Python’s Abstract Syntax Tree (AST) grammar to ensure mutants are valid.
No source code modification, only the
__pycache__
is changed.Uses
coverage
to create only meaningful mutants.Built for efficiency with multiple running modes and random sampling of mutation targets.
Flexible enough to run on a whole package or a single file.
mutatest
is alpha-software, see the CHANGELOG [1] for updates.
Installation
mutatest
requires Python 3.7. You can install with pip
:
$ pip install mutatest
Alternatively, clone this repo and install locally:
$ cd mutatest
$ pip install .
mutatest
is designed to work when your test files are separated from your source directory
and are prefixed with test_
. See Pytest Test Layout [2] for more details.
Using mutatest
mutatest
is designed to be a diagnostic command line tool for your test coverage assessment.
The mutation trial process follows these steps when mutatest
is run:
Scan for your existing Python package, or use the input source location.
Create an abstract syntax tree (AST) from the source files.
Identify locations in the code that may be mutated (line and column). If you are running with
coverage
the sample is restricted only to lines that are marked as covered in the.coverage
file.Take a random sample of the identified locations.
Apply a mutation at the location by modifying a copy of the AST and writing a new cache file to the appropriate
__pycache__
location with the source file statistics.Run the test suite. This will use the mutated
__pycache__
file since the source statistics are the same for modification time.See if the test suite detected the mutation by a failed test.
Remove the modified
__pycache__
file.Repeat steps 5-9 for the remaining selected locations to mutate.
Write an output report of the various mutation results.
A “clean trial” of your tests is run before any mutations are applied. This same “clean trial” is
run at the end of the mutation testing. This ensures that your original test suite passes before
attempting to detect surviving mutations and that the __pycache__
has been appropriately
reset when the mutation trials are finished.
Specifying source files and test commands
If you have a Python package in a directory with an associated tests/
folder
(or internal test_
prefixed files, see the examples below) that are auto-detected
with pytest
, then you can run mutatest
without any arguments.
$ mutatest
It will detect the package, and run pytest
by default. If you want to run with special
arguments, such as to exclude a custom marker, you can pass in the --testcmds
argument
with the desired string.
Here is the command to run pytest
and exclude tests marked with pytest.mark.slow
.
$ mutatest --testcmds "pytest -m 'not slow'"
# using shorthand arguments
$ mutatest -t "pytest -m 'not slow'"
You can use this syntax if you want to specify a single module in your package to run and test.
$ mutatest --src mypackage/run.py --testcmds "pytest tests/test_run.py"
# using shorthand arguments
$ mutatest -s mypackage/run.py -t "pytest tests/test_run.py"
There is an option to exclude files from the source set.
Exclude files using the --exclude
argument and pointing to the file.
Multiple --exclude
statements may be used to exclude multiple files. The default behavior
is that no files are excluded.
$ mutatest --exclude mypackage/__init__.py --exclude mypackage/_devtools.py
# using shorthand arguments
$ mutatest -e mypackage/__init__.py -e mypackage/_devtools.py
These commands can all be combined in different ways to target your sample space for mutations.
Coverage optimization
Any command combination that generates a .coverage
file will use that as a restriction
mechanism for the sample space to only select mutation locations that are covered. For example,
running:
$ mutatest --testcmds "pytest --cov=mypackage tests/test_run.py"
# using shorthand arguments
$ mutatest -t "pytest --cov=mypackage tests/test_run.py"
would generate the .coverage
file based on tests/test_run.py
. Therefore, even though
the entire package is seen only the lines covered by tests/test_run.py
will be mutated
during the trials.
If you specified a source with -s
only the covered lines in that source file would become
valid targets for mutation. Excluded files with -e
are still skipped.
You can override this behavior with the --nocov
flag on the command line.
If you have a pytest.ini
file that includes the --cov
command the default behavior
of mutatest
will generate the coverage file. You will see this in the CLI output at the
beginning of the trials:
$ mutatest -n 4 -t "pytest --cov=mypackage"
... prior output...
... Get mutatest targets from AST.
... Full sample space size: 115
... Coverage optimized sample space size: 75
... Selecting 4 locations from 75 potentials.
... continued output...
Auto-detected package structures
The following package structures would be auto-detected if you ran mutatest
from the
same directory holding examplepkg/
. You can always point to a specific directory using
the --source
argument. These are outlined in the Pytest Test Layout [2] documentation.
Example with internal tests
.
└── examplepkg
├── __init__.py
├── run.py
└── test_run.py
Example with external tests
.
├── examplepkg
│ ├── __init__.py
│ └── run.py
└── tests
└── test_run.py
Selecting a running mode
mutatest
has different running modes to make trials faster. The running modes determine
what will happen after a mutation trial. For example, you can choose to stop further mutations at a
location as soon as a survivor is detected. The different running mode choices are:
- Run modes:
f: full mode, run all possible combinations (slowest but most thorough).
s: break on first SURVIVOR per mutated location e.g. if there is a single surviving mutation at a location move to the next location without further testing. This is the default mode.
d: break on the first DETECTION per mutated location e.g. if there is a detected mutation on at a location move to the next one.
sd: break on the first SURVIVOR or DETECTION (fastest, and least thorough).
The API for mutatest.controller.run_mutation_trials
offers finer control over the run
method beyond the CLI.
A good practice when first starting is to set the mode to sd
which will stop if a mutation
survives or is detected, effectively running a single mutation per candidate location. This is the
fastest running mode and can give you a sense of investigation areas quickly.
$ mutatest --mode sd
# using shorthand arguments
$ mutatest -m sd
Controlling randomization behavior and trial number
mutatest
uses random sampling of all source candidate locations and of potential mutations
to substitute at a location. You can set a random seed for repeatable trials using the
--rseed
argument. The --nlocations
argument controls the size of the sample
of locations to mutate. If it exceeds the number of candidate locations then the full set of
candidate locations is used.
$ mutatest --nlocations 5 --rseed 314
# using shorthand arguments
$ mutatest -n 5 -r 314
Setting the output location
By default, mutatest
will only create CLI output to stdout
.
You can set path location using the --output
argument for a written RST report of the
mutation trial results.
$ mutatest --output path/to/my_custom_file.rst
# using shorthand arguments
$ mutatest -o path/to/my_custom_file.rst
The output report will include the arguments used to generate it along with the total runtimes. The SURVIVORS section of the output report is the one you should pay attention to. These are the mutations that were undetected by your test suite. The report includes file names, line numbers, column numbers, original operation, and mutation for ease of diagnostic investigation.
Putting it all together
If you want to run 5 trials, in fast sd
mode, with a random seed of 345 and an output
file name of mutation_345.rst
, you would do the following if your directory structure
has a Python package folder and tests that are auto-discoverable and run by pytest
.
$ mutatest -n 5 -m sd -r 345 -o mutation_345.rst
With coverage
optimization if your pytest.ini
file does not already specify it:
$ mutatest -n 5 -m sd -r 345 -o mutation_345.rst -t "pytest --cov=mypackage"
Getting help
Run mutatest --help
to see command line arguments and supported operations:
$ mutatest --help
usage: Mutatest [-h] [-e STR_LIST] [-m {f,s,d,sd}] [-n INT] [-o PATH] [-r INT]
[-s PATH] [-t STR_CMDS] [--debug]
Python mutation testing. Mutatest will manipulate local __pycache__ files.
optional arguments:
-h, --help show this help message and exit
-e PATH, --exclude PATH
Path to .py file to exclude, multiple -e entries supported. (default: None)
-m {f,s,d,sd}, --mode {f,s,d,sd}
Running modes, see the choice option descriptions below. (default: s)
-n INT, --nlocations INT
Number of locations in code to randomly select for mutation from possible targets. (default: 10)
-o PATH, --output PATH
Output file location for results. (default: mutation_report.rst)
-r INT, --rseed INT Random seed to use for sample selection.
-s PATH, --src PATH Source code (file or directory) for mutation testing. (default: auto-detection attempt).
-t STR_CMDS, --testcmds STR_CMDS
Test command string to execute. (default: 'pytest')
--debug Turn on DEBUG level logging output.
--nocov Ignore coverage files for optimization.
Mutations
mutatest
is early in development and supports the following mutation operations based
on the Python AST grammar [3]:
- Supported operations:
AugAssign
mutations e.g.+= -= *= /=
.BinOp
mutations e.g.+ - / *
.BinOp Bitwise Comparison
mutations e.g.x&y x|y x^y
.BinOp Bitwise Shift
mutations e.g.<< >>
.BoolOp
mutations e.g.and or
.Compare
mutations e.g.== >= < <= !=
.Compare In
mutations e.g.in, not in
.Compare Is
mutations e.g.is, is not
.Index
mutations e.g.i[0]
becomesi[1]
andi[-1]
.NameConstant
mutations e.g.True
,False
, andNone
.Slice
mutations e.g. changingx[:2]
tox[2:]
These are the current operations that are mutated as compatible sets.
AugAssign
Augmented assignment e.g. += -= /= *=
.
- Members:
AugAssign_Add
AugAssign_Div
AugAssign_Mult
AugAssign_Sub
Example:
# source code
x += y
# mutations
x -= y # AugAssign_Sub
x *= y # AugAssign_Mult
x /= y # AugAssign_Div
BinOp
Binary operations e.g. add, subtract, divide, etc.
- Members:
ast.Add
ast.Div
ast.FloorDiv
ast.Mod
ast.Mult
ast.Pow
ast.Sub
Example:
# source code
x = a + b
# mutations
x = a / b # ast.Div
x = a - b # ast.Sub
BinOp Bit Comparison
Bitwise comparison operations e.g. x & y, x | y, x ^ y
.
- Members:
ast.BitAnd
ast.BitOr
ast.BitXor
Example:
# source code
x = a & y
# mutations
x = a | y # ast.BitOr
x = a ^ y # ast.BitXor
BinOp Bit Shifts
Bitwise shift operations e.g. << >>
.
- Members:
ast.LShift
ast.RShift
Example:
# source code
x >> y
# mutation
x << y
BoolOp
Boolean operations e.g. and or
.
- Members:
ast.And
ast.Or
Example:
# source code
if x and y:
# mutation
if x or y:
Compare
Comparison operations e.g. == >= <= > <
.
- Members:
ast.Eq
ast.Gt
ast.GtE
ast.Lt
ast.LtE
ast.NotEq
Example:
# source code
x >= y
# mutations
x < y # ast.Lt
x > y # ast.Gt
x != y # ast.NotEq
Compare In
Compare membership e.g. in, not in
.
- Members:
ast.In
ast.NotIn
Example:
# source code
x in [1, 2, 3, 4]
# mutation
x not in [1, 2, 3, 4]
Compare Is
Comapre identity e.g. is, is not
.
- Members:
ast.Is
ast.IsNot
Example:
# source code
x is None
# mutation
x is not None
Index
Index values for iterables e.g. i[-1], i[0], i[0][1]
. It is worth noting that this is a
unique mutation form in that any index value that is positive will be marked as Index_NumPos
and the same relative behavior will happen for negative index values to Index_NumNeg
. During
the mutation process there are three possible outcomes: the index is set to 0, -1 or 1.
The alternate values are chosen as potential mutations e.g. if the original operation is classified
as Index_NumPos
such as x[10]
then valid mutations are to x[0]
or
x[-1]
.
- Members:
Index_NumNeg
Index_NumPos
Index_NumZero
Example:
# source code
x = [a[10], a[-4], a[0]]
# mutations
x = [a[-1], a[-4], a[0]] # a[10] mutated to Index_NumNeg
x = [a[10], a[0], a[0]] # a[-4] mutated to Index_NumZero
x = [a[10], a[1], a[0]] # a[-4] mutated to Index_NumPos
x = [a[10], a[-4], a[1]] # a[0] mutated to Index_NumPos
NameConstant
Named constant mutations e.g. True, False, None
.
- Members:
False
None
True
Example:
# source code
x = True
# mutations
x = False
X = None
Slices
Slice mutations to swap lower/upper values, or change range e.g. x[2:] to x[:2]
or x[1:5] to x[1:4]
. This is a unique mutation. If the upper or lower bound is set to
None
then the bound values are swapped. This is represented by the operations of
Slice_UnboundedUpper
for swap None to the “upper” value from “lower”.
The “ToZero” operations
change the list by moving the upper bound by one unit towards zero from the absolute value and
then applying the original sign e.g. x[0:2]
would become x[0:1]
, and
x[-4:-1]
would become x[-4:0]
. In the positive case, which is assumed to be the
more common pattern, this results in shrinking the index slice by 1. Note that these “ToZero”
operations appear self-referential in the report output. This is because an operation identified
as a Slice_UPosToZero
remains as a Slice_UPosToZero
but with updated values.
- Members:
Slice_Unbounded
Slice_UnboundedLower
Slice_UnboundedUpper
Slice_UNegToZero
Slice_UPosToZero
Example:
# source code
w = a[:2]
x = a[4:]
y = a[1:5]
z = a[-5:-1]
# mutation
w = a[2:] # Slice_UnboundedUpper, upper is now unbounded and lower has a value
x = a[4:]
y = a[1:5]
z = a[-5:-1]
# mutation
w = a[:2]
x = a[:4] # Slice_UnboundedLower, lower is now unbounded and upper has a value
y = a[1:5]
z = a[-5:-1]
# mutation
w = a[:2]
x = a[:] # Slice_Unbounded, both upper and lower are unbounded
y = a[1:5]
z = a[-5:-1]
# mutation
w = a[:2]
x = a[4:]
y = a[1:4] # Slice_UPosToZero, upper bound moves towards zero bound by 1 when positive
z = a[-5:-1]
# mutation
w = a[:2]
x = a[4:]
y = a[1:5]
z = a[-5:0] # Slice_UNegToZero, upper bound moves by 1 from absolute value when negative
Known limitations
Since mutatest
operates on the local __pycache__
it is a serial execution process.
This means it will take as long as running your test suite in series for the
number of operations. It’s designed as a diagnostic tool, and you should try to find the combination
of test commands, source specifiers, and exclusions that generate meaningful diagnostics.
For example, if you have 600 tests, running mutatest
over the entire test suite may take
some time. A better strategy would be:
Select a subset of your tests and run
pytest
withcoverage
to see the covered percentage per source file.Run
mutatest
with the samepytest
command passed in with-t
and generating a coverage file. Use-s
to pick the source file of interest to restrict the sample space, or use-e
to exclude files if you want to target multiple files.
If you kill the mutatest
process before the trials complete you may end up
with partially mutated __pycache__
files. If this happens the best fix is to remove the
__pycache__
directories and let them rebuild automatically the next time your package is
imported (for instance, by re-running your test suite).
The mutation status is based on the return code of the test suite e.g. 0 for success, 1 for failure.
mutatest
can theoretically be run with any test suite that you pass with the
--testcmds
argument; however, only pytest
has been tested to date. The
mutatest.maker.MutantTrialResult
namedtuple contains the definitions for translating
return codes into mutation trial statuses.
Changelog
mutatest
is alpha software, and backwards compatibility between releases is
not guaranteed while under development.
0.8.0
- Breaking changes to the CLI arguments and new defaults:
Output files are now optional, the default behavior has changed from always writing an RST file using the
-o
option on the command line.Exclusions are still marked as
-e
; however, now multiple-e
arguments are supported and arguments must point to a Python file. The argument used to be:mutatest -e "__init__.py _devtools.py"
and now it ismutatest -e src/__init__.py -e src/_devtools.py
. There are no longer default exclusions applied.Improved CLI reporting, including selected test counts and line/col locations for trial results while processing.
0.7.1
Internal changes to
Slice
mutations for clearer categorization and report output.Includes clearing names to
Slice_Swap
andSlice_RangeChange
for categories.Updates operation names to
Slice_Unbounded...
with “lower” or “upper”.
0.7.0
- Added new slice mutations:
Slice_SwapNoneUL
andSlice_SwapNoneLU
for swapping the upper and lower bound values when only one is specified e.g.x[1:]
tox[:1]
.
Slice_UPosToZero
andSlice_UNegToZero
for moving the upper bound of a slice by 1 unit e.g.x[1:5]
becomesx[1:4]
.
0.6.1
Added explicit tests for
argparse
cli options.Added mechanism to sort reporting mutations by source file, then line number, then column number.
0.6.0
Including
pytest
in the installation requirements. Technically, any test runner can be used but with all base package assumptions being built aroundpytest
this feels like the right assumption to call out as an install dependency. It is the default behavior.Updated
controller
for test file exclusion to explicitly match prefix or suffix cases for"test_"
and"_test"
perpytest
conventions.Changed error and unknown status results to console color as yellow instead of red.
Added multiple invariant property tests, primarily to
controller
andcache
.Added
hypothesis
to the test components ofextras_require
.Moved to
@property
decorators for internal class properties that should only be set at initialization, may add customsetters
at a later time.Fixed a zero-division bug in the
cli
when reporting coverage percentage.
0.5.0
Addition of
optimizers
, including the new classCoverageOptimizer
.This optimizer restricts the full sample space only to source locations that are marked as covered in the
.coverage
file. If you have apytest.ini
that includes the--cov=
command it will automatically generate during the clean-trial run.
0.4.2
More behind the scenes maintenance: updated debug level logging to include source file names and line numbers for all visit operations and separated colorized output to a new function.
0.4.1
Updated the reporting functions to return colorized display results to CLI.
0.4.0
- Added new mutation support for:
AugAssign
in AST e.g.+= -= *= /=
.
Index
substitution in lists e.g. take a positive number likei[1]
and mutate to zero and a negative number e.g.i[-1] i[0]
.Added a
desc
attribute totransformers.MutationOpSet
that is used in the cli help display.Updated the cli help display to show the description and valid members.
0.3.0
Added new mutation support for
NameConstant
in AST.This includes substitutions for singleton assignments such as:
True
,False
, andNone
.This is the first non-type mutation and required adding a
readonly
parameter to thetransformers.MutateAST
class. Additionally, the type-hints for theLocIndex
andMutationOpSet
were updated toAny
to support the mixed types. This was more flexible than a series ofoverload
signatures.
0.2.0
- Added new compare mutation support for:
Compare Is
mutations e.g.is, is not
.
Compare In
mutations e.g.in, not in
.
0.1.0
Initial release!
Requires Python 3.7 due to the
importlib
internal references for manipulating cache.Run mutation tests using the
mutatest
command line interface.Supported operations:
BinOp
mutations e.g.+ - / *
including bit-operations.
Compare
mutations e.g.== >= < <= !=
.
BoolOp
mutations e.g.and or
.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for mutatest-0.8.0-py37-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 19cfab90dad5e507e911aafa5fe742eb37f2cecc00906ab7f0938ea1fcd4b088 |
|
MD5 | 439fece5419fb92ce55c45e7c56847a2 |
|
BLAKE2b-256 | b4cf9cf3313a6c1f6e3ec7bc21597e247c2774eea3400b17edcebd21c58c71c6 |