Skip to main content

Near Zero-Overhead Python Code Coverage

Project description

slipcover

SlipCover: Near Zero-Overhead Python Code Coverage

by Juan Altmayer Pizzorno and Emery Berger at UMass Amherst's PLASMA lab.

license pypi Downloads pyversions tests

About Slipcover

SlipCover is a fast code coverage tool. It tracks a Python program as it runs and reports on the parts that executed and those that didn't. That can help guide your testing (showing code that isn't being tested), debugging, fuzzing or to find "dead" code.

Past code coverage tools can make programs significantly slower; it is not uncommon for them to take twice as long to execute. SlipCover aims to provide the same information with near-zero overhead, often almost as fast as running the original Python program.

How it works

Previous coverage tools like Coverage.py rely on Python's tracing facilities, which add significant overhead. Instead, SlipCover uses just-in-time instrumentation and de-instrumentation. When SlipCover gathers coverage information, it modifies the program's Python byte codes, inserting instructions that let it keep track the lines executed by the program. As the program executes, SlipCover gradually removes instrumentation that is no longer needed, allowing those parts to run at full speed. Care is taken throughout SlipCover to keep things as efficient as possible. On Python 3.12, rather than rewrite bytecode, SlipCover uses the new sys.monitoring API to collect coverage information.

Performance

The first image on the right shows SlipCover's speedup, ranging from 1.1x to 3.4x, in relation to Coverage.py, running on CPython 3.10.5.

The first two benchmarks are the test suites for scikit-learn and Flask; "sudoku" runs Peter Norvig's Sudoku solver while the others were derived from the Python Benchmark Suite.

More "Python-intensive" programs such as sudoku and those from the benchmark suite (with a larger proportion of execution time spent in Python, rather than in native code) generate more tracing events, causing more overhead in Coverage.py. While each program's structure can affect SlipCover's ability to de-instrument, its running time stays relatively close to the original.

On PyPy 3.9, the speedup ranges from 2.1x to 104.9x. Since it is so high for some of the benchmarks, we plot it on a logarithmic scale (see the second image on the right).

In a proof-of-concept integration with a property-based testing package, SlipCover sped up coverage-based testing 22x.

Accuracy

We verified SlipCover's accuracy against Coverage.py and against a simple script of our own that collects coverage using Python tracing. We found SlipCover's results to be accurate, in fact, in certain cases more accurate.

Getting started

SlipCover is available from PyPI. You can install it like any other Python module with

pip3 install slipcover

You could then run your Python script with:

python3 -m slipcover myscript.py

Using it with a test harness

SlipCover can also execute a Python module, as in:

python3 -m slipcover -m pytest -x -v

which starts pytest, passing it any options (-x -v in this example) after the module name. No plug-in is required for pytest.

Usage example

$ python3 -m slipcover -m pytest
================================================================ test session starts ================================================================
platform darwin -- Python 3.9.12, pytest-7.1.2, pluggy-1.0.0
rootdir: /Users/juan/project/wally/d2k-5, configfile: pytest.ini
plugins: hypothesis-6.39.3, mock-3.7.0, repeat-0.9.1, doctestplus-0.12.0, arraydiff-0.5.0
collected 439 items                                                                                                                                 

tests/box_test.py .........................                                                                                                   [  5%]
tests/image_test.py ...............                                                                                                           [  9%]
tests/network_equivalence_test.py .........................................s................................................................. [ 33%]
..............................................................................                                                                [ 51%]
tests/network_test.py ....................................................................................................................... [ 78%]
...............................................................................................                                               [100%]

=================================================== 438 passed, 1 skipped, 62 warnings in 48.43s ====================================================

File                                 #lines    #miss    Cover%  Lines missing
---------------------------------  --------  -------  --------  ------------------------
d2k/__init__.py                           3        0       100
d2k/box.py                              105       27        74  73, 142-181
d2k/image.py                             38        4        89  70-73
d2k/network.py                          359        1        99  236
tests/box_test.py                       178        0       100
tests/darknet.py                        132       11        91  146, 179-191
tests/image_test.py                      45        0       100
tests/network_equivalence_test.py       304       30        90  63, 68, 191-215, 455-465
tests/network_test.py                   453        0       100
$ 

As can be seen in the coverage report, d2k lacks some coverage, especially in its box.py and image.py components.

Platforms

Our GitHub workflows run the automated test suite on Linux, MacOS and Windows, but really it should work anywhere where CPython/PyPy does.

Contributing

SlipCover is under active development; contributions are welcome! Please also feel free to create a new issue with any suggestions or issues you may encounter.

Technical Information

For more details about how SlipCover works please see the following paper, published at ISSTA'23: SlipCover: Near Zero-Overhead Code Coverage for Python.

Acknowledgements

Logo design by Sophia Berger.

This material is based upon work supported by the National Science Foundation under Grant No. 1955610. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

slipcover-1.0.15.tar.gz (52.4 kB view details)

Uploaded Source

Built Distributions

slipcover-1.0.15-py312-none-any.whl (30.1 kB view details)

Uploaded Python 3.12

slipcover-1.0.15-cp310-abi3-win_amd64.whl (86.0 kB view details)

Uploaded CPython 3.10+ Windows x86-64

slipcover-1.0.15-cp310-abi3-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl (48.1 kB view details)

Uploaded CPython 3.10+ manylinux: glibc 2.24+ x86-64 manylinux: glibc 2.28+ x86-64

slipcover-1.0.15-cp310-abi3-macosx_14_0_universal2.whl (40.5 kB view details)

Uploaded CPython 3.10+ macOS 14.0+ universal2 (ARM64, x86-64)

slipcover-1.0.15-cp310-abi3-macosx_13_0_universal2.whl (40.9 kB view details)

Uploaded CPython 3.10+ macOS 13.0+ universal2 (ARM64, x86-64)

slipcover-1.0.15-cp39-cp39-win_amd64.whl (86.1 kB view details)

Uploaded CPython 3.9 Windows x86-64

slipcover-1.0.15-cp39-cp39-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl (50.9 kB view details)

Uploaded CPython 3.9 manylinux: glibc 2.24+ x86-64 manylinux: glibc 2.28+ x86-64

slipcover-1.0.15-cp39-cp39-macosx_14_0_universal2.whl (40.5 kB view details)

Uploaded CPython 3.9 macOS 14.0+ universal2 (ARM64, x86-64)

slipcover-1.0.15-cp39-cp39-macosx_13_0_universal2.whl (41.0 kB view details)

Uploaded CPython 3.9 macOS 13.0+ universal2 (ARM64, x86-64)

slipcover-1.0.15-cp38-cp38-win_amd64.whl (86.1 kB view details)

Uploaded CPython 3.8 Windows x86-64

slipcover-1.0.15-cp38-cp38-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl (51.5 kB view details)

Uploaded CPython 3.8 manylinux: glibc 2.24+ x86-64 manylinux: glibc 2.28+ x86-64

slipcover-1.0.15-cp38-cp38-macosx_14_0_universal2.whl (40.8 kB view details)

Uploaded CPython 3.8 macOS 14.0+ universal2 (ARM64, x86-64)

slipcover-1.0.15-cp38-cp38-macosx_13_0_universal2.whl (41.0 kB view details)

Uploaded CPython 3.8 macOS 13.0+ universal2 (ARM64, x86-64)

File details

Details for the file slipcover-1.0.15.tar.gz.

File metadata

  • Download URL: slipcover-1.0.15.tar.gz
  • Upload date:
  • Size: 52.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.0 CPython/3.12.3

File hashes

Hashes for slipcover-1.0.15.tar.gz
Algorithm Hash digest
SHA256 05a3cab36b2a15cbc42a8c3a1094e74f9502866e772be608b683daffd39c8a70
MD5 683c84e1203ef091616a8558e85c68f5
BLAKE2b-256 5b17ba04830fc224b898071f944bf74549ed08c8d1d3b0af8ed175a15b8234b5

See more details on using hashes here.

File details

Details for the file slipcover-1.0.15-py312-none-any.whl.

File metadata

  • Download URL: slipcover-1.0.15-py312-none-any.whl
  • Upload date:
  • Size: 30.1 kB
  • Tags: Python 3.12
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.0 CPython/3.12.3

File hashes

Hashes for slipcover-1.0.15-py312-none-any.whl
Algorithm Hash digest
SHA256 573851ce547b6086dfe4ed48c65c3663c6d9f149e2f62bb69fb3931c9753de14
MD5 47ded37f3129d8240b5413a8edb5303c
BLAKE2b-256 e5b98e05a0ea1a275bd68f3918f5227fc4201de7b368882f20c9d85a1bc5479e

See more details on using hashes here.

File details

Details for the file slipcover-1.0.15-cp310-abi3-win_amd64.whl.

File metadata

File hashes

Hashes for slipcover-1.0.15-cp310-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 052fd9a0fd8d5da19d33f4e51a01ca8cb6ca98db5d703bd2b772cf10eb7844ac
MD5 256cc90787fe063258fa16bd93cafd4a
BLAKE2b-256 d6e6c43a72b4ff8537d25dc175ce4558b2ee3fe1f6d2694c8dc7c9915066b685

See more details on using hashes here.

File details

Details for the file slipcover-1.0.15-cp310-abi3-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for slipcover-1.0.15-cp310-abi3-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 1f56ae4eec9a6e6fa9e0023c06a14c759a36e90705babd74d5a42249fadf9f1e
MD5 8ee14b81468b78e8549c972a8018ffc3
BLAKE2b-256 f330209f27dd1a8397793754e875c37dfd8747ad63318cd13d7dbf948262b530

See more details on using hashes here.

File details

Details for the file slipcover-1.0.15-cp310-abi3-macosx_14_0_universal2.whl.

File metadata

File hashes

Hashes for slipcover-1.0.15-cp310-abi3-macosx_14_0_universal2.whl
Algorithm Hash digest
SHA256 107c480e560d2e9241faadc5dc303f796ab84009a35fb773f76254069572464b
MD5 85ef30941ca6a4ecea48f3754a7e7963
BLAKE2b-256 4e48c991ce7487f1c193e268568421099ed5443cb10fe0d30c2875ee3adfd9ac

See more details on using hashes here.

File details

Details for the file slipcover-1.0.15-cp310-abi3-macosx_13_0_universal2.whl.

File metadata

File hashes

Hashes for slipcover-1.0.15-cp310-abi3-macosx_13_0_universal2.whl
Algorithm Hash digest
SHA256 7d21b341115e3853dffce8859be89af8b9a407849fbe4db6af5b4dfefed2bc20
MD5 5571e1f73e17dd89487a3509d5dc20cd
BLAKE2b-256 ff2303d34e7e0a319adf9e703e04fb707f714bd5176ff904087698f470598653

See more details on using hashes here.

File details

Details for the file slipcover-1.0.15-cp39-cp39-win_amd64.whl.

File metadata

File hashes

Hashes for slipcover-1.0.15-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 e01873c444a4bb4b89f829561d6d37a0c4939dcd47612f39d2fb0a4d90863d2d
MD5 e647a52207c80427aeda527b69a71a3a
BLAKE2b-256 97713241d0681d9c89e41a7ef9e3a341b3e20cd2bc8398ca492c6d98f03fc9bd

See more details on using hashes here.

File details

Details for the file slipcover-1.0.15-cp39-cp39-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for slipcover-1.0.15-cp39-cp39-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 b31d4cd91d23dd444fc1920cae6f13c4cb102a35be6e1064574e9a4f0a9b9380
MD5 03a3432ad67d9aab02b3aa62364911da
BLAKE2b-256 456647143277da738782ba2a0d1d2516844d68ad8ceaa9ba13fe88aa4c89f3dd

See more details on using hashes here.

File details

Details for the file slipcover-1.0.15-cp39-cp39-macosx_14_0_universal2.whl.

File metadata

File hashes

Hashes for slipcover-1.0.15-cp39-cp39-macosx_14_0_universal2.whl
Algorithm Hash digest
SHA256 328f9c206df82e7101e35a05916a2ecb1bc3d2f951a91f29471ee7f517620719
MD5 b63667be992147f257fb31b19aae8e4d
BLAKE2b-256 fef3bf386340e401cff94763ec1601f7fb40a084caabfba9e1ce6143e0b5fb9e

See more details on using hashes here.

File details

Details for the file slipcover-1.0.15-cp39-cp39-macosx_13_0_universal2.whl.

File metadata

File hashes

Hashes for slipcover-1.0.15-cp39-cp39-macosx_13_0_universal2.whl
Algorithm Hash digest
SHA256 8867ef33506034906b434b4301e1327301f88b9e7bdf5b23280e9030ff5bf338
MD5 7868eaedd91c5b0dbb72edc57456f959
BLAKE2b-256 68b3bd9e48b1315b7b8eaecde583472c599877bbe6a624af6c9f86e25fc23c84

See more details on using hashes here.

File details

Details for the file slipcover-1.0.15-cp38-cp38-win_amd64.whl.

File metadata

File hashes

Hashes for slipcover-1.0.15-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 88f3b03d4d1803272a8c88987df845600833b2e090d67a876f8c30f9883f9ebd
MD5 70d51fa14787567e0dc8c9f303b02817
BLAKE2b-256 371c5244f11c1d7d00cae46f4fb1c89ea1892a9317b94cfac169cf132b1f7e47

See more details on using hashes here.

File details

Details for the file slipcover-1.0.15-cp38-cp38-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for slipcover-1.0.15-cp38-cp38-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 fffb53c52c64fc467d94507c1392d6592a5af17b9112eba42da0657209cff33e
MD5 05d79b2667f8f57d7fc7a0daf5fcacea
BLAKE2b-256 d0e12273d69a013f719fa67b87fa739d6af5c37cd821d8b8c44f022b1623a18a

See more details on using hashes here.

File details

Details for the file slipcover-1.0.15-cp38-cp38-macosx_14_0_universal2.whl.

File metadata

File hashes

Hashes for slipcover-1.0.15-cp38-cp38-macosx_14_0_universal2.whl
Algorithm Hash digest
SHA256 b5db4a27b66783b2f211eff2a9f80ad7c3adf6ea0e05a10bd5b9193117879d67
MD5 eb2e79162fb2ad246df2bacfd1b076c1
BLAKE2b-256 ecde93783f3547de25f9f69309519b0c3fa07851083769663facf16a54b96ea9

See more details on using hashes here.

File details

Details for the file slipcover-1.0.15-cp38-cp38-macosx_13_0_universal2.whl.

File metadata

File hashes

Hashes for slipcover-1.0.15-cp38-cp38-macosx_13_0_universal2.whl
Algorithm Hash digest
SHA256 5224372fd1b014c7c1b02f0ca11a52280841902e254e55c3b66269cb20e65d65
MD5 de99a062c908671c2eddc19f2d16b980
BLAKE2b-256 3fcfa72b86a60551cc2b78bbab8f27ee48b771e65e75f75b7a2c92957bac57b9

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page