Skip to main content

A low-overhead memory profiler.

Project description

PyPI Build Status PyPI - License PyPI - Python Version PyPI - Downloads Code style: black

mprofile

A low-overhead sampling memory profiler for Python, derived from heapprof, with an interface similar to tracemalloc. mprofile attempts to give results comparable to tracemalloc, but uses statistical sampling to lower memory and CPU overhead. The sampling algorithm is the one used by tcmalloc and Golang heap profilers.

Installation & usage

  1. Install the profiler package using PyPI:

    pip3 install mprofile
    
  2. Enable the profiler in your application, get a snapshot of (sampled) memory usage:

    import mprofile
    
    mprofile.start(sample_rate=128 * 1024)
    snap = mprofile.take_snapshot()
    

See the tracemalloc for API documentation. The API and objects returned by mprofile are compatible.

Compatibility

mprofile is compatible with Python >= 3.4. It can also be used with earlier versions of Python, but you must build CPython from source and apply the pytracemalloc patches.

Benchmarks

We are primarily interested in profiling the memory usage of webservers, so used the tornado_http benchmark from pyperformance to estimate overhead. mprofile has similar performance to tracemalloc when comprehensively tracing all allocations, but when statistical sampling is used, the overhead is significantly reduced. In addition, mprofile interns call stacks in a tree data structure that reduces memory overhead of storing the traces.

With the recommended setting of sample_rate=128kB, we observe ~5% slow down in the tornado_http benchmark.

TODO: Run the full pyperformance suite of benchmarks.

Baseline

Python 2.7.16, no profiling:
tornado_http: Mean +- std dev: 664 ms +- 30 ms
Maximum resident set size (kbytes): 39176

tracemalloc

Python 2.7.16, tracemallocframes=128:
tornado_http: Mean +- std dev: 1.74 sec +- 0.04 sec
Maximum resident set size (kbytes): 43752

# Saving only one frame in each stack trace rather than full call stacks.
Python 2.7.16, tracemallocframes=1:
tornado_http: Mean +- std dev: 960 ms +- 30 ms
Maximum resident set size (kbytes): 40000

mprofile

Python 2.7.16, mprofileframes=128, mprofilerate=1 (i.e. tracemalloc):
tornado_http: Mean +- std dev: 1.78 sec +- 0.05 sec
Maximum resident set size (kbytes): 40588

Python 2.7.16, mprofileframes=128, mprofilerate=1024:
tornado_http: Mean +- std dev: 888 ms +- 28 ms
Maximum resident set size (kbytes): 39752

Python 2.7.16, mprofileframes=128, mprofilerate=128 * 1024:
tornado_http: Mean +- std dev: 700 ms +- 26 ms
Maximum resident set size (kbytes): 39388

# Saving only one frame in each stack trace rather than full call stacks.
Python 2.7.16, mprofileframes=1, mprofilerate=1 (i.e. tracemalloc):
tornado_http: Mean +- std dev: 890 ms +- 19 ms
Maximum resident set size (kbytes): 40152

Python 2.7.16, mprofileframes=1, mprofilerate=1024:
tornado_http: Mean +- std dev: 738 ms +- 24 ms
Maximum resident set size (kbytes): 39568

Python 2.7.16, mprofileframes=1, mprofilerate=128 * 1024:
tornado_http: Mean +- std dev: 678 ms +- 22 ms
Maximum resident set size (kbytes): 39328

Developer notes

Run the unit tests:

bazel test --test_output=streamed //src:profiler_test

Run the benchmarks:

bazel test -c opt --test_output=streamed //src:profiler_bench

Run the end-to-end (Python) tests:

bazel test --config asan --test_output=streamed //test:*

Run tests with ASAN and UBSAN:

bazel test --config asan --test_output=streamed //src:* //test:*

Contributing

Pull requests and issues are welcomed!

License

mprofile is released under the MIT License and incorporates code from heapprof, which is also released under the MIT license.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

mprofile-0.0.15.tar.gz (126.1 kB view details)

Uploaded Source

Built Distributions

mprofile-0.0.15-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (837.4 kB view details)

Uploaded CPython 3.12 manylinux: glibc 2.17+ x86-64

mprofile-0.0.15-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (835.4 kB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

mprofile-0.0.15-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (833.8 kB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

mprofile-0.0.15-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (833.2 kB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

mprofile-0.0.15-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (833.6 kB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

mprofile-0.0.15-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (832.7 kB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ x86-64

mprofile-0.0.15-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (832.8 kB view details)

Uploaded CPython 3.6m manylinux: glibc 2.17+ x86-64

File details

Details for the file mprofile-0.0.15.tar.gz.

File metadata

  • Download URL: mprofile-0.0.15.tar.gz
  • Upload date:
  • Size: 126.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.0 CPython/3.10.7

File hashes

Hashes for mprofile-0.0.15.tar.gz
Algorithm Hash digest
SHA256 fb21d99dd99bc4672f057ac53bc6bd2ddc2139a18b83bc4565a0303ebd5b7efd
MD5 8ccfa9c642d8648adf30aa7fd3d2f9bd
BLAKE2b-256 f906a2a99bd3caf8daa6ce17154a19a2e05412496379f49cf28f654033072d4c

See more details on using hashes here.

File details

Details for the file mprofile-0.0.15-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for mprofile-0.0.15-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 6ad8ca31cc9b923be9e2ef35db83b0a6f2df14289d604577a9d09293b09957ae
MD5 00496e41f41c5674055c08bfc4dd5fa5
BLAKE2b-256 a4610aa0d3a9c454b9b7b880173706091e273634aa17a63f0e723eb0af4b69f5

See more details on using hashes here.

File details

Details for the file mprofile-0.0.15-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for mprofile-0.0.15-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 e33edf1d469723d33184e14a757aa3b202f54afb991da1d0b7f7fba5c49aef27
MD5 6344b2925219c21b02e62af286b385b2
BLAKE2b-256 6950e859b298acda9e01fe018826d8961d6ac3b4a32b499ded1c9cc49793c131

See more details on using hashes here.

File details

Details for the file mprofile-0.0.15-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for mprofile-0.0.15-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 31c893321be93910e80052d0e7dfc19c8083db153cd8b2bb31498dcd920a7da9
MD5 3c2d5b145b26f85cccdd9304c1c3e200
BLAKE2b-256 67c9909a46d1573198685be8fb3590396e02bc2cb195668c5b11ea7d86bb21c9

See more details on using hashes here.

File details

Details for the file mprofile-0.0.15-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for mprofile-0.0.15-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 6f5d4d7fad4005abae0f16f76ee234cffdad7cb9d763a103672b5ea6cd3f732d
MD5 7832a6c7083a830eb821bc0f0eeb0d44
BLAKE2b-256 3779d3cf2183fcae42197b576ce4fecfbe29226e3862318d2471f8b42e08ada2

See more details on using hashes here.

File details

Details for the file mprofile-0.0.15-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for mprofile-0.0.15-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 88218635ec951f1e449e23e96e0bf40b43a2ae3ca3b5c9f2506eb992515fca52
MD5 677471ab74ba6356733fc8d097516a0e
BLAKE2b-256 71b51db1430c5aeb79aae47927cb2a82f551e5d12ddbd7e5f23b95a93aa8bf95

See more details on using hashes here.

File details

Details for the file mprofile-0.0.15-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for mprofile-0.0.15-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 2bd8efc8d898d28181c404bfe304f13fd87ab8a439b6034235b5ea0f25670127
MD5 cf8f4e62dc0e61af1a6fdaf433774a90
BLAKE2b-256 6a2f3ea66c1fda13de4efbaab2d8522a7700112595e2103d4be22d303502ede4

See more details on using hashes here.

File details

Details for the file mprofile-0.0.15-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for mprofile-0.0.15-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 c544275c4d1373696987bedb516a36e56600ef43ab0920cf3fb53f18344190a6
MD5 03e2c3b16171f62080d0e81a4170afbf
BLAKE2b-256 57c5a82e119d6f3fe374a224c1dfc39081402fb78bc1ae95dd1d779162961c12

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page