Skip to main content

Python memory tracing.

Project description

pymemtrace provides tools for tracking and understanding Python memory usage at different levels, at different granularities and with different runtime costs.

Full documentation: https://pymemtrace.readthedocs.io

pymemtrace Tools

The tools provided by pymemtrace:

  • process is a very lightweight way of logging the total memory usage at regular time intervals. It can plot memory over time with plotting programs such as gnuplot. See some process examples
  • cPyMemTrace is a memory tracer written in C that can report total memory usage for every function call/return for both C and Python sections. See some cPyMemTrace examples and a technical note on cPyMemTrace.
  • DTrace: Here are a number of D scripts that can trace the low level malloc() and free() system calls and report how much memory was allocated and by whom. See some DTrace examples and a technical note on DTrace.
  • trace_malloc is a convenience wrapper around the Python standard library tracemalloc module. This can report Python memory usage by module and line compensating for the cost of tracemalloc. This can take memory snapshots before and after code blocks and show the change on memory caused by that code. See some trace_malloc examples
  • debug_malloc_stats is a wrapper around the sys._debugmallocstats function that can take snapshots of memory before and after code execution and report the significant differences of the Python small object allocator. See some debug_malloc_stats examples

Tool Characteristics

Each tool can be characterised by:

  • Memory Granularity: In how much detail is a memory change is observed. An example of coarse memory granularity is measuring the Resident Set Size which is normally in chunks of 4096 bytes. An example of fine memory granularity is recording every malloc() and free().
  • Execution Granularity: In how much code detail is the memory change observed. An example of coarse execution granularity is measuring the memory usage every second. An example of fine execution granularity is recording the memory usage every Python line.
  • Memory Cost: How much extra memory the tool needs.
  • Execution Cost: How much the execution time is increased.

Clearly there are trade-offs between these depending on the problem you are trying to solve.

Tool Characteristics
Tool Memory Granularity Execution Granularity Memory Cost Execution Cost
process RSS (total Python and C memory). Regular time intervals. Near zero. Near zero.
cPyMemTrace RSS (total Python and C memory). Per Python line, Python function and C function call. Near zero. x10 to x20.
DTrace Every malloc() and free(). Per function call and return. Minimal. x90 to x100.
trace_malloc Every Python object. Per Python line, per function call. Significant but compensated. x900 for small objects, x6 for large objects.
debug_malloc_stats Python memory pool. Snapshots the CPython memory pool either side of a block of code. Minimal. x2000+ for small objects, x12 for large objects.

Package Metadata

https://img.shields.io/pypi/v/pymemtrace.svg https://img.shields.io/travis/paulross/pymemtrace.svg Documentation Status Updates

Licence

Python memory tracing.

Credits

Phil Smith (AHL) with whom a casual lunch time chat lead to the creation of an earlier, but quite different implementation, of cPyMemTrace in pure Python.

This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.

History

0.1.1 (2020-11-17)

  • Add cPyMemTrace the C level profiler.
  • Add DTrace scripts for low level tracing.
  • Add debug_malloc_stats the wrapper around sys._debugmallocstats.
  • Add process from the TotalDepth project.
  • Add redirect_stdout for debug_malloc_stats.
  • Add trace_malloc, a wrapper around the tracemalloc module.
  • Includes extensive documentation and performance measurement.
  • First release on PyPI.

0.1.0 (2017-12-04)

  • Initial idea and implementation, never released.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for pymemtrace, version 0.1.1
Filename, size File type Python version Upload date Hashes
Filename, size pymemtrace-0.1.1.tar.gz (368.9 kB) File type Source Python version None Upload date Hashes View
Filename, size pymemtrace-0.1.1-cp39-cp39-macosx_10_9_x86_64.whl (59.5 kB) File type Wheel Python version cp39 Upload date Hashes View
Filename, size pymemtrace-0.1.1-cp38-cp38-macosx_10_9_x86_64.whl (61.3 kB) File type Wheel Python version cp38 Upload date Hashes View
Filename, size pymemtrace-0.1.1-cp37-cp37m-macosx_10_9_x86_64.whl (59.3 kB) File type Wheel Python version cp37 Upload date Hashes View
Filename, size pymemtrace-0.1.1-cp36-cp36m-macosx_10_6_intel.whl (67.4 kB) File type Wheel Python version cp36 Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page