Skip to main content

Python memory tracing.

Project description

pymemtrace provides tools for tracking and understanding Python memory usage at different levels, at different granularities and with different runtime costs.

Full documentation: https://pymemtrace.readthedocs.io

pymemtrace Tools

The tools provided by pymemtrace:

  • process is a very lightweight way of logging the total memory usage at regular time intervals. It can plot memory over time with plotting programs such as gnuplot. See some process examples

  • cPyMemTrace is a memory tracer written in C that can report total memory usage for every function call/return for both C and Python sections. See some cPyMemTrace examples and a technical note on cPyMemTrace.

  • DTrace: Here are a number of D scripts that can trace the low level malloc() and free() system calls and report how much memory was allocated and by whom. See some DTrace examples and a technical note on DTrace.

  • trace_malloc is a convenience wrapper around the Python standard library tracemalloc module. This can report Python memory usage by module and line compensating for the cost of tracemalloc. This can take memory snapshots before and after code blocks and show the change on memory caused by that code. See some trace_malloc examples

  • debug_malloc_stats is a wrapper around the sys._debugmallocstats function that can take snapshots of memory before and after code execution and report the significant differences of the Python small object allocator. See some debug_malloc_stats examples

Tool Characteristics

Each tool can be characterised by:

  • Memory Granularity: In how much detail is a memory change is observed. An example of coarse memory granularity is measuring the Resident Set Size which is normally in chunks of 4096 bytes. An example of fine memory granularity is recording every malloc() and free().

  • Execution Granularity: In how much code detail is the memory change observed. An example of coarse execution granularity is measuring the memory usage every second. An example of fine execution granularity is recording the memory usage every Python line.

  • Memory Cost: How much extra memory the tool needs.

  • Execution Cost: How much the execution time is increased.

Clearly there are trade-offs between these depending on the problem you are trying to solve.

Tool Characteristics

Tool

Memory Granularity

Execution Granularity

Memory Cost

Execution Cost

process

RSS (total Python and C memory).

Regular time intervals.

Near zero.

Near zero.

cPyMemTrace

RSS (total Python and C memory).

Per Python line, Python function and C function call.

Near zero.

x10 to x20.

DTrace

Every malloc() and free().

Per function call and return.

Minimal.

x90 to x100.

trace_malloc

Every Python object.

Per Python line, per function call.

Significant but compensated.

x900 for small objects, x6 for large objects.

debug_malloc_stats

Python memory pool.

Snapshots the CPython memory pool either side of a block of code.

Minimal.

x2000+ for small objects, x12 for large objects.

Package Metadata

https://img.shields.io/pypi/v/pymemtrace.svg https://img.shields.io/travis/paulross/pymemtrace.svg Documentation Status Updates

Licence

Python memory tracing.

Credits

Phil Smith (AHL) with whom a casual lunch time chat lead to the creation of an earlier, but quite different implementation, of cPyMemTrace in pure Python.

This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.

History

0.1.7 (2024-09-12)

  • Minor fix for a single test.

0.1.6 (2024-09-11)

  • Add support for Python versions 3.12, 3.13. Now supports Python versions 3.7, 3.8, 3.9, 3.10, 3.11, 3.12, 3.13.

0.1.5 (2023-06-21)

  • Add support for Python versions 3.10, 3.11. Now supports Python versions 3.7, 3.8, 3.9, 3.10, 3.11.

0.1.4 (2022-03-19)

  • Fix Linux build.

0.1.3 (2022-03-17)

  • Fix some tests.

0.1.2 (2022-03-17)

  • Fix source distribution that had missing headers.

0.1.1 (2020-11-17)

  • Add cPyMemTrace the C level profiler.

  • Add DTrace scripts for low level tracing.

  • Add debug_malloc_stats the wrapper around sys._debugmallocstats.

  • Add process from the TotalDepth project.

  • Add redirect_stdout for debug_malloc_stats.

  • Add trace_malloc, a wrapper around the tracemalloc module.

  • Includes extensive documentation and performance measurement.

  • First release on PyPI.

0.1.0 (2017-12-04)

  • Initial idea and implementation, never released.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pymemtrace-0.1.7.tar.gz (370.0 kB view details)

Uploaded Source

Built Distributions

pymemtrace-0.1.7-cp313-cp313-macosx_10_13_universal2.whl (73.9 kB view details)

Uploaded CPython 3.13 macOS 10.13+ universal2 (ARM64, x86-64)

pymemtrace-0.1.7-cp312-cp312-macosx_10_9_universal2.whl (73.9 kB view details)

Uploaded CPython 3.12 macOS 10.9+ universal2 (ARM64, x86-64)

pymemtrace-0.1.7-cp311-cp311-macosx_10_9_universal2.whl (73.7 kB view details)

Uploaded CPython 3.11 macOS 10.9+ universal2 (ARM64, x86-64)

pymemtrace-0.1.7-cp310-cp310-macosx_10_9_universal2.whl (73.7 kB view details)

Uploaded CPython 3.10 macOS 10.9+ universal2 (ARM64, x86-64)

pymemtrace-0.1.7-cp39-cp39-macosx_10_9_universal2.whl (73.7 kB view details)

Uploaded CPython 3.9 macOS 10.9+ universal2 (ARM64, x86-64)

pymemtrace-0.1.7-cp38-cp38-macosx_10_9_x86_64.whl (61.2 kB view details)

Uploaded CPython 3.8 macOS 10.9+ x86-64

pymemtrace-0.1.7-cp37-cp37m-macosx_10_9_x86_64.whl (61.0 kB view details)

Uploaded CPython 3.7m macOS 10.9+ x86-64

File details

Details for the file pymemtrace-0.1.7.tar.gz.

File metadata

  • Download URL: pymemtrace-0.1.7.tar.gz
  • Upload date:
  • Size: 370.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.11.6

File hashes

Hashes for pymemtrace-0.1.7.tar.gz
Algorithm Hash digest
SHA256 7511ac985926530facf8e0141e1c2f38f2e4963adaf56668d36adffb50f256ff
MD5 4590158f8d32005099e0d3967abc8657
BLAKE2b-256 6a1c890b494b279727b8b8261c4b2312ec52313ac96fe8951e528963ad3e9bc0

See more details on using hashes here.

File details

Details for the file pymemtrace-0.1.7-cp313-cp313-macosx_10_13_universal2.whl.

File metadata

File hashes

Hashes for pymemtrace-0.1.7-cp313-cp313-macosx_10_13_universal2.whl
Algorithm Hash digest
SHA256 d32e074755733f9da87276972c29f7da4bb7f8cd6bf9c69ca9e67846aa2247e0
MD5 ba57584444a88a58222aca5d22a85212
BLAKE2b-256 8849bd12e23ee0ceeee54307191867ee9d63a27868e29c6e1b47aab47ff7c072

See more details on using hashes here.

File details

Details for the file pymemtrace-0.1.7-cp312-cp312-macosx_10_9_universal2.whl.

File metadata

File hashes

Hashes for pymemtrace-0.1.7-cp312-cp312-macosx_10_9_universal2.whl
Algorithm Hash digest
SHA256 a7ae78ea5bedbd5b427636aa2f58fa08f38e64c5205674dd244fab801c0304d3
MD5 962427401ad1121b3847f8c947206295
BLAKE2b-256 cecf869a8770771359417b513748ecf975b61de858bc2539b8506232d3981e4e

See more details on using hashes here.

File details

Details for the file pymemtrace-0.1.7-cp311-cp311-macosx_10_9_universal2.whl.

File metadata

File hashes

Hashes for pymemtrace-0.1.7-cp311-cp311-macosx_10_9_universal2.whl
Algorithm Hash digest
SHA256 432823cfab79ce5b7d4b360ac36c57149d9cdcf8750ea544dae97302e88c32ff
MD5 7946e9006732e2ed4c8da96f875bc5c5
BLAKE2b-256 0d659a0947acf9fb056e90131f4cca766a01690e2b635a6e75725fdfc9beda17

See more details on using hashes here.

File details

Details for the file pymemtrace-0.1.7-cp310-cp310-macosx_10_9_universal2.whl.

File metadata

File hashes

Hashes for pymemtrace-0.1.7-cp310-cp310-macosx_10_9_universal2.whl
Algorithm Hash digest
SHA256 ba4e21b82a3e3e1e2e339166600e06a4b4ac4a7ec4dd4d691f65f7d9fcb3b931
MD5 60dc2cb92f0d1ac5612896a9d2608216
BLAKE2b-256 555016204714bebb8f145966803d7de73ff3c4cdf42a2aebe4a6d13a7abbe87c

See more details on using hashes here.

File details

Details for the file pymemtrace-0.1.7-cp39-cp39-macosx_10_9_universal2.whl.

File metadata

File hashes

Hashes for pymemtrace-0.1.7-cp39-cp39-macosx_10_9_universal2.whl
Algorithm Hash digest
SHA256 c6e8ad99fd45c8386f950a40368d51b02b9a5d5a95e14a944d6364c08dea6384
MD5 3921faf38cf5f0cfb8c6a4436e64822e
BLAKE2b-256 8ded07bafc270774990a28b39a6dc633701e0bc81233f0f507e1c8798825e58e

See more details on using hashes here.

File details

Details for the file pymemtrace-0.1.7-cp38-cp38-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for pymemtrace-0.1.7-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 965d3799c231cf6a3e7289dbf5bcb26918ab9353c7fde2dd578ab42d2cbccb9f
MD5 cb128c44834e303ffa0c7184157bf83c
BLAKE2b-256 d4924607f34f5342148374337afce68e00d3184fff1607b3dd17548a296b0aa6

See more details on using hashes here.

File details

Details for the file pymemtrace-0.1.7-cp37-cp37m-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for pymemtrace-0.1.7-cp37-cp37m-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 b18f4edad5e00bdab734411bf000205f388d3710cd5e92c434500a21b8449e03
MD5 a7d1b92287cda70d00f38cba7eaf6389
BLAKE2b-256 4bf9ca46382985b08486bf1343652d5ff6259987790d79b38b992e4ddae68067

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page