Skip to main content

Python memory tracing.

Project description

pymemtrace provides tools for tracking and understanding Python memory usage at different levels, at different granularities and with different runtime costs.

Full documentation: https://pymemtrace.readthedocs.io

pymemtrace Tools

The tools provided by pymemtrace:

  • process is a very lightweight way of logging the total memory usage at regular time intervals. It can plot memory over time with plotting programs such as gnuplot. See some process examples

  • cPyMemTrace is a memory tracer written in C that can report total memory usage for every function call/return for both C and Python sections. See some cPyMemTrace examples and a technical note on cPyMemTrace.

  • DTrace: Here are a number of D scripts that can trace the low level malloc() and free() system calls and report how much memory was allocated and by whom. See some DTrace examples and a technical note on DTrace.

  • trace_malloc is a convenience wrapper around the Python standard library tracemalloc module. This can report Python memory usage by module and line compensating for the cost of tracemalloc. This can take memory snapshots before and after code blocks and show the change on memory caused by that code. See some trace_malloc examples

  • debug_malloc_stats is a wrapper around the sys._debugmallocstats function that can take snapshots of memory before and after code execution and report the significant differences of the Python small object allocator. See some debug_malloc_stats examples

Tool Characteristics

Each tool can be characterised by:

  • Memory Granularity: In how much detail is a memory change is observed. An example of coarse memory granularity is measuring the Resident Set Size which is normally in chunks of 4096 bytes. An example of fine memory granularity is recording every malloc() and free().

  • Execution Granularity: In how much code detail is the memory change observed. An example of coarse execution granularity is measuring the memory usage every second. An example of fine execution granularity is recording the memory usage every Python line.

  • Memory Cost: How much extra memory the tool needs.

  • Execution Cost: How much the execution time is increased.

Clearly there are trade-offs between these depending on the problem you are trying to solve.

Tool Characteristics

Tool

Memory Granularity

Execution Granularity

Memory Cost

Execution Cost

process

RSS (total Python and C memory).

Regular time intervals.

Near zero.

Near zero.

cPyMemTrace

RSS (total Python and C memory).

Per Python line, Python function and C function call.

Near zero.

x10 to x20.

DTrace

Every malloc() and free().

Per function call and return.

Minimal.

x90 to x100.

trace_malloc

Every Python object.

Per Python line, per function call.

Significant but compensated.

x900 for small objects, x6 for large objects.

debug_malloc_stats

Python memory pool.

Snapshots the CPython memory pool either side of a block of code.

Minimal.

x2000+ for small objects, x12 for large objects.

Package Metadata

https://img.shields.io/pypi/v/pymemtrace.svg https://img.shields.io/travis/paulross/pymemtrace.svg Documentation Status Updates

Licence

Python memory tracing.

Credits

Phil Smith (AHL) with whom a casual lunch time chat lead to the creation of an earlier, but quite different implementation, of cPyMemTrace in pure Python.

This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.

History

0.2.0 (2024-11-17)

  • cPyMemTrace:
    • Add P/T, stack depth and python version to log file name, example: “20241107_195847_62264_P_0_PY3.13.0b3.log”

    • Add stacking of trace/profile functions with linked list of tTraceFileWrapperLinkedList.

    • Add an option to log to a specific file.

    • Add an API write_to_log() to inject text into the log file.

    • Add an optional message to the log file in cPyMemTrace.

    • Add Python API to get log file being written to by cPyMemTrace.

    • Bug fixes in cPyMemTrace.c

    • Safety fix for file path name lengths.

    • Fix for log files where ‘#’ was being concatenated.

0.1.7 (2024-09-12)

  • Minor fix for a single test.

0.1.6 (2024-09-11)

  • Add support for Python versions 3.12, 3.13. Now supports Python versions 3.7, 3.8, 3.9, 3.10, 3.11, 3.12, 3.13.

0.1.5 (2023-06-21)

  • Add support for Python versions 3.10, 3.11. Now supports Python versions 3.7, 3.8, 3.9, 3.10, 3.11.

0.1.4 (2022-03-19)

  • Fix Linux build.

0.1.3 (2022-03-17)

  • Fix some tests.

0.1.2 (2022-03-17)

  • Fix source distribution that had missing headers.

0.1.1 (2020-11-17)

  • Add cPyMemTrace the C level profiler.

  • Add DTrace scripts for low level tracing.

  • Add debug_malloc_stats the wrapper around sys._debugmallocstats.

  • Add process from the TotalDepth project.

  • Add redirect_stdout for debug_malloc_stats.

  • Add trace_malloc, a wrapper around the tracemalloc module.

  • Includes extensive documentation and performance measurement.

  • First release on PyPI.

0.1.0 (2017-12-04)

  • Initial idea and implementation, never released.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pymemtrace-0.2.0.tar.gz (377.1 kB view details)

Uploaded Source

Built Distributions

pymemtrace-0.2.0-cp313-cp313-macosx_10_13_universal2.whl (91.4 kB view details)

Uploaded CPython 3.13 macOS 10.13+ universal2 (ARM64, x86-64)

pymemtrace-0.2.0-cp312-cp312-macosx_10_9_universal2.whl (91.4 kB view details)

Uploaded CPython 3.12 macOS 10.9+ universal2 (ARM64, x86-64)

pymemtrace-0.2.0-cp311-cp311-macosx_10_9_universal2.whl (90.3 kB view details)

Uploaded CPython 3.11 macOS 10.9+ universal2 (ARM64, x86-64)

pymemtrace-0.2.0-cp310-cp310-macosx_10_9_universal2.whl (90.0 kB view details)

Uploaded CPython 3.10 macOS 10.9+ universal2 (ARM64, x86-64)

pymemtrace-0.2.0-cp39-cp39-macosx_10_9_universal2.whl (89.9 kB view details)

Uploaded CPython 3.9 macOS 10.9+ universal2 (ARM64, x86-64)

pymemtrace-0.2.0-cp38-cp38-macosx_10_9_x86_64.whl (72.8 kB view details)

Uploaded CPython 3.8 macOS 10.9+ x86-64

pymemtrace-0.2.0-cp37-cp37m-macosx_10_9_x86_64.whl (72.3 kB view details)

Uploaded CPython 3.7m macOS 10.9+ x86-64

File details

Details for the file pymemtrace-0.2.0.tar.gz.

File metadata

  • Download URL: pymemtrace-0.2.0.tar.gz
  • Upload date:
  • Size: 377.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.11.6

File hashes

Hashes for pymemtrace-0.2.0.tar.gz
Algorithm Hash digest
SHA256 d0a8c3b1131f5e6f19ca6215c1f924e95504a488991cf849a8466c6feb351f5a
MD5 f931bfaf53dc7b86fa28feb41079d5cc
BLAKE2b-256 95d3e049b9e1f8c1b61e221b8fc45f4743e366632720ca59578ff74f4a35b288

See more details on using hashes here.

File details

Details for the file pymemtrace-0.2.0-cp313-cp313-macosx_10_13_universal2.whl.

File metadata

File hashes

Hashes for pymemtrace-0.2.0-cp313-cp313-macosx_10_13_universal2.whl
Algorithm Hash digest
SHA256 91bc7fb7805d403506afd2aad370c0b731ab2140b010657800cd870cd7afd35d
MD5 d570867789d70d2b6ef7d569613cc0e3
BLAKE2b-256 31854dd464af110a4ea5850ad9284de78213f18da363d4f3444fabf2ccf8a5e4

See more details on using hashes here.

File details

Details for the file pymemtrace-0.2.0-cp312-cp312-macosx_10_9_universal2.whl.

File metadata

File hashes

Hashes for pymemtrace-0.2.0-cp312-cp312-macosx_10_9_universal2.whl
Algorithm Hash digest
SHA256 499da61026e588ef44f25a9a977796be44dbe8402155efacfbdb52773c896d70
MD5 e012deb8d087dd47a1384d37e36c5d64
BLAKE2b-256 f119204e6da714d1ae30135393ccf8700b1bfbff7cf640bd28ed4c8f87ddb557

See more details on using hashes here.

File details

Details for the file pymemtrace-0.2.0-cp311-cp311-macosx_10_9_universal2.whl.

File metadata

File hashes

Hashes for pymemtrace-0.2.0-cp311-cp311-macosx_10_9_universal2.whl
Algorithm Hash digest
SHA256 27d2df543860eb30dde9ba30d2d540101a3fdca20ad3cc88fc2014442dd67fe4
MD5 1817bb579d1dac1a53baf721af7890c9
BLAKE2b-256 60b6406ee890d485ee6ed7ffa40c54c5cc182374eb8a5275d6e82c63b3401ff4

See more details on using hashes here.

File details

Details for the file pymemtrace-0.2.0-cp310-cp310-macosx_10_9_universal2.whl.

File metadata

File hashes

Hashes for pymemtrace-0.2.0-cp310-cp310-macosx_10_9_universal2.whl
Algorithm Hash digest
SHA256 e79ca3be49aaec471a3a0420202c9b75c4d9f07c440ab1ee90d32ad36f0f9382
MD5 667f791a6a3b7c060f6baba805d931e7
BLAKE2b-256 accb20afafc78f7ee48a92986c349794b8f7e60007932be18f66fdf72b385419

See more details on using hashes here.

File details

Details for the file pymemtrace-0.2.0-cp39-cp39-macosx_10_9_universal2.whl.

File metadata

File hashes

Hashes for pymemtrace-0.2.0-cp39-cp39-macosx_10_9_universal2.whl
Algorithm Hash digest
SHA256 870d4bec1d15d0235a1e4f1b4d7130565af13d344a6f5c502c76cbcaca780c9b
MD5 616b05f64bb9f1fc2320e75b0771126d
BLAKE2b-256 d5a03b2962369ea8b96fc6aea76390e1abc3006b6f2a88697956656a6e71b980

See more details on using hashes here.

File details

Details for the file pymemtrace-0.2.0-cp38-cp38-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for pymemtrace-0.2.0-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 e8873a415d02c2a2a630f56c8f1fe0ce1cdad47b8f819645d98e5662cfb28bb5
MD5 71d66ecc7fed59a238b4c57cc07947f6
BLAKE2b-256 b36e8c5a9744d597d014d42cefdc901921d3d403c9772d6fdf216bf08baa53d2

See more details on using hashes here.

File details

Details for the file pymemtrace-0.2.0-cp37-cp37m-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for pymemtrace-0.2.0-cp37-cp37m-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 d5b3efd27e5bccb32ea4ea872affdc320e30f464314d8ae1d792332c003667e6
MD5 b81162476f3e83ef3cf0726b5e70f4d2
BLAKE2b-256 f1dc16bec9b57a317153dff557306924295bd19c8b7d356dfdb6f4e5988023e9

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page