This is a pre-production deployment of Warehouse, however changes made here WILL affect the production instance of PyPI.
Latest Version Dependencies status unknown Test status unknown Test coverage unknown
Project Description

memory-tools

A set of simple yet effective tools to troubleshoot memory leaks.

When debugging memory issues in Python 2.6, the author has tried memory_profiler and heapy, unfortunately neither worked. And so memory-tools was born with the goal of being simple - it should always work, yet effective - it is good at helping you find memory leaks.

Quick Start Tutorial

Installation

$ pip install memory-tools

Show Memory Usage / Delta

Use the show-mem command to show system or process memory usage. When paired with watch, this becomes even more useful.

Show system memory:

$ show-mem

Commit Mem (MB):       27,852.80 total   17,278.42 used
Physical Mem (MB):     16,384.00 total   13,128.05 used

Re-run to show delta from last run:

$ show-mem

Commit Mem (MB):       27,852.80 total   17,888.59 used (delta: 310.15)
Physical Mem (MB):     16,384.00 total   13,126.40 used (delta: -1.65)

Show memory for process:

$ show-mem -p python

1 process matching "python":
  PID 26143 (MB):           4.79 rss          1.23 private

$ show-mem -p 26143

  PID 26143 (MB):           4.80 rss          1.24 private

Watch system/process memory using watch:

$ watch show-mem -s -p python

Commit Mem (MB):       27,852.80 total   17,888.59 used (delta: 310.15)
Physical Mem (MB):     16,384.00 total   13,126.40 used (delta: -1.65)

2 processes matching "python" (showing 1st & last):
  PID 26143 (MB):          40.79 rss         30.23 private
  PID 24118 (MB):           4.79 rss          1.23 private

Summarize / Save GC Objects

After running your program, view summary of gc.get_objects():

from memorytools import summarize_objects

summarize_objects()

And here is a sample output:

Objects count 3,790
Objects size 833,344

      Size Count Type
   476,864   296 <type 'dict'>
    76,320   954 <type 'wrapper_descriptor'>
    64,920   541 <type 'function'>
    ...

Count       Size Type
  954     76,320 <type 'wrapper_descriptor'>
  541     64,920 <type 'function'>
  515     37,080 <type 'builtin_function_or_method'>
  ...

Save all objects (along with the above summary) to a file:

from memorytools import save_objects

save_objects()

# Output: Wrote 3887 objects to /var/tmp/objects-45271 (882040 bytes)

Looping / Stress Testing

Use the loop command to run a command, module:method, or code in a forever loop to perform stress testing, which is useful in finding memory leaks. The command/code should, of course, act against a long running server for this to be useful.

Run a script in a loop:

$ loop show-mem 1

Physical Mem (MB):     16,384.00 total    9,415.50 used (delta: -190.67)
Physical Mem (MB):     16,384.00 total    9,415.27 used (delta: -0.23)
Physical Mem (MB):     16,384.00 total    9,415.85 used (delta: 0.58)
^C                   [ User CTRL-C here as it loops forever by default ]
Looped 3 times in 2.80 secs

Run a module:method in a loop - count of 10:

$ loop memorytools:summarize_objects 10 -c 10

# Results from summarize_objects() every 10 seconds

Looped 10 times in 100 secs

Run adhoc code in a loop - count of 2 and concurrency of 3:

$ loop 'print("Hello World!")' 0.1 -c 2 -cc 3
Hello World!
... 5 more times

Looped 2 times in 0.21 secs with concurrency of 3 (6 runs, 0.10 secs per loop, 0.03 secs per run)

Log Stack / Start Debugger on Signal

If you need to get a stacktrace of a running process, or start the debugger in specific situations to look at memory footprint, then a signal handler could help:

from memorytools import add_debug_handler

add_debug_handler(start_debugger_password='test')  # remove start_debugger_password to skip rpdb2 debugger

The above will add a handler to SIGUSR2 that will log a stacktrace on trigger and also start the rpdb2 debugger.

Release History

Release History

1.0.3

This version

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

1.0.2

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

1.0.1

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

1.0.0

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

Download Files

Download Files

TODO: Brief introduction on what you do with files - including link to relevant help section.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
memory-tools-1.0.3.tar.gz (16.3 kB) Copy SHA256 Checksum SHA256 Source Jul 1, 2016

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS HPE HPE Development Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting