Skip to main content

Tools to help with memory leaks

Project description


|Build Status|

Yeah Memory Issues!!

Memory Issues happens to the best of us. ``memory_utils`` will give you some simple tools to help you quickly issolate
The cuplrit. And ideally warn you before you run into issues.

From my experience there is no silver-bullet in dealing with memory issues. You just have roll up your sleeve and get
dirty with print statements. In our teams recent fight with a memory issue, we created a utility that we found useful
and we wanted to share.

``memory_utils`` deal primarly with RSS memory (Resident Set Size) The *resident set size* is the portion of a
process's memory that is held in RAM. The rest of the memory exists in swap of the file system. This is in
general the most important memory concept to be aware of when dealing with memory constrained systems.


.. code:: bash

pip install memory_utils


The workhorse of this package is ``print_memory`` It simply prints out 3 columns of data, the current memory, the delta
since the previous statement and an message that you pass it. If there is an additional memory used -- the line will
be printed RED and if there is a decrease, the line will be printed GREEN.

It is a very simple approach -- but it really helped us at glance to find out where the issue was, The output could
look like this::

RSS Delta Message
14,393,344 14,393,344 BEFORE BLOAT
14,397,440 4,096 DURING BLOAT (1)
14,413,824 16,384 DURING BLOAT (102)
14,417,920 4,096 DURING BLOAT (211)
14,438,400 20,480 DURING BLOAT (1002)
14,442,496 4,096 DURING BLOAT (2034)
14,462,976 20,480 DURING BLOAT (2056)

memory_watcher and check_memory
We have worker processes that run in containers. I like to fail hard and early. So we have two helper functions
that help us with that


Will check the current rss memory against the memory_utils set memory limit. And if it crosses that limit it will
raise a ``MemoryToBigException``::

import memory_utils
memory_utils.set_memory_limit(200 * memory_utils.MEGABTYES)

.... else where



Often you will want to do your ``check_memory`` at a _safe_ place. Also memory leak often happen within a loop.
We created ``memory_watcher`` with those concepts in mind::

for account in memory_watcher(Account.objects):

This will call ``check_memory`` before each iteration

By default ``print_memory`` will only print statements that move the memory
and ``memory_watcher`` will not print its memory useage
If you want additional verbosity set this to true::

import memory_utils

By default the memory limit at 200 MB

Use this method to change the default.

This setting is used in ``print_memory`` and ``memory_watcher``

Note: for all methods that deal with this limit -- you can also override it at
the function level as well::

import memory_utils
memory_utils.set_memory_limit(500 * memory_utils.MEGABYTES)


By default we will print to standard out. Feel free to override here like so::

import memory_utils
from StringIO import StringIO

out = StringIO()

Questions / Issues

Feel free to ping me on twitter: `@tushman`_
or add issues or PRs at

.. _@tushman:

.. |Build Status| image::

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

memory_utils-0.1.1.tar.gz (4.3 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page