Skip to main content

A memory profiler for data batch processing applications.

Project description

The Fil memory profiler for Python

Fil a memory profiler designed for data processing applications. At the moment it only runs on Linux and macOS.

Your code reads some data, processes it, and—uses too much memory. What you need to reduce is peak memory usage.

And that's exactly what Fil will help you find: exactly which code was responsible for allocating memory at peak memory usage.

For more information see https://pythonspeed.com/products/filmemoryprofiler/

What Fil tracks

Fil will track memory allocated by:

  • Normal Python code.
  • C code using malloc()/calloc()/realloc().
  • C++ code using new (including via aligned_alloc).
  • Anonymous mmap()s.
  • Fortran 90 explicitly allocated memory (tested with gcc's gfortran).

Still not supported, but planned:

  • mremap() (resizing of mmap()).
  • File-backed mmap(). The usage here is inconsistent since the OS can swap it in or out, so probably supporting this will involve a different kind of resource usage.
  • Other forms of shared memory.
  • Anonymous mmap()s created via /dev/zero (not common, since it's not cross-platform, e.g. macOS doesn't support this).
  • posix_memalign(), memalign, valloc(), pvalloc().
  • memfd_create().

Installation

To install:

$ pip install filprofiler

Measuring peak (high-water mark) memory usage

Instead of doing:

$ python yourscript.py --input-file=yourfile

Just do:

$ fil-profile run yourscript.py --input-file=yourfile

And it will generate a report.

Debugging out-of-memory crashes

First, run free to figure out how much memory is available—in this case about 6.3GB—and then set a corresponding limit on virtual memory with ulimit:

$ free -h
       total   used   free  shared  buff/cache  available
Mem:   7.7Gi  1.1Gi  6.3Gi    50Mi       334Mi      6.3Gi
Swap:  3.9Gi  3.0Gi  871Mi
$ ulimit -Sv 6300000

Then, run your program under Fil, and it will generate a SVG at the point in time when memory runs out:

$ fil-profile run oom.py 
...
=fil-profile= Wrote memory usage flamegraph to fil-result/2020-06-15T12:37:13.033/out-of-memory.svg

You've found where memory usage is coming from—now what?

If you're using data processing or scientific computing libraries, I have written a relevant guide to reducing memory usage.

License

Copyright 2020 Hyphenated Enterprises LLC

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

 http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

filprofiler-0.7.1-cp38-cp38-manylinux1_x86_64.whl (558.9 kB view details)

Uploaded CPython 3.8

filprofiler-0.7.1-cp38-cp38-macosx_10_14_x86_64.whl (269.3 kB view details)

Uploaded CPython 3.8macOS 10.14+ x86-64

filprofiler-0.7.1-cp37-cp37m-manylinux1_x86_64.whl (559.9 kB view details)

Uploaded CPython 3.7m

filprofiler-0.7.1-cp37-cp37m-macosx_10_14_x86_64.whl (269.3 kB view details)

Uploaded CPython 3.7mmacOS 10.14+ x86-64

filprofiler-0.7.1-cp36-cp36m-manylinux1_x86_64.whl (558.8 kB view details)

Uploaded CPython 3.6m

filprofiler-0.7.1-cp36-cp36m-macosx_10_14_x86_64.whl (269.3 kB view details)

Uploaded CPython 3.6mmacOS 10.14+ x86-64

File details

Details for the file filprofiler-0.7.1-cp38-cp38-manylinux1_x86_64.whl.

File metadata

  • Download URL: filprofiler-0.7.1-cp38-cp38-manylinux1_x86_64.whl
  • Upload date:
  • Size: 558.9 kB
  • Tags: CPython 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.47.0 CPython/3.8.3

File hashes

Hashes for filprofiler-0.7.1-cp38-cp38-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 36fd5ab695550f59ca89b8a624c7f0a77ba959a29ec9355267edebf6b5c615b5
MD5 7c25e773f6a653a5cfe31ca5e66a05c3
BLAKE2b-256 c2fc532098659a06dbcdc167a103ccfd821e456bdb922c1b7f4aa69fa4c95b22

See more details on using hashes here.

File details

Details for the file filprofiler-0.7.1-cp38-cp38-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: filprofiler-0.7.1-cp38-cp38-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 269.3 kB
  • Tags: CPython 3.8, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.47.0 CPython/3.8.3

File hashes

Hashes for filprofiler-0.7.1-cp38-cp38-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 55f2fb4c2db44c0d129a1bbe337d7ec16b7ca6f53cd70f4e1d8d39ea136143ef
MD5 8db8dffc12c05a287cf07143117aba55
BLAKE2b-256 c4d798eeb9eecfa92270b23d038be2f35790e22b47d251cd8b4c8d533f1cd049

See more details on using hashes here.

File details

Details for the file filprofiler-0.7.1-cp37-cp37m-manylinux1_x86_64.whl.

File metadata

  • Download URL: filprofiler-0.7.1-cp37-cp37m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 559.9 kB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.47.0 CPython/3.8.3

File hashes

Hashes for filprofiler-0.7.1-cp37-cp37m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 256fcc578d188711c670e0d6450ecfff77db457ed6e13847a6bb7c6fb87b5afc
MD5 e92c4ccdd160bf4e71b374ca07b2b367
BLAKE2b-256 830b875f428db5942704bcd23bee89b269b7469ca5a774814a10a800089bfee7

See more details on using hashes here.

File details

Details for the file filprofiler-0.7.1-cp37-cp37m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: filprofiler-0.7.1-cp37-cp37m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 269.3 kB
  • Tags: CPython 3.7m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.47.0 CPython/3.7.7

File hashes

Hashes for filprofiler-0.7.1-cp37-cp37m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 8e9e532975fdca74bed4f66828cfabcb04cb32bda5610c723b56cc36f2a7e517
MD5 c242617c3ecd69b5cd3eecb4a8ea1e4e
BLAKE2b-256 7f8450391b4616e72a54cb333907977960c5add7ae3ed33a59dddaf44c36d552

See more details on using hashes here.

File details

Details for the file filprofiler-0.7.1-cp36-cp36m-manylinux1_x86_64.whl.

File metadata

  • Download URL: filprofiler-0.7.1-cp36-cp36m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 558.8 kB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.47.0 CPython/3.8.3

File hashes

Hashes for filprofiler-0.7.1-cp36-cp36m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 dc3b8d70ae173cf1837c9391bd7267bffd62ea4d4c40fa5d7b87cf5e53695a29
MD5 ce7b6c5c40158c8586f91558e9ddd42a
BLAKE2b-256 6940b4073833f4f0bdfb7cbb10b813a241db71403bd3e52f0963805e9366e56c

See more details on using hashes here.

File details

Details for the file filprofiler-0.7.1-cp36-cp36m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: filprofiler-0.7.1-cp36-cp36m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 269.3 kB
  • Tags: CPython 3.6m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/40.6.2 requests-toolbelt/0.9.1 tqdm/4.47.0 CPython/3.6.10

File hashes

Hashes for filprofiler-0.7.1-cp36-cp36m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 7088ccd5a040c44ff80ed8630265316cbc1760e3ffd7a8a6a8c4029636ed9cc4
MD5 a60fb37a3017b02be28448ef25537863
BLAKE2b-256 130ba1d7cbb86fafec3bbe45f3056735f46b075a6b14b0a3084b380c5b72a52d

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page