Skip to main content

A memory profiler for data batch processing applications.

Project description

The Fil memory profiler for Python

Fil a memory profiler designed for data processing applications. At the moment it only runs on Linux and macOS.

Your code reads some data, processes it, and—uses too much memory. What you need to reduce is peak memory usage.

And that's exactly what Fil will help you find: exactly which code was responsible for allocating memory at peak memory usage.

For more information see https://pythonspeed.com/products/filmemoryprofiler/

What Fil tracks

Fil will track memory allocated by:

  • Normal Python code.
  • C code using malloc()/calloc()/realloc().
  • C++ code using new (including via aligned_alloc).
  • Anonymous mmap()s.
  • Fortran 90 explicitly allocated memory (tested with gcc's gfortran).

Still not supported, but planned:

  • mremap() (resizing of mmap()).
  • File-backed mmap(). The usage here is inconsistent since the OS can swap it in or out, so probably supporting this will involve a different kind of resource usage.
  • Other forms of shared memory.
  • Anonymous mmap()s created via /dev/zero (not common, since it's not cross-platform, e.g. macOS doesn't support this).
  • posix_memalign(), memalign, valloc(), pvalloc().
  • memfd_create().

Installation

To install:

$ pip install filprofiler

Measuring peak (high-water mark) memory usage

Instead of doing:

$ python yourscript.py --input-file=yourfile

Just do:

$ fil-profile run yourscript.py --input-file=yourfile

And it will generate a report.

Debugging out-of-memory crashes

First, run free to figure out how much memory is available—in this case about 6.3GB—and then set a corresponding limit on virtual memory with ulimit:

$ free -h
       total   used   free  shared  buff/cache  available
Mem:   7.7Gi  1.1Gi  6.3Gi    50Mi       334Mi      6.3Gi
Swap:  3.9Gi  3.0Gi  871Mi
$ ulimit -Sv 6300000

Then, run your program under Fil, and it will generate a SVG at the point in time when memory runs out:

$ fil-profile run oom.py 
...
=fil-profile= Wrote memory usage flamegraph to fil-result/2020-06-15T12:37:13.033/out-of-memory.svg

You've found where memory usage is coming from—now what?

If you're using data processing or scientific computing libraries, I have written a relevant guide to reducing memory usage.

License

Copyright 2020 Hyphenated Enterprises LLC

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

 http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

filprofiler-0.7.2-cp38-cp38-manylinux1_x86_64.whl (565.2 kB view details)

Uploaded CPython 3.8

filprofiler-0.7.2-cp38-cp38-macosx_10_14_x86_64.whl (272.3 kB view details)

Uploaded CPython 3.8macOS 10.14+ x86-64

filprofiler-0.7.2-cp37-cp37m-manylinux1_x86_64.whl (565.9 kB view details)

Uploaded CPython 3.7m

filprofiler-0.7.2-cp37-cp37m-macosx_10_14_x86_64.whl (272.3 kB view details)

Uploaded CPython 3.7mmacOS 10.14+ x86-64

filprofiler-0.7.2-cp36-cp36m-manylinux1_x86_64.whl (565.1 kB view details)

Uploaded CPython 3.6m

filprofiler-0.7.2-cp36-cp36m-macosx_10_14_x86_64.whl (272.3 kB view details)

Uploaded CPython 3.6mmacOS 10.14+ x86-64

File details

Details for the file filprofiler-0.7.2-cp38-cp38-manylinux1_x86_64.whl.

File metadata

  • Download URL: filprofiler-0.7.2-cp38-cp38-manylinux1_x86_64.whl
  • Upload date:
  • Size: 565.2 kB
  • Tags: CPython 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.47.0 CPython/3.8.3

File hashes

Hashes for filprofiler-0.7.2-cp38-cp38-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 3a340096193892ca4906249d6c52eeeee62732b37953e7dadaa54fd2dd0313d8
MD5 3ffdbf6366230f669bd7a1e5571e83f3
BLAKE2b-256 5b0fcfdb86614ddd026f9d0dc75965372c70f58b98c95ca19f2b54a14ddc4cc6

See more details on using hashes here.

File details

Details for the file filprofiler-0.7.2-cp38-cp38-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: filprofiler-0.7.2-cp38-cp38-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 272.3 kB
  • Tags: CPython 3.8, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.47.0 CPython/3.8.3

File hashes

Hashes for filprofiler-0.7.2-cp38-cp38-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 17da30e1072a01a7587fa15ab12299c873aa93e261c8a7f8f3499d84803b982d
MD5 c0c7250c727ae6b7e0f7c26f6b9f73d0
BLAKE2b-256 749fd40fad8be9e302db90c91349165f47702a5b359be1993d28839ea2f27f7e

See more details on using hashes here.

File details

Details for the file filprofiler-0.7.2-cp37-cp37m-manylinux1_x86_64.whl.

File metadata

  • Download URL: filprofiler-0.7.2-cp37-cp37m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 565.9 kB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.47.0 CPython/3.8.3

File hashes

Hashes for filprofiler-0.7.2-cp37-cp37m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 d789d6b8092a9e11c93248ab9ab03fcd24b7d06742a45dab112ae08ceedb3cdb
MD5 4b48e068a380f29d6be24943076bfef5
BLAKE2b-256 40fcca3f02c765aa91ce9afc1837df06802868f6bced005513bdd66886e6c09a

See more details on using hashes here.

File details

Details for the file filprofiler-0.7.2-cp37-cp37m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: filprofiler-0.7.2-cp37-cp37m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 272.3 kB
  • Tags: CPython 3.7m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/47.1.0 requests-toolbelt/0.9.1 tqdm/4.47.0 CPython/3.7.8

File hashes

Hashes for filprofiler-0.7.2-cp37-cp37m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 26ffce207cb0171ab148e50eb15192d509739bce5528db1cb3586196f557e4a9
MD5 2044393469649e91a4027c45c6557471
BLAKE2b-256 56ce1554851e4dbc03e9d3760141f7bee863b1b9e6ae73b3ee9295697d764dda

See more details on using hashes here.

File details

Details for the file filprofiler-0.7.2-cp36-cp36m-manylinux1_x86_64.whl.

File metadata

  • Download URL: filprofiler-0.7.2-cp36-cp36m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 565.1 kB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.47.0 CPython/3.8.3

File hashes

Hashes for filprofiler-0.7.2-cp36-cp36m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 bc2c357a04f11732db01fe0235c95b4af9cc0827ca22ba6b0ad538123bb9d3ea
MD5 bc08d8bc7820c5f09dcddec14f2a6b0b
BLAKE2b-256 e3a2843e7b5f1aba27effb0146c7e564e2592bfc9344a8c8ef0d55245bd47508

See more details on using hashes here.

File details

Details for the file filprofiler-0.7.2-cp36-cp36m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: filprofiler-0.7.2-cp36-cp36m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 272.3 kB
  • Tags: CPython 3.6m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/40.6.2 requests-toolbelt/0.9.1 tqdm/4.47.0 CPython/3.6.11

File hashes

Hashes for filprofiler-0.7.2-cp36-cp36m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 8b2e3146f9c484d347c7a292a4283474bcd2502443149783e3c6caed54ecb837
MD5 316a28df9c84741bebdabe35e41d1964
BLAKE2b-256 f47c7264e41277c1110fa8ca4b3ccc1af809d504ab6e0e0f49098276587adb21

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page