Skip to main content

A memory profiler for data batch processing applications.

Project description

The Fil memory profiler for Python

Fil a memory profiler designed for data processing applications. At the moment it only runs on Linux and macOS.

Your code reads some data, processes it, and—uses too much memory. What you need to reduce is peak memory usage.

And that's exactly what Fil will help you find: exactly which code was responsible for allocating memory at peak memory usage.

For more information see https://pythonspeed.com/products/filmemoryprofiler/

What Fil tracks

Fil will track memory allocated by:

  • Normal Python code.
  • C code using malloc()/calloc()/realloc().
  • C++ code using new (including via aligned_alloc).
  • Anonymous mmap()s.
  • Fortran 90 explicitly allocated memory (tested with gcc's gfortran).

Still not supported, but planned:

  • mremap() (resizing of mmap()).
  • File-backed mmap(). The usage here is inconsistent since the OS can swap it in or out, so probably supporting this will involve a different kind of resource usage.
  • Other forms of shared memory, need to investigate if any of them allow sufficient allocation.
  • Anonymous mmap()s created via /dev/zero (not common, since it's not cross-platform, e.g. macOS doesn't support this).
  • posix_memalign(), memalign, valloc(), pvalloc(), reallocarray().
  • memfd_create().

Installation

Prerequisites

To install the latest version of Fil you'll need Pip 19 or newer. You can check like this:

$ pip --version
pip 20.0.2

If you're using something older than that, do:

$ pip install --upgrade pip

If that doesn't work, try running that a virtualenv or Conda env.

Installing Fil

Assuming you're on macOS or Linux, and are using Python 3.6 or later:

$ pip install filprofiler

Measuring peak (high-water mark) memory usage

Instead of doing:

$ python yourscript.py --input-file=yourfile

Just do:

$ fil-profile run yourscript.py --input-file=yourfile

And it will generate a report.

Debugging out-of-memory crashes

First, run free to figure out how much memory is available—in this case about 6.3GB—and then set a corresponding limit on virtual memory with ulimit:

$ free -h
       total   used   free  shared  buff/cache  available
Mem:   7.7Gi  1.1Gi  6.3Gi    50Mi       334Mi      6.3Gi
Swap:  3.9Gi  3.0Gi  871Mi
$ ulimit -Sv 6300000

Then, run your program under Fil, and it will generate a SVG at the point in time when memory runs out:

$ fil-profile run oom.py 
...
=fil-profile= Wrote memory usage flamegraph to fil-result/2020-06-15T12:37:13.033/out-of-memory.svg

You've found where memory usage is coming from—now what?

If you're using data processing or scientific computing libraries, I have written a relevant guide to reducing memory usage.

License

Copyright 2020 Hyphenated Enterprises LLC

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

 http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

filprofiler-0.8.0-cp38-cp38-manylinux2010_x86_64.whl (1.7 MB view details)

Uploaded CPython 3.8manylinux: glibc 2.12+ x86-64

filprofiler-0.8.0-cp38-cp38-macosx_10_14_x86_64.whl (257.5 kB view details)

Uploaded CPython 3.8macOS 10.14+ x86-64

filprofiler-0.8.0-cp37-cp37m-manylinux2010_x86_64.whl (1.7 MB view details)

Uploaded CPython 3.7mmanylinux: glibc 2.12+ x86-64

filprofiler-0.8.0-cp37-cp37m-macosx_10_14_x86_64.whl (257.5 kB view details)

Uploaded CPython 3.7mmacOS 10.14+ x86-64

filprofiler-0.8.0-cp36-cp36m-manylinux2010_x86_64.whl (1.7 MB view details)

Uploaded CPython 3.6mmanylinux: glibc 2.12+ x86-64

filprofiler-0.8.0-cp36-cp36m-macosx_10_14_x86_64.whl (257.5 kB view details)

Uploaded CPython 3.6mmacOS 10.14+ x86-64

File details

Details for the file filprofiler-0.8.0-cp38-cp38-manylinux2010_x86_64.whl.

File metadata

  • Download URL: filprofiler-0.8.0-cp38-cp38-manylinux2010_x86_64.whl
  • Upload date:
  • Size: 1.7 MB
  • Tags: CPython 3.8, manylinux: glibc 2.12+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.48.0 CPython/3.8.3

File hashes

Hashes for filprofiler-0.8.0-cp38-cp38-manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 dea5cdf1100014447d824bd469f8cf199fa9482606e26cc1d91c1bb015dde42f
MD5 98ad2b006db29fbc922329d7d4689355
BLAKE2b-256 1b8b93cf518d198df4d6863a2e604ac86605d0fb495664d3ba7ff263f3c4ef8b

See more details on using hashes here.

File details

Details for the file filprofiler-0.8.0-cp38-cp38-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: filprofiler-0.8.0-cp38-cp38-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 257.5 kB
  • Tags: CPython 3.8, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.48.0 CPython/3.8.3

File hashes

Hashes for filprofiler-0.8.0-cp38-cp38-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 da1e51f5cf6345b207da59fafaaf14128055abf4ddf418a9b5dee0a4c674d461
MD5 5a7324f49a5d798193affd7005ef7694
BLAKE2b-256 2a0a13eee0dd204fde625be92e8a77f7fc16f9e1307e0ff8e8b39d89a0fd71b2

See more details on using hashes here.

File details

Details for the file filprofiler-0.8.0-cp37-cp37m-manylinux2010_x86_64.whl.

File metadata

  • Download URL: filprofiler-0.8.0-cp37-cp37m-manylinux2010_x86_64.whl
  • Upload date:
  • Size: 1.7 MB
  • Tags: CPython 3.7m, manylinux: glibc 2.12+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.48.0 CPython/3.8.3

File hashes

Hashes for filprofiler-0.8.0-cp37-cp37m-manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 16b868af1391950537176956007bb974d663de7785cad2f153bcd505dc8e8ecc
MD5 a394809dcae2ec1f3dc219f90c983ead
BLAKE2b-256 41f39db509c81cb75ed8fe0b197e27ff6c9f991bd77268402f724524a79a26a2

See more details on using hashes here.

File details

Details for the file filprofiler-0.8.0-cp37-cp37m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: filprofiler-0.8.0-cp37-cp37m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 257.5 kB
  • Tags: CPython 3.7m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/47.1.0 requests-toolbelt/0.9.1 tqdm/4.48.0 CPython/3.7.8

File hashes

Hashes for filprofiler-0.8.0-cp37-cp37m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 c4faa26fe9d09c6c949777d80e313408cd2d56b87d32945383faf677eea7b6dc
MD5 7086ff42dd7b8871411920554e1bc0c4
BLAKE2b-256 25eadbf1eb4e540f398569190c8b3159e474e6c2c61ea96471ca841d51a2807c

See more details on using hashes here.

File details

Details for the file filprofiler-0.8.0-cp36-cp36m-manylinux2010_x86_64.whl.

File metadata

  • Download URL: filprofiler-0.8.0-cp36-cp36m-manylinux2010_x86_64.whl
  • Upload date:
  • Size: 1.7 MB
  • Tags: CPython 3.6m, manylinux: glibc 2.12+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.48.0 CPython/3.8.3

File hashes

Hashes for filprofiler-0.8.0-cp36-cp36m-manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 c1831c62737c9455e1fba57e4619c56e66e4d5876a63ee520df3947879d33bb1
MD5 539331edea8784560419ae8c714af3ee
BLAKE2b-256 f57ba84ca5d5c82fced56e9120915109322f83a993d03bf4441c35cda21b576f

See more details on using hashes here.

File details

Details for the file filprofiler-0.8.0-cp36-cp36m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: filprofiler-0.8.0-cp36-cp36m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 257.5 kB
  • Tags: CPython 3.6m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/40.6.2 requests-toolbelt/0.9.1 tqdm/4.48.0 CPython/3.6.11

File hashes

Hashes for filprofiler-0.8.0-cp36-cp36m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 aae9eca15eb19a41b1cd3e8cca1b3dd18a1ebeec98d72e3207aa6beebe7a7477
MD5 a4aebe4021d97a0609e9e44d0dc9c77d
BLAKE2b-256 acdbb970f82ae61ccfcb00f25de3ec6e06fb86ac24d1c2013dbad8d2e8514a62

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page