Skip to main content

A memory profiler for data batch processing applications.

Project description

The Fil memory profiler for Python

Fil a memory profiler designed for data processing applications. At the moment it only runs on Linux and macOS.

Your code reads some data, processes it, and—uses too much memory. What you need to reduce is peak memory usage.

And that's exactly what Fil will help you find: exactly which code was responsible for allocating memory at peak memory usage.

For more information see https://pythonspeed.com/products/filmemoryprofiler/

Installation

To install:

$ pip install filprofiler

Measuring peak (high-water mark) memory usage

Instead of doing:

$ python yourscript.py --input-file=yourfile

Just do:

$ fil-profile run yourscript.py --input-file=yourfile

And it will generate a report.

Debugging out-of-memory crashes

First, run free to figure out how much memory is available—in this case about 6.3GB—and then set a corresponding limit on virtual memory with ulimit:

$ free -h
       total   used   free  shared  buff/cache  available
Mem:   7.7Gi  1.1Gi  6.3Gi    50Mi       334Mi      6.3Gi
Swap:  3.9Gi  3.0Gi  871Mi
$ ulimit -Sv 6300000

Then, run your program under Fil, and it will generate a SVG at the point in time when memory runs out:

$ fil-profile run oom.py 
...
=fil-profile= Wrote memory usage flamegraph to fil-result/2020-06-15T12:37:13.033/out-of-memory.svg

You've found where memory usage is coming from—now what?

If you're using data processing or scientific computing libraries, I have written a relevant guide to reducing memory usage.

License

Copyright 2020 Hyphenated Enterprises LLC

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

 http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

filprofiler-0.6.0-cp38-cp38-manylinux1_x86_64.whl (557.6 kB view details)

Uploaded CPython 3.8

filprofiler-0.6.0-cp38-cp38-macosx_10_14_x86_64.whl (268.6 kB view details)

Uploaded CPython 3.8 macOS 10.14+ x86-64

filprofiler-0.6.0-cp37-cp37m-manylinux1_x86_64.whl (558.6 kB view details)

Uploaded CPython 3.7m

filprofiler-0.6.0-cp37-cp37m-macosx_10_14_x86_64.whl (268.6 kB view details)

Uploaded CPython 3.7m macOS 10.14+ x86-64

filprofiler-0.6.0-cp36-cp36m-manylinux1_x86_64.whl (557.6 kB view details)

Uploaded CPython 3.6m

filprofiler-0.6.0-cp36-cp36m-macosx_10_14_x86_64.whl (268.6 kB view details)

Uploaded CPython 3.6m macOS 10.14+ x86-64

File details

Details for the file filprofiler-0.6.0-cp38-cp38-manylinux1_x86_64.whl.

File metadata

  • Download URL: filprofiler-0.6.0-cp38-cp38-manylinux1_x86_64.whl
  • Upload date:
  • Size: 557.6 kB
  • Tags: CPython 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.47.0 CPython/3.8.3

File hashes

Hashes for filprofiler-0.6.0-cp38-cp38-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 b23496eec621a932d1f92ea0ea7297dd05817581b1a95d05d9a10d64f1294d85
MD5 fb63d676b1f8012b9029c4270c92630a
BLAKE2b-256 a82cff82101c38378d5b2c3406e3e348772b3187fa9def320d1d27a693b23345

See more details on using hashes here.

File details

Details for the file filprofiler-0.6.0-cp38-cp38-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: filprofiler-0.6.0-cp38-cp38-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 268.6 kB
  • Tags: CPython 3.8, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.47.0 CPython/3.8.3

File hashes

Hashes for filprofiler-0.6.0-cp38-cp38-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 1107fc9b586aaceb142f9b06e4664f191dff7ef853d9d06e59c071c86b3b9f3b
MD5 6109fe7b4307fb12ede64b60de215b15
BLAKE2b-256 ee18f558ec2dc11bd4748dc6cb61adf9384729c6f7077c79ac9f53507090e020

See more details on using hashes here.

File details

Details for the file filprofiler-0.6.0-cp37-cp37m-manylinux1_x86_64.whl.

File metadata

  • Download URL: filprofiler-0.6.0-cp37-cp37m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 558.6 kB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.47.0 CPython/3.8.3

File hashes

Hashes for filprofiler-0.6.0-cp37-cp37m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 42f6a80dc62364a1bf239c1a830ad6f512e1308cb550eca56e4422818a488f70
MD5 4285dd7e64b7cd5764b4df25f4ac69fb
BLAKE2b-256 10e91c9e50dbdde93be83c491756d0b529671adb58c277e70a95060809b1eedc

See more details on using hashes here.

File details

Details for the file filprofiler-0.6.0-cp37-cp37m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: filprofiler-0.6.0-cp37-cp37m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 268.6 kB
  • Tags: CPython 3.7m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.47.0 CPython/3.7.7

File hashes

Hashes for filprofiler-0.6.0-cp37-cp37m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 67201a1c3fbf72ac914a15d3f96c4ab24678bd43d68d91adbbdd11ba47272fb1
MD5 8dc4eb95b1b8b1adfa89875b83d5e082
BLAKE2b-256 df574e04c68635b5a621a693d0208955a3e0ee042cdffc9014221f5a35e5934c

See more details on using hashes here.

File details

Details for the file filprofiler-0.6.0-cp36-cp36m-manylinux1_x86_64.whl.

File metadata

  • Download URL: filprofiler-0.6.0-cp36-cp36m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 557.6 kB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.47.0 CPython/3.8.3

File hashes

Hashes for filprofiler-0.6.0-cp36-cp36m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 54a566f1238692890a7faa65b56ee4196b0f3f1121404b94fcbaef0309a2842b
MD5 732aa73250a20ef41acf9bbccbcd8d1e
BLAKE2b-256 0c616f3132d7f2885f524da91903dcfc93dfd2ea524ef6c78d44daf7e52f8921

See more details on using hashes here.

File details

Details for the file filprofiler-0.6.0-cp36-cp36m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: filprofiler-0.6.0-cp36-cp36m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 268.6 kB
  • Tags: CPython 3.6m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/40.6.2 requests-toolbelt/0.9.1 tqdm/4.47.0 CPython/3.6.10

File hashes

Hashes for filprofiler-0.6.0-cp36-cp36m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 478d722aafee57515da69d109878377ea06329dad89e873e55de4fbde0ebcf4d
MD5 7f1e11cc02b0959508ab6d48ea6543b6
BLAKE2b-256 b3e07d25e118a695f94eae74ebeb0ac8f835f42f9e4fbc368151f284200602f2

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page