Low-impact, task-level memory profiling for Dask.
Project description
dask-memusage
If you're using Dask with tasks that use a lot of memory, RAM is your bottleneck for parallelism. That means you want to know how much memory each task uses:
- So you can set the highest parallelism level (process or threads) for each machine, given available to RAM.
- In order to know where to focus memory optimization efforts.
dask-memusage
is an MIT-licensed statistical memory profiler for Dask's Distributed scheduler that can help you with both these problems.
dask-memusage
polls your processes for memory usage and records the minimum and maximum usage in a CSV:
task_key,min_memory_mb,max_memory_mb
"('from_sequence-map-sum-part-e15703211a549e75b11c63e0054b53e5', 0)",44.84765625,96.98046875
"('from_sequence-map-sum-part-e15703211a549e75b11c63e0054b53e5', 1)",47.015625,97.015625
"('sum-part-e15703211a549e75b11c63e0054b53e5', 0)",0,0
"('sum-part-e15703211a549e75b11c63e0054b53e5', 1)",0,0
sum-aggregate-apply-no_allocate-4c30eb545d4c778f0320d973d9fc8ea6,0,0
apply-no_allocate-4c30eb545d4c778f0320d973d9fc8ea6,47.265625,47.265625
task_key,min_memory_mb,max_memory_mb
"('from_sequence-map-sum-part-e15703211a549e75b11c63e0054b53e5', 0)",44.84765625,96.98046875
"('from_sequence-map-sum-part-e15703211a549e75b11c63e0054b53e5', 1)",47.015625,97.015625
"('sum-part-e15703211a549e75b11c63e0054b53e5', 0)",0,0
"('sum-part-e15703211a549e75b11c63e0054b53e5', 1)",0,0
sum-aggregate-apply-no_allocate-4c30eb545d4c778f0320d973d9fc8ea6,0,0
apply-no_allocate-4c30eb545d4c778f0320d973d9fc8ea6,47.265625,47.265625
Usage
Important: Make sure your workers only have a single thread! Otherwise the results will be wrong.
Installation
On the machine where you are running the Distributed scheduler, run:
$ pip install dask_memusage
API usage
# Add to your Scheduler object, which is e.g. your LocalCluster's scheduler
# attribute:
from dask_memoryusage import install
install(scheduler, "/tmp/memusage.csv")
CLI usage
$ dask-scheduler --preload dask_memusage --memusage.csv /tmp/memusage.csv
Limitations
- Again, make sure you only have one thread per worker process.
- This is statistical profiling, running every 10ms. Tasks that take less than that won't have accurate information.
Help
Need help? File a ticket at https://github.com/itamarst/dask-memusage/issues/new
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for dask_memusage-1.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 32c82e9a97d1133da9519bf3fc25f31f6e4bf4a45d2a941540c3b0716cc6b67f |
|
MD5 | 7fd515960b056169cc4bb71aa3661ca1 |
|
BLAKE2b-256 | a845884279ea392565bded3582f835bbeac8681a900c454338eef22b1848c626 |