Skip to main content

Deploy Dask on Marathon

Project description

Build Status

Deploy dask-worker processes on Marathon in response to load on a Dask scheduler. This creates a Marathon application of dask-worker processes. It watches a Dask Scheduler object in the local process and, based on current requested load, scales the Marathon application up and down.


It’s not yet clear how to expose all of the necessary options to a command line interface. For now we’re doing everything manually.

Make an IOLoop running in a separate thread:

with MarathonCluster(marathon='http://localhost:8080',
                     cpus=1, mem=512, adaptive=True) as mc:
    with Client(mc.scheduler_address) as c:
        x = c.submit(lambda x: x + 1, 1)
        assert x.result() == 2

Create a Client and submit work to the scheduler. Marathon will scale workers up and down as neccessary in response to current workload.

from distributed import Client
c = Client(s.address)

future = c.submit(lambda x: x + 1, 10)


  • [x] Deploy the scheduler on the cluster
  • [x] Support a command line interface

Docker Testing Harness

This sets up a docker cluster of one Mesos master and two Mesos agents using docker-compose.


  • docker version >= 1.11.1
  • docker-compose version >= 1.7.1
docker-compose up

Run py.test:

py.test dask-marathon

Web UIs


Dask-marathon originally forked from

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for daskathon, version 1.1.1
Filename, size File type Python version Upload date Hashes
Filename, size daskathon-1.1.1.tar.gz (5.4 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page