Skip to main content

Scale Distribution Framework

Project description

Scaled

This project is aiming the target that provides simple and efficient and reliable way for distributing computing framework, centralized scheduler and stable protocol when client and worker talking to scheduler

Introduction

The goal for this project should be as simple as possible

  • It built on top of zmq
  • it has ready python version of Client, Scheduler, Worker
  • I will provide golang or Rust version of Scheduler, the goal for the Scheduler should be completely computer language agnostic, which means they follow the same protocol
  • Scheduler might support function based computing tree in the future

Installation

pip install scaled

if you want to use uvloop, please do: pip install uvloop, default we are using python builtin uvloop

How to use it

Start local scheduler and cluster at the same time in the code

import random

from scaled.client import Client
from scaled.cluster.combo import SchedulerClusterCombo


def calculate(sec: int):
    return sec * 1


def main():
    address = "tcp://127.0.0.1:2345"

    cluster = SchedulerClusterCombo(address=address, n_workers=10, event_loop="uvloop")
    client = Client(address=address)

    tasks = [random.randint(0, 100) for _ in range(100000)]
    futures = [client.submit(calculate, i) for i in tasks]

    results = [future.result() for future in futures]

    assert results == tasks

    client.disconnect()
    cluster.shutdown()


if __name__ == "__main__":
    main()

Start scheduler and cluster independently

use scaled_scheduler to start scheduler, for example:

scaled_scheduler tcp://0.0.0.0:8516

use scaled_cluster to start 10 workers:

scaled_worker -n 10 tcp://127.0.0.1:8516

for detail options of above 2 program, please use argument -h to check out all available options

Then you can write simply write client code as:

from scaled.client import Client


def foobar(foo: int):
    return foo


client = Client(address="tcp://127.0.0.1:2345")
future = client.submit(foobar, 1)

print(future.result())

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

scaled-0.18.tar.gz (24.0 kB view details)

Uploaded Source

Built Distribution

scaled-0.18-py3-none-any.whl (34.4 kB view details)

Uploaded Python 3

File details

Details for the file scaled-0.18.tar.gz.

File metadata

  • Download URL: scaled-0.18.tar.gz
  • Upload date:
  • Size: 24.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.1

File hashes

Hashes for scaled-0.18.tar.gz
Algorithm Hash digest
SHA256 32ff5260f7136c5674547e1d103053c03341625a11178c55a056792d3aa3ece9
MD5 ebce97a9f875889a0555c6e9c6cd6257
BLAKE2b-256 b563e6b7e49a02407b98153bca87487d76415ff5e568b2596ba522851adf7cc8

See more details on using hashes here.

File details

Details for the file scaled-0.18-py3-none-any.whl.

File metadata

  • Download URL: scaled-0.18-py3-none-any.whl
  • Upload date:
  • Size: 34.4 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.1

File hashes

Hashes for scaled-0.18-py3-none-any.whl
Algorithm Hash digest
SHA256 24801a0d89a35d2fd73d15f809be3fb777019977115d87343c169e3acc58326b
MD5 33404b0fe9e8632f20587ec337dbe968
BLAKE2b-256 4c3a1dfb473eedfc1bef77ebca0c7de5a5e21236d88c20edbced767654739cb2

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page