distributed task distribution framework
Project description
Fleet
Fleet is a generic distributed task distribution framework based on a distributed file system. Task distribution frameworks like Ray and Celery require network connections for communication, which makes them difficult to use in clusters with poor network conditions. Fleet is a distributed framework based on a shared file system, independent of any network communication, allowing for task distribution among nodes without any network connections.
Features
- Distributed task distribution based on a shared file system
- Supports dynamic scaling
- Supports worker node heartbeat
- Supports manager node restart
- Supports set timeout for each task
- Supports set max_job and max_work_time for worker
- Pure Python implementation
Install
pip install open-fleet
Usage
See more examples in ./examples.
# run_manager.py
from fleet.manager import Manager
from fleet.config.config import get_args
base_dir = "./share_dir"
def main():
job_list = [1,2,3,4]
args = get_args(f"--base_dir {base_dir}")
manager = Manager(args=args, job_list=job_list)
manager.run()
if __name__ == '__main__':
main()
# run_worker.py
from fleet.worker import Worker
from fleet.config.config import get_args
from run_manager import base_dir
def add_one(x, info):
return {"status": "success", "result": x + 1}
def main():
args = get_args(f"--base_dir {base_dir}")
worker = Worker(args=args, job_func=add_one)
worker.run()
if __name__ == '__main__':
main()
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
open_fleet-0.0.4.tar.gz
(15.0 kB
view hashes)
Built Distribution
open_fleet-0.0.4-py3-none-any.whl
(15.6 kB
view hashes)
Close
Hashes for open_fleet-0.0.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cb74e18833be2309ab618f6dc8a01dc16d81d24e1035b2014ba3eb9fe0602609 |
|
MD5 | ae2e3c6bba56df73dd986d6a1cccc823 |
|
BLAKE2b-256 | 081cfd59f4b43ec74524a4a8c2f7a2b06c1248f97de384dcccc72aa241581e02 |