daskcheck 0.0.4

Automatically created environment for python package

Project daskcheck

Help with a simple use of the dask

The idea

1. define properly the core function (xcorefunc in the example) with THE return
2. daskcheck then will take care about:
   • sending the parameters
   • collecting results and saving them to local json file

Instalation of daskcheck

pip install daskcheck

Installation of dask

See https://docs.dask.org/en/stable/install.html

pip install "dask[complete]"

Launching dask scheduler/workers

Pay attention to correct/compatible libraries on different workers

Environment needed

to be done

Launching scheduler

dask scheduler --port 8786

Launching worker

dask     worker 127.0.0.1:8786 --nworkers 5 --nthreads 1

Testing dask

IN DEVELOPMENT...

This runs (sched and workers are ON) 40x getcpuinfo

./daskcheck.py test

This is solved .... possible to load (unload first). See tesmod.py

./daskcheck.py dask py_file_with_main  1,3
./daskcheck.py dask py_file_with_main  11..33

Monitoring dask

xdg-open http://localhost:8787

Recollection the data from json

Usage

from daskcheck import daskcheck

from fire import Fire
import time
import platform
import datetime as dt
import json

def main( parlist ):
    parameters = daskcheck.prepare_params( parlist )

    if type(parameters)==list:
        print("i... viable for DASK ....")
        daskcheck.submit( daskcheck.get_cpu_info , parameters)
    else:
        print("i... running only locally")
        my_results = xcorefunc( 1 , parameters )
        # Write LOG file.
        now = dt.datetime.now()
        stamp = now.strftime("%Y%m%d_%H%M%S")
        with open(f"dask_results_log_{stamp}.json", "w") as fp:
            json.dump( my_results , fp, sort_keys=True, indent='\t', separators=(',', ': '))
    return

def xcorefunc( order, param):
    """
    Function to be sent to dask server with order# + parameter
    """
    import ROOT # I need to avoid breaking pickle
    start_time = time.perf_counter()

    return order, [platform.node(),  f"{time.perf_counter() - start_time:.1f} s" , ni]


if __name__=="__main__":
    Fire(main)