Skip to main content

A LIBRARY FOR DISTRIBUTED COMPUTING IN PYTHON

Project description

DISCOMPUTIFY

  • PROJECT LINK GITHUB

  • PROJECT LINK PYPI

  • A library written purely in python to do distributed computing using CYPHER-PROTOCOL.

  • Many times the task is very large and only single thread or single computer is not sufficient to do the computation, specially when the tasks are independent of each other or partially depend on each other.

  • Thus we need to run the programs in parallel maner to complete the computation process faster and effeciently.

COMPATIBLITY

  • READPY works with Linux, OS-X, and Windows (OS INDEPENDENT).

  • Supported on python 3.10 and above

INSTALLATION

Install it using pip.

USAGE

CONFIGURATION

  • For configuration of server and client need a configuration file named discomputify_conf.json

CONFIG PARAMETERS

  • RECV_BUFFER and TRANSMISSION_BUFFER are parameters both should have same corresponding value for both server and client configuration.

  • ENCRYPTION_KEY and DECRYPTION_KEY are parameters which ensure encryption of data sent over connection, see CYPHER-PROTOCOL documentation for more information.

  • TIMEOUT is the time after which a connection is closed or reset from client side or server side.

  • RECV_CHUNK_SIZE and TRANSMISSION_CHUNK_SIZE are the parameters that control chunk size for file transfers

  • DEBUG1 and DEBUG2 control the debug levels of CYPHER_PROTOCOL

  • INPUT_PATH and OUTPUT_PATH are parameters that specify the relative path for storing and retrieving input and output files

  • FTP_NODE, QUERY_NODE, MASTER_NODE, SHARED_MEM_NODE specify the addresses where the particiler service nodes are running

  • FTP_PORT, QUERY_PORT, MASTER_PORT, SHARED_MEM_PORT specify the ports at which these particular services are running or will be running

  • FTP_BASE_PATH is to set the base path of files in FTP so that there is no clash between namespaces and files are perfectly organised

  • COMPUTING_NODES is an array of addresses of computing services available

  • COMP_NODE is used to set address of computing node while starting a computing service

  • Example discomputify_conf is given in CONF_FILES folder, consider that for more details.

STARTING A COMPUTING SERVICE

  • Put discomputify_conf.json in a directory and run the command
python3 -m DISCOMPUTIFY computing_node

STARTING A FTP SERVICE

  • Put discomputify_conf.json in a directory and run the command
python3 -m DISCOMPUTIFY ftp_node

STARTING A MASTER NODE SERVICE

  • Put discomputify_conf.json in a directory and run the command
python3 -m DISCOMPUTIFY master_node

USAGE OF CLIENT FOR DISTRIBUTED COMPUTING

  • For starting computation, create a directory and put all the input files for the program in a directory as specified in discompitify_conf.json

  • then add the program that you want to run with all of its dependencies in the directory, but make sure that the starting poin of your program should be main.py

  • See an example of main.py in test_scripts

  • Here is the example for using the CLIENT_NODE for computing

from DISCOMPUTIFY.CLIENT_NODE.client_node import CLIENT_NODE
import time

if __name__ == "__main__" :
    CLIENT_NODE_OBJ = CLIENT_NODE()
    CLIENT_NODE_OBJ.set_process_id("1684340918413322")

    print("UPLOADING FILES")
    CLIENT_NODE_OBJ.upload_files("qwe")
    print("UPLOADING COMPLETE")

    print("STARTING COMPUTING")
    print(CLIENT_NODE_OBJ.start_computing())
    print("COMPUTING STARTED")

    time.sleep(1)
    
    print("GETTING NODE LOAD")
    print(CLIENT_NODE_OBJ.get_node_load())
    print("GETTING PROCESS STATUS")
    
    print(CLIENT_NODE_OBJ.get_process_status())
    
    #print("TERMINATING PROCESS")
    #CLIENT_NODE_OBJ.stop_computing()
    #print("PROCESS TERMINATED")

    print("GETTING PROCESS STATUS")
    print(CLIENT_NODE_OBJ.get_process_status())
    print("GOT PROCESS STATUS")

    print("GETTING FILES")
    CLIENT_NODE_OBJ.download_files()
    print("DOWNLOADED FILES")

    #print("CLEARING DATA")
    #print(CLIENT_NODE_OBJ.clear_data())
    #print("CLEARED DATA")

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

discomputify_P-Y-R-O-B-O-T-1.0.0.tar.gz (46.4 kB view details)

Uploaded Source

Built Distribution

discomputify_P_Y_R_O_B_O_T-1.0.0-py3-none-any.whl (36.8 kB view details)

Uploaded Python 3

File details

Details for the file discomputify_P-Y-R-O-B-O-T-1.0.0.tar.gz.

File metadata

File hashes

Hashes for discomputify_P-Y-R-O-B-O-T-1.0.0.tar.gz
Algorithm Hash digest
SHA256 41a0e04e57b8c486948bb45ef9117ed52f567290b38f3376a0327d21dffd5f22
MD5 e0061ead123707339d9c0f3c23400141
BLAKE2b-256 0c91e76d39bf0b5a222f87143e69a71c9d9ad56d127a84bdf5328b23e5eaa54a

See more details on using hashes here.

File details

Details for the file discomputify_P_Y_R_O_B_O_T-1.0.0-py3-none-any.whl.

File metadata

File hashes

Hashes for discomputify_P_Y_R_O_B_O_T-1.0.0-py3-none-any.whl
Algorithm Hash digest
SHA256 84522be09c9f74843d23722a41c946c12a8d3ac37838e63407dc1524414c6fdb
MD5 089e39ea45b9509ee1ff3be53cb127ec
BLAKE2b-256 ab3195ea23d5bfae0e4ee79eab929a9147fa23efde63b2e0a3148cbe3cae0cee

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page