Skip to main content

Client application to interface with the branchkey system

Project description

BranchKey Python Client Application

BK_logo

This application runs against the BranchKey backend aggregation service. It allows to perform federated averaging across a sample of the given files

It provides python interface to login/logout a client, upload files to the system for aggregation, and download aggregated output files. It also spawns a rabbitmq consumer thread to receive updates whenever an aggregated output is available.

Setup Instructions

  • To build the dependencies:
    • make setup, or
    • pip install -r requirements.txt
  • To run the tests: make test
    • make test, or
    • python3 -m unittest -v

Usage instructions:

  • To use a client:
    from branchkey.client import Client
    
    credentials = {"leaf_name": "guest",
                     "leaf_password": "abc123",
                     "tree_id": "tree-1",
                     "branch_id": "group-1",
                     "queue_password": "guest"}
    
    
    # initialise the client
    c = Client(credentials)
    
    # login and authenticate your credentials
    c.login()
    
    # upload the file to the system
    c.file_upload("./file/path")
    
    # Download a file with the file_id value
    # same as the one received from the consumer
    # It downloads the files in the ./aggregated_files directory
    c.file_download("file-id")
    

File format

In src/examples there is a sample weights.npy file.

Weights file numpy format:

[num_samples, [n_d parameter matrix]]
num_samples - the number of samples that contributed to this update
n_d parameter matrix - parameters

From model export; parameter.data.numpy() values for all in parameters to get required file format

NOTE:

These parameters are not the same as those used in the numpy example below

(2486, [['conv1.weight', Parameter containing:
tensor([[[[-4.8906e-02, -1.1447e-03, -2.7956e-02, -1.7628e-01,  1.2711e-01],
          [-1.3940e-02, -1.7490e-01,  1.9408e-01, -1.4146e-01, -1.9384e-01],
          [ 1.6216e-01, -5.7605e-02, -2.6069e-02, -9.5061e-02, -8.6440e-02],
          [ 4.1506e-02, -9.2765e-02,  2.3566e-02, -6.4725e-02,  1.1439e-01],
          [-1.1091e-01,  6.8872e-02,  1.6387e-01,  5.6428e-02,  1.4058e-01]]]]]],
       device='cuda:0', requires_grad=True)], ['conv1.bias', Parameter containing:
tensor([ 0.1031, -0.1715, -0.1133, -0.0628, -0.0625,  0.0822, -0.0405, -0.1773,
         0.1003,  0.0762, -0.0489, -0.1638, -0.1598, -0.0859,  0.0661,  0.1164,
        -0.0803,  0.1263,  0.1396, -0.1557, -0.1488, -0.0836,  0.0559, -0.1944,
        -0.1192, -0.0261, -0.1164,  0.1215, -0.1154, -0.0822,  0.1301, -0.1932],
       device='cuda:0', requires_grad=True)], ['conv2.weight', Parameter containing:
tensor([[[[-1.7591e-02, -1... etc

Required file format

The required numpy arrays after exports

[1329, list([array([[[[ 1.71775490e-01,    [[[ 8.74867663e-02,  5.19692302e-02, -1.64664671e-01,,          -2.23452481e-03,  1.11475676e-01],,    [-1.75505821e-02, -1...
(1329, [array([[[[ 1.71775490e-01,  3.02851666e-02,  2.90171858e-02,
          -4.27578250e-03,  1.14474617e-01],
         [-8.07138346e-03,  1.44909814e-01, -5.36724664e-02,
          -3.51673253e-02, -1.82426855e-01],
         [ 6.75795972e-02, -1.72839850e-01, -7.25025982e-02,
          -1.59504730e-02,  1.60634145e-01],
         [ 6.62277341e-02, -2.26575769e-02, -1.65369093e-01,
          -8.67117420e-02,  1.80021569e-01],
         [-6.11407161e-02, -1.59245610e-01,  1.45820528e-01,
          -5.40512279e-02, -5.19061387e-02]]],
        ....
         [-1.44068539e-01,  6.15987852e-02,  1.83321223e-01,
          -1.79076958e-02, -1.53445438e-01],
         [-7.76787996e-02,  7.64556080e-02,  9.43044946e-02,
           1.63337544e-01, -1.69042274e-01],
         [-8.55994076e-02, -1.23661250e-01,  1.48442864e-01,
          -1.35983482e-01,  2.05254350e-02]]]], dtype=float32), array([ 0.13065006,  0.12797254, -0.12818147, -0.09621437,  0.04100017,
       -0.07248228,  0.02753541,  0.00476395, -0.11270998,  0.11353076,
       -0.0167569 ,  0.12654744, -0.05019006, -0.07281244,  0.03892357,
       -0.09698197, -0.06845284, -0.04604543, -0.01372138, -0.052395  ,
        0.04833373,  0.16228785,  0.09982517,  0.19556762,  0.10631064,
        0.02496212, -0.14297573, -0.10442089,  0.01970248, -0.1684099 ,
       -0.05076171,  0.19325127], dtype=float32), array([[[[-3.42470817e-02,  8.76816106e-04, -2.13724039e-02,
          -2.62880027e-02, -1.86583996e-02],
         [ 2.56936941e-02, -1.97169576e-02, -3.45735364e-02,
          -4.32738848e-03, -1.22306980e-02],
         [ 8.36322457e-03,  3.26042138e-02, -1.50063485e-02,
          -1.85401291e-02,  2.39207298e-02],
         [-1.15280924e-02, -3.47947963e-02,  2.17274204e-02,
           1.80862695e-02,  2.19682772e-02],
...
etc

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

branchkey-2.0.4.2.tar.gz (8.9 kB view hashes)

Uploaded Source

Built Distribution

branchkey-2.0.4.2-py3-none-any.whl (8.0 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page