substrafl 1.0.0rc2

A high-level federated learning Python library to run federated learning experiments at scale on a Substra network

Substra is an open source federated learning (FL) software. It enables the training and validation of machine learning models on distributed datasets. It provides a flexible Python interface and a web application to run federated learning training at scale. This specific repository is about SubstraFL the high-level federated learning Python library based on the low-level Substra python library. SubstraFL is used to run complex federated learning experiments at scale.

Substra's main usage is in production environments. It has already been deployed and used by hospitals and biotech companies (see the MELLODDY project for instance). Substra can also be used on a single machine to perform FL simulations and debug code.

Substra was originally developed by Owkin and is now hosted by the Linux Foundation for AI and Data. Today Owkin is the main contributor to Substra.

Join the discussion on Slack and subscribe here to our newsletter.

How to install

pip install substrafl

To start using Substra

Have a look at our documentation.

Try out our MNIST example.

Support

If you need support, please either raise an issue on Github or ask on Slack.

Contributing

Substra warmly welcomes any contribution. Feel free to fork the repo and create a pull request.

How to test

Install substrafl in editable mode with developper dependencies. In addition, install substra and substra-tools in editable mode. It is recommended to install all the libraries in a Python virtual env.

git clone [e-mail]:Substra/substrafl.git
pip install -e "substrafl[dev]"
git clone [e-mail]:Substra/substra.git
pip install -e substra
git clone [e-mail]:Substra/substra-tools.git
pip install -e substra-tools

Now you can use the following command from subtrafl top level directory to run tests:

cd substrafl
make test-subprocess

Running advanced test suites

Substra can be used in three different modes: using Python subprocesses (subprocess), using Docker (docker) and using Kubernetes (remote).

The command make test-subprocess runs the test suite in subprocess mode. It's lightweight and perfect to start.

To test with the Docker mode, you will need Docker installed and running on your machine. If necessary, install it using Docker Desktop.

The following command runs the test suites in subprocess and Docker mode:

make test-local
``

Please be warned that some of these tests are slow and the whole test suite might require a couple hours to complete.

To try out a local deployment with Kubernetes, please follow the [installation instructions](https://docs.substra.org/en/stable/contributing/local-deployment.html) provided in the documentation.
The following command runs the remote tests:

```sh
make test-remote

How to generate the changelog

The changelog is managed with towncrier. To add a new entry in the changelog, add a file in the changes folder. The file name should have the following structure: <unique_id>.<change_type>. The unique_id is a unique identifier, we currently use the PR number. The change_type can be of the following types: added, changed, removed, fixed.

To generate the changelog (for example during a release), use the following command (you must have the dev dependencies installed):

towncrier build --version=<x.y.z>

You can use the --draft option to see what would be generated without actually writing to the changelog (and without removing the fragments).

Appendix

Building the documentation

The API documentation is generated from the SubstraFL repository thanks to the auto doc module. It is automatically built by https://github.com/Substra/substra-documentation and integrated into the general documentation here.

You can build the API documentation locally to see the changes made by your PR.

Requirements

You need to have substrafl.dev installed on your machine and some extra requirements. From the SubstraFL repo:

pip install -e '.[dev]'
cd docs
pip install -r requirements.txt

Build

You can build the documentation to see if your changes are well taken into account. From the ./docs folder :

make clean html

No warning should be thrown by this command.

Then open the ./docs/_build/index.html file to see the results.

You can also generate the documentation live so each of your changes are taken into account on the fly:

make livehtml

NB: Sometimes make livehtml does not take changes into account so running the make html command in parallel might be needed.