fedbench 0.1.1

A Federated Learning Benchmarking Framework

Federated Learning Benchmark Framework

Website | Paper

A strategic framework for systematically benchmarking federated learning algorithms across diverse datasets and client distributions. Built for researchers and practitioners to make informed algorithm selection decisions based on rigorous comparative analysis.

Strategic Overview

FLBenchmark provides a comprehensive ecosystem for evaluating federated learning strategies in realistic scenarios. Our framework enables:

• Strategic Algorithm Selection: Identify optimal algorithms for specific data distributions and application constraints
• Performance Trade-off Analysis: Evaluate critical trade-offs between accuracy, communication efficiency, and privacy preservation
• Heterogeneity Impact Assessment: Measure how different heterogeneity factors affect algorithm performance
• Systematic Comparison Methodology: Ensure fair, reproducible comparisons between federated optimization approaches

Key Strategic Capabilities

• Algorithm Portfolio: Production-ready implementations of 8+ state-of-the-art federated optimization strategies
• Heterogeneity Simulation: Fine-grained control over client data distributions to simulate real-world deployment challenges
• Multi-dimensional Evaluation: Comprehensive metrics covering accuracy, efficiency, fairness, and robustness
• Decision-Supporting Visualizations: Analytics tools for identifying optimal algorithm selection criteria

Quick Start Tutorials

fedBench's mission is to democratize federated learning by providing researchers and practitioners with an intuitive, flexible, and powerful framework. This comprehensive toolkit allows you to implement, customize, and evaluate federated learning systems with minimal effort.

pip install fedbench

0. The statistics of datasets?

   (or open the Jupyter Notebook)

Direct Repository Usage

# Clone the repository
git clone [e-mail]:NechbaMohammed/FLBenchmark.git
cd FLBenchmark
# Install dependencies
pip install -r requirements.txt

Strategic Dataset Selection

Choose datasets that match your strategic evaluation needs:

Dataset	Classes	Partitioning Method	Partition Settings
MNIST, FMNIST, SVHN, CINIC-10, CIFAR-10	10	label_quantity, dirichlet, iid_noniid, noise, iid	labels_per_client, alpha, similarity, segma
FedISIC2019	8	label_quantity, dirichlet, iid_noniid, noise, iid	labels_per_client, alpha, similarity, segma
CIFAR-100	100	label_quantity, dirichlet, iid_noniid, noise, iid	labels_per_client, alpha, similarity, segma
Adult	2	label_quantity, dirichlet, iid_noniid, iid	labels_per_client, alpha, similarity
FCUBE	2	synthetic	-
FEMNIST	62	real-world	-

Dataset Usage Examples

Below are examples demonstrating how to use the FLBenchmark dataset module.

from fedbench.datasets import FederatedDataset
from omegaconf import DictConfig

# Example 1: MNIST Dataset

data_config = DictConfig({"name": "mnist", "partitioning": "iid", "batch_size": 64})

federated_dataset = FederatedDataset(data_config, num_clients=1)
trainloaders, valloaders, testloader = federated_dataset.get_dataloaders()
federated_dataset.print_dataset_stats()

Number of training instances: 60000
Number of test instances: 10000
Number of features: 784
Number of classes: 10

---

# Example 2: Adult Dataset

data_config = DictConfig({"name": "adult", "partitioning": "synthetic", "batch_size": 64})

federated_dataset = FederatedDataset(data_config, num_clients=1)
trainloaders, valloaders, testloader = federated_dataset.get_dataloaders()
federated_dataset.print_dataset_stats()

Number of training instances: 26048
Number of test instances: 6513
Number of features: 99
Number of classes: 2

---

# Example 3: FEMNIST Dataset

data_config = DictConfig({"name": "femnist", "partitioning": "real-world", "batch_size": 64})

federated_dataset = FederatedDataset(data_config, num_clients=1)
trainloaders, valloaders, testloader = federated_dataset.get_dataloaders()
federated_dataset.print_dataset_stats()

Number of training instances: 649184
Number of test instances: 165093
Number of features: 784
Number of classes: 62

# Example 5: Label Quantity Partitiin

data_config = DictConfig({"name": "mnist", "partitioning": "label_quantity","labels_per_client":2, "batch_size": 64})

federated_dataset = FederatedDataset(data_config, num_clients=10)
trainloaders, valloaders, testloader = federated_dataset.get_dataloaders()

# Example 6: Dirichlet Partitiin

data_config = DictConfig({"name": "mnist", "partitioning": "dirichlet","alpha":0.5, "batch_size": 64})

federated_dataset = FederatedDataset(data_config, num_clients=10)
trainloaders, valloaders, testloader = federated_dataset.get_dataloaders()

# Example 7: Quantity Skew Partitiin

data_config = DictConfig({"name": "mnist", "partitioning": "iid_noniid","similarity":0.5, "batch_size": 64})

federated_dataset = FederatedDataset(data_config, num_clients=10)
trainloaders, valloaders, testloader = federated_dataset.get_dataloaders()

# Example 8: Feature Distribution Partitiin

data_config = DictConfig({"name": "mnist", "partitioning": "noise","segma":0.1, "batch_size": 64})

federated_dataset = FederatedDataset(data_config, num_clients=10)
trainloaders, valloaders, testloader = federated_dataset.get_dataloaders()

Strategic Algorithm

🧠 Model Configurations

FedBench provides pre-configured models for various datasets:

Dataset	Model Type	Input Dim	Hidden Dims	Num Classes	MOON Variant
MNIST, FMNIST, FEMNIST	MNISTModel	256	[120, 84]	10 (MNIST, FMNIST), 62 (FEMNIST)	MnistModelMOON
CIFAR-10, SVHN, CINIC-10, FedISIC2019	CNN	400	[120, 84]	10	CNNModelMOON
CIFAR-100	CNN	400	[120, 84]	100	CNNModelMOON
Adult	MLP	99	[32, 16, 8]	2	MLPModelMOON
FCUBE	MLP	3	[32, 16, 8]	2	MLPModelMOON

📖 Usage Examples

FedAvg on MNIST

from omegaconf import OmegaConf
import torch
from fedbench.algorithms.fedavg.simulation import run_fedavg

model_cfg = OmegaConf.create({
    "_target_": "fedbench.models.MNISTModel",
    "input_dim": 256,
    "hidden_dims": [120, 84],
    "num_classes": 10,
})

backend_config = {
    "num_cpus": 1,
    "num_gpus": 0
}

data_config = {
    "name": "mnist",
    "partitioning": "iid",
    "batch_size": 64,
}

history = run_fedavg(
    data_config=data_config,
    model_cfg=model_cfg,
    backend_config=backend_config,
    num_clients=10,
    num_rounds=20,
    num_epochs=15,
    learning_rate=0.01,
    device=torch.device("cuda" if torch.cuda.is_available() else "cpu"),
)

FedProx on CIFAR-10

from omegaconf import OmegaConf
import torch
from fedbench.algorithms.fedprox.simulation import run_fedprox

model_cfg = OmegaConf.create({
    "_target_": "fedbench.models.CNN",
    "input_dim": 400,
    "hidden_dims": [120, 84],
    "num_classes": 10,
})

backend_config = {
    "num_cpus": 1,
    "num_gpus": 0
}

data_config = {
    "name": "cifar10",
    "partitioning": "dirichlet",
    "alpha":0.5,
    "batch_size": 32,
}

history = run_fedprox(
    data_config=data_config,
    model_cfg=model_cfg,
    backend_config=backend_config,
    num_clients=20,
    num_rounds=30,
    num_epochs=10,
    learning_rate=0.005,
    mu=0.01,
    device=torch.device("cuda" if torch.cuda.is_available() else "cpu"),
)

FedAdam on CIFAR-100

from omegaconf import OmegaConf
import torch
from fedbench.algorithms.fedadam.simulation import run_fedadam

model_cfg = OmegaConf.create({
            "_target_": "fedbench.models.CNN",
            "input_dim": 400,
            "hidden_dims": [120, 84],
            "num_classes": 100,
        })

backend_config = {
    "num_cpus": 1,
    "num_gpus": 0
}

data_config = {
    "name": "cifar100",
    "partitioning": "iid",
    "batch_size": 32,
}

history = run_fedadam(
    data_config=data_config,
    model_cfg=model_cfg,
    backend_config=backend_config,
    num_clients=100,
    num_rounds=25,
    num_epochs=20,
    learning_rate=0.001,
    beta_1=0.9,
    beta_2=0.999,
    device=torch.device("cuda" if torch.cuda.is_available() else "cpu"),
)

FedAdagrad on SVHN

from omegaconf import OmegaConf
import torch
from fedbench.algorithms.fedadagrad.simulation import run_fedadagrad

backend_config = {
    "num_cpus": 1,
    "num_gpus": 0
}
 model_cfg = OmegaConf.create({
            "_target_": "fedbench.models.CNN",
            "input_dim": 400,
            "hidden_dims": [120, 84],
            "num_classes": 10,
        })
data_config = {
    "name": "svhn",
    "partitioning": "noise",
    "segma":0.1,
    "batch_size": 32,
}

history = run_fedadagrad(
    data_config=data_config,
    model_cfg=model_cfg,
    backend_config=backend_config,
    num_clients=10,
    num_rounds=25,
    num_epochs=20,
    learning_rate=0.001,
    device=torch.device("cuda" if torch.cuda.is_available() else "cpu"),
)

FedYogi on CINIC-10

from omegaconf import OmegaConf
import torch
from fedbench.algorithms.fedyogi.simulation import run_fedyogi

model_cfg = OmegaConf.create({
            "_target_": "fedbench.models.CNN",
            "input_dim": 400,
            "hidden_dims": [120, 84],
            "num_classes": 10,
        })

backend_config = {
    "num_cpus": 1,
    "num_gpus": 0
}
data_config = {
    "name": "cinic10",
    "partitioning": "iid",
    "batch_size": 32,
}

history = run_fedyogi(
    data_config=data_config,
    model_cfg=model_cfg,
    backend_config=backend_config,
    num_clients=10,
    num_rounds=25,
    num_epochs=20,
    learning_rate=0.001,
    beta_1=0.9,
    beta_2=0.999,
    device=torch.device("cuda" if torch.cuda.is_available() else "cpu"),
)

FedNova on FCUBE

from fedbench.algorithms.fednova.simulation import run_fednova
from omegaconf import OmegaConf
import torch

backend_config = {
    "num_cpus": 1,
    "num_gpus": 0
}

model_cfg = OmegaConf.create({
            "_target_": "fedbench.models.MLP",
            "input_dim": 3,
            "hidden_dims" : [32, 16, 8] ,
            "num_classes": 2,
        })
data_config = {
            "name": "fcube",
            "partitioning": "synthetic",
            "batch_size": 64,
        }
history = run_fednova(
    data_config=data_config,
    model_cfg=model_cfg,
    backend_config=backend_config,
    num_clients=15,
    num_rounds=25,
    num_epochs=20,
    learning_rate=0.001,
    device=torch.device("cuda" if torch.cuda.is_available() else "cpu"),
)

Scaffold on FedISIC2019

from fedbench.algorithms.scaffold.simulation import run_scaffold
from omegaconf import OmegaConf
import torch

backend_config = {
    "num_cpus": 1,
    "num_gpus": 0
}
model_cfg = OmegaConf.create({
            "_target_": "fedbench.models.CNN" ,
            "input_dim": 400,
            "hidden_dims": [120, 84],
            "num_classes": 10,
        })
data_config = {
    "name": "fedisic2019",
    "partitioning": "iid_noniid",
    "similarity":0.5,
    "batch_size": 32,
}


history = run_scaffold(
    data_config=data_config,
    model_cfg=model_cfg,
    backend_config=backend_config,
    num_clients=15,
    num_rounds=25,
    num_epochs=20,
    learning_rate=0.001,
    model_dir="weights",
    device=torch.device("cuda" if torch.cuda.is_available() else "cpu"),
)

MOON on Adult

from fedbench.algorithms.moon.simulation import run_moon
from omegaconf import OmegaConf
import torch

backend_config = {
    "num_cpus": 1,
    "num_gpus": 0
}

model_cfg = OmegaConf.create({
            "_target_": "fedbench.models.MLPModelMOON",
            "input_dim": 99,
            "hidden_dims" : [32, 16, 8] ,
            "output_dim":256,
            "num_classes": 2,
})

data_config = {
    "name": "adult",
    "partitioning": "iid",
    "batch_size": 64,
}

history = run_moon(
    data_config=data_config,
    model_cfg=model_cfg,
    backend_config=backend_config,
    num_clients=10,
    num_rounds=25,
    num_epochs=20,
    learning_rate=0.001,
    model_dir="weights",
    device=torch.device("cuda" if torch.cuda.is_available() else "cpu"),
)

FedBN on FEMNIST

from fedbench.algorithms.fedbn.simulation import run_fedbn
from omegaconf import OmegaConf
import torch

backend_config = {
    "num_cpus": 1,
    "num_gpus": 0
}
 model_cfg = OmegaConf.create({
            "_target_": "fedbench.models.MNISTModel",
            "input_dim": 256,
            "hidden_dims": [120, 84],
            "num_classes": 62,
        })
data_config = {
    "name": "femnist",
    "partitioning": "real-world",
    "batch_size": 32,
}


history = run_fedbn(
    data_config=data_config,
    model_cfg=model_cfg,
    backend_config=backend_config,
    num_clients=15,
    num_rounds=25,
    num_epochs=20,
    learning_rate=0.001,
     save_path="weights",
    device=torch.device("cuda" if torch.cuda.is_available() else "cpu"),
)

🧪 Benchmarking

You can easily benchmark different algorithms and configurations

# Clone the repository
git clone [e-mail]:NechbaMohammed/FLBenchmark.git
cd FLBenchmark
# Install dependencies
pip install -r requirements.txt

# Run all benchmarks
python benchmark_runner.py

# Aggregate results
python aggregate_experiments.py

# Generate plots
python learning_curve_plots.py
python local_epoch_comparison_plots.py

Cit📚 Citation

If you use FedBench in your research, please cite our paper:ation

@article{,
  title={},
  author={},
  journal={},
  year={}
}

🤝 Contributing

We welcome contributions! Please check out our contribution guidelines for details.