peerannot 0.0.1.post43

Running aggregation strategies

In python, we can run classical aggregation strategies from the current dataset as follows

for strat in ["majority_voting", "naive_soft", "dawid_skene", "glad", "wds"]:
    ! peerannot aggregate . -s {strat}

This will create a new folder names labels containing the labels in the labels_cifar10H_${strat}.npy file.

Training your network

Once the labels are available, we can train a neural network with PyTorch as follows. In a terminal:

for strat in ["majority_voting", "naive_soft", "dawid_skene", "glad", "wds"]:
    ! peerannot train . -o cifar10H_${strat} \
                -K 10 \
                --labels=./labels/labels_cifar-10h_${strat}.npy \
                --model resnet18 \
                --img-size=32 \
                --n-epochs=1000 \
                --lr=0.1 --scheduler -m 100 -m 250 \
                --num-workers=8

End-to-end strategies

Finally, for the end-to-end strategies using deep learning (as CoNAL or CrowdLayer), the command line is:

peerannot aggregate-deep . -o cifar10h_crowdlayer \
                     --answers ./answers.json \
                     --model resnet18 -K=10 \
                     --n-epochs 150 --lr 0.1 --optimizer sgd \
                     --batch-size 64 --num-workers 8 \
                     --img-size=32 \
                     -s crowdlayer

For CoNAL, the hyperparameter scaling can be provided as -s CoNAL[scale=1e-4].

Create you own dataset

The dataset.py is not mandatory but is here to facilitate the dataset’s installation procedure. A minimal example:

class mydataset:
    def __init__(self):
        self.DIR = Path(__file__).parent.resolve()
        # download the data needed
        # ...

    def setfolders(self):
        print(f"Loading data folders at {self.DIR}")
        train_path = self.DIR / "train"
        test_path = self.DIR / "test"
        valid_path = self.DIR / "val"

        # Create train/val/test tasks with matching index
        # ...

        print("Created:")
        for set, path in zip(
            ("train", "val", "test"), [train_path, valid_path, test_path]
        ):
            print(f"- {set}: {path}")
        self.get_crowd_labels()
        print(f"Train crowd labels are in {self.DIR / 'answers.json'}")

    def get_crowd_labels(self):
        # create answers.json dictionnary in presented format
        # ...
        with open(self.DIR / "answers.json", "w") as answ:
            json.dump(dictionnary, answ, ensure_ascii=False, indent=3)