fflib 0.3.0

PyTorch-based library for the Forward-Forward algorithm

Forward-Forward Neural Networks Library

FFLib is a neural network library based on PyTorch [2] that aims to implement several different types of layers and networks based on the Forward-Forward algorithm [1]. The library also provides a suite of tools for training, validating, testing, debugging and experimenting with Forward-Forward-based networks. We aim to make this library as close as possible to the original design and structure of the PyTorch library.

Getting Started

Installing dependencies

Dependencies needed to run the code and/or the examples:

• PyTorch
• torchvision
• tqdm

Examples

To get started with our library check out the examples in the ./examples folder. You can run the examples either from the CLI or from inside VSCode interactively. We recommend running them from the CLI for the first time, so you can fix any dependency issues and making interactive execution only afterwards.

The Forward-Forward Algorithm

The Forward-Forward Algorithm was introduced in Geoffrey Hinton's paper "The Forward-Forward Algorithm: Some Preliminary Investigations" [1] with the following abstract:

The aim of this paper is to introduce a new learning procedure for neural networks and to
demonstrate that it works well enough on a few small problems to be worth further investigation.
The Forward-Forward algorithm replaces the forward and backward passes of backpropagation by
two forward passes, one with positive (i.e. real) data and the other with negative data which
could be generated by the network itself. Each layer has its own objective function which is
simply to have high goodness for positive data and low goodness for negative data.
The sum of the squared activities in a layer can be used as the goodness
but there are many other possibilities, including minus the sum of the squared activities.
If the positive and negative passes could be separated in time,
the negative passes could be done offline, which would make the learning much simpler
in the positive pass and allow video to be pipelined through the network without ever
storing activities or stopping to propagate derivatives.

Types of FF Networks

There are 3 different types of Forward-Forward-based Neural Networks implemented in the FFLib:

• FFNet - example usage
• FF+C - example usage
• FFRNN - example usage

[!NOTE] There's also a backpropagation example implemented as a base case.

FFNet

The basic example of a Neural Network based on the Forward-Forward Algorithm. In the example file the following network specifications are used:

• 2 Dense layers with 2000 neurons each
• Predicting the MNIST dataset with TryAll probe
• Batch Size of 128

When a dense layer accepts an input, it is first detached from the graph and normalized before applying the weight multiplication. Essentially, it's just a linear layer that gets a detached normalized input. The trick is that each layer optimizes its "goodness" to be positive for "positive" data and negative for "negative" data. In the case of FFNet, the input and output has to be combined and given as an input to the network (to the first layer). Since there's no output of the network, we have to try all inputs when predicting. In the FFLib, there are special classes called probes. We use the TryAll probe to try all possible one-hot labels.

FF+C

FF+C is a type of hybrid Neural Network that uses both, the Forward-Forward Algorithm and ordinary backpropagation to avoid the need ot the TryAll probe. The FF part is used as a feature extractor and it is trained the same way as the original FF network. The Classifier is another dense layer that as input gets the concatenated activations from both of the FF Dense layers and tries to predict the one-hot label.

FFRNN

The Forward-Forward Recurrent Neural Network is the most complex type of FF network implemented in the FFLib. It is made of a special type of layer called FFRecurrentLayer. The example file contains an example usage of the FFRNN with 2 FF Recurrent Layers, both with 2000 neurons. These networks are quite large due to the fact that each layer has not only weights from the previous layer, but also backward weights from the next layer. These networks have to be trained with multiple frames per batch, thus requiring even more time for both, training and inference.

Utilities

We aim to implement many small utilities for training, validating, testing, developing, debugging and analysing FF networks. For each type of network, we have build small network-specifc train-test suites that allow you to quickly train and test a FF network with some specific DataProcessor. For each new dataset, you just need to define a data processor. The suite will take in the network and the data processor and train the network in the way it is supposed to. There are currently implemented 4 different testing suites:

• FF Suite
• FF+C Suite
• FFRNN Suite
• BP Suite

Furthermore, each network implements some analysis functions, currently only a sparsity function is implemented. Sparsity can be computed in two ways HOYER [3] and ENTROPY_BASED.

Contributions

We really appreciate contributions from the community! We especially welcome the reports of issues and bugs.

However, one may note that since this library is currently being heavily developed, the API may drastically change and all projects depending on this library have to deal with the changes downstream. We will however try to keep these at minimum.

The main maintainer of this library is Mitko Nikov.

Developing the library

We are using poetry to manage, build and publish the python package. We recommend downloading poetry and running poetry install to install all of the dependencies instead of doing so manually.

To activate the virtual env created by poetry, run poetry env activate to get the command to activate the env. After activation, you can run anything from within.

Contributing to GitHub

There are three things that we are very strict about:

• Type-checking - powered by mypy
• Coding style - powered by Black
• Unit Tests - powered by pytest

Run the following commands in the virtual env to ensure that everything is according to the guidelines:

mypy . --strict
black .
pytest .

Guidelines are now checked using GitHub Workflows. When developing the library locally, you can install act to run the GitHub workflows on your machine through Docker. We also recommend installing the VSCode extension GitHub Local Actions to run the workflows from inside VSCode, making the process painless.

Example scenarios are also tested in GitHub Actions by running them from the CLI.

General Guidelines

Here are a few guidelines to following while contributing on the library:

• We aim to keep this library with as little run-time-necessary dependencies as possible.
• Unit tests for as many functions as possible. (we know that we can't cover everything)
• Strict Static Type-checking using mypy
• Strict formatting style guidelines using black
• No recursion (at our abstraction level)
• Nicely documented functions and classes

References

• [1] - Hinton, G. (2022). The Forward-Forward Algorithm: Some preliminary investigations.
• [2] - PyTorch.
• [3] - Hoyer, P. (2004). Non-negative Matrix Factorization with Sparseness Constraints.