Skip to main content

Multilayer Perceptron

Multilayer perceptron (MLP) is a simple neural network (NN), can solve both regression and classification tasks. The project provides a reusable and modular implementation.

[Demo Video]

Requirements

The project is built from scratch with Python, with only Numpy, Pandas, Matplotlib, without high-level ML libraries.

Use Python 3.12 and uv to manage the environment and dependencies. Install uv before continuing. Run the following commands from the project root:

git clone https://github.com/swangarch/multilayer_perceptron
cd multilayer_perceptron

# Create .venv and install the locked dependencies and the project
uv sync --locked

# Alternatively, use the Makefile (runs uv sync)
make setup

Commands below use uv run, so activating the virtual environment is not required. To activate it manually on macOS/Linux, use source .venv/bin/activate.

Usage

Run these commands from the project root. Makefile targets use uv run -m mlp.

uv run -m mlp --help

For example dataset classification

# To split the dataset
make split

# To train with training configuration file and training dataset
make train

# To test with test dataset
make test

Breast cancer diagnosis dataset, a binary classification based on 30 features of a cell.

MNIST handwrite digits classification (0–9)

# To split, train and test the dataset
make digit

For the handwritten digit classification task (0–9), the model achieved 97.19% test accuracy using 33,000 training samples of 28 × 28 images in the optimal configuration.

For the handwritten character recognition task with 94 categories, the model reached 91% accuracy based on 175,000 of 28 × 28 images training samples.

For random generated 1d data regression

# To train and test generated data, with relu activation
make regre-relu

# To train and test generated data, with sigmoid activation
make regre-sigmoid

Packaging

Build the wheel and source distribution with uv:

uv build

# Alternatively
make build

The output files are written to dist/:

  • sw_mlp-1.0.0-py3-none-any.whl
  • sw_mlp-1.0.0.tar.gz

To install the wheel into a Python 3.12 virtual environment:

uv venv --python 3.12
uv pip install dist/sw_mlp-1.0.0-py3-none-any.whl

# macOS/Linux: run the installed package
.venv/bin/python -m mlp --help

The wheel contains the Python package. Example configurations and datasets in the repository's config/ and data/ directories must be distributed separately. Pass their paths when running the installed package. The split command writes to data/train.csv and data/test.csv, so the working directory must contain a data/ directory.

Features

  • Training methods → supports Stochastic Gradient Descent (SGD) and mini-batch training.

  • Visualization → includes real-time animations to track the learning process.

  • Configurable architecture → adjustable network shape, activation functions, initialization methods, max iterations, learning rate, batch size, etc via a configuration file.

  • Data handling → data loading and preprocessing.

  • Support multiple activation functions: ReLU, Sigmoid, Softmax, Leaky ReLU

  • Support multiple loss functions:

    • Categorical Cross Entropy (CCE) with automatic one-hot encoding for categorical labels
    • Binary Cross Entropy (BCE)
    • Mean Squared Error (MSE)
  • Support Fine-tuning: continue training from saved weights for further optimization

Methodology

Demo

A Multilayer Perceptron is a basic artificial neural network that learns to map input data to the correct output by adjusting internal parameters called weights and biases. It is composed of multiple layers of artificial neurons, where each neuron performs a simple computation — combining inputs, multiplying weights, adding a bias, and passing the result through an activation function to introduce non-linearity. Then using back propagation and gradient descent to find ideal weights and biases.

Learning Process

The learning process consists of three main stages:

  • Feedforward — compute predictions by passing data through all layers.
  • Back propagation — calculate gradients of the loss with respect to the weights, using chain rule.
  • Gradient Descent — update weights and biases to minimize the loss over time.

Through repeated iterations of these steps, the MLP gradually improves its predictions and learns to generalize from the training data.

Feedfoward

Demo

In the feedforward step, the input data passes through several layers of neurons. Each neuron performs a weighted sum of its inputs, adds a bias, and then applies an activation function such as ReLU, Sigmoid, or Softmax.

When neurons are grouped into a layer, their computations can be efficiently expressed using matrix multiplication. This allows the model to process many neurons at once, using vectorized operations in NumPy. Each layer's output becomes the input to the next layer, forming a feedforward chain that passes information through the network.

$$Z=WX+b$$

$$A=f(Z)$$

The final layer produces the output — either a value (for regression) or class probabilities (for classification).

Back probagation

Demo

In backpropagation, the model calculates how far its predictions are from the correct answers using a loss function (like MSE or Cross-Entropy). Then, it computes the gradient of this loss to each weight and bias using the chain rule. This process tells the model which direction each weight and bias should move to reduce the overall error.

For the output layer, the gradient simplification is elegant:

MSE Regression: The output layer typically uses no activation function (linear output). The gradient of the loss with respect to the output is:

$$\frac{\partial L}{\partial \hat{y}} = \hat{y} - y$$

Cross-Entropy Classification: The output layer uses Softmax (multi-class) or Sigmoid (binary) activation. Remarkably, when combined with Cross-Entropy loss, the gradient simplifies to the same form:

$$\frac{\partial L}{\partial z} = \hat{y} - y$$

If you're interested in the details, this article by Shivam Mehta explains in depth how this simplification was derived. This mathematical convenience makes backpropagation efficient for both regression and classification tasks.

Gradient descent

Demo

Gradient descent is the optimization process that updates the weights and biases. It moves the weights and biases in the direction that reduces the loss the most, step by step.

$$ W := W - 𝜂 \frac{\partial J}{\partial W} $$

$$ b := b - 𝜂 \frac{\partial J}{\partial b} $$

The learning rate controls how big each step is — too high can make learning unstable, too low can make it too slow.

There are two main modes implemented in this project:

Stochastic Gradient Descent (SGD): updates weights for each training sample, when batch size is set to 1.

Mini-batch Gradient Descent: updates weights after a small batch of samples, balancing speed and stability.

Non linear classification

Demo

This diagram shows how an MLP can separate data that is not linearly separable — for instance, when data points form circular or spiral patterns. With multiple layers and non-linear activations, the network learns complex boundaries that a simple linear model cannot.

Programs

In addition to the provided Makefile, the following section explains the usage in detail and describes how you can customize the training process to fit your specific needs.

Configuration

An json config file to pass network structure and training configuration is required.

      {
          "shape": [784, 64, 32, 10],  
          // Network shape, number of neurons in each layer

          "activation_funcs": ["relu", "relu", "softmax"],  
          // Activation functions for each layer

          "weights_init": ["he", "he", "xavier"],  
          // Layer weights initialization methods, e.g. he / xavier / zero

          "loss": "CategoricalCrossEntropy",  
          // Loss function, supporting MeanSquareError and CrossEntropy

          "max_epoch": 1000,  
          // Maximum number of training epochs

          "learning_rate": 0.01,  
          // Learning rate for gradient descent

          "batch_size": 50,  
          // Batch size for mini-batch training, 1 for SGD

          "classification": true,  
          // Whether to calculate accuracy (true for classification tasks)

          "animation": "none",  
          // To show training animation, recommended only for 1D data

          "train_ratio": 0.8,  
          // Ratio to split dataset into training and validation

          "threshold": false,  
          // Threshold for early stopping, false means disabled

          "index": false  
          // Whether the input CSV file contains an index column
      }

Train and predict

We recommand you to use --help option before the first usage.

      Usage:

        uv run -m mlp <option> <config.json> <data.csv> [params.json]

      Options:

        -s:  Split dataset into (train.csv) and (test.csv), no params required

        -t:  Train dataset, if params.json is provided, it performs fine tuning.

        -p:  Test on a test dataset, if params.json is provided, it performs prediction based on trained model.

        --help:  Show help messages.

        -More features to come.

      Note:

        There is built in dataset generated by script, use --gen-data1d to replace dataset

The program will:

  • Defines the network structure (layers, activation functions).

  • Trains using backpropagation and gradient descent.

  • Visualize loss and prediction:

  • By default, regression task will visualize the loss, and the data distribution for first dimension.

  • In addition, classification task will add an accuracy plot.

Demo

The example graph on both classification and regression.

Demo

Datasets

Datasets are provided in this repository, one example corresponding to breast cancer diagnosis classification. It contains 32 columns, where the diagnosis column is the label to predict. The label can take two values:

M → Malignant (cancerous)

B → Benign (non-cancerous)

The remaining 30 numerical features describe various characteristics of a cell nucleus extracted with fine-needle aspiration (FNA), such as radius, texture, perimeter, area, and smoothness. These measurements are widely used for early breast cancer detection.

To split data:

It is split into two parts: one for training and validation one for testing.

uv run -m mlp -s config/data.json data/data.csv

The objective is to classify whether a tumor is malignant or benign based on the given features.

To train

# Train a new model
uv run -m mlp -t config/data.json data/train.csv

# Fine-tune an existing model
uv run -m mlp -t config/data.json data/train.csv params.json

After training, the params.json will be saved, which contain weights, biases and network structure. If params.json is provided, the program will perform fine tuning.

To test

uv run -m mlp -p config/data.json data/test.csv params.json

The loss function is selected explicitly by the loss field in the configuration. Use BinaryCrossEntropy with a sigmoid output for binary classification, CategoricalCrossEntropy with a softmax output for multi-class classification, or MeanSquareError with a linear output ("none") for regression. The output activation alone does not select the loss function.

The program will make prediction, an accuracy will be calculated and prediction will be saved in predictions.json.

Other datasets and configurations provided

  • 0-9.csv _____ 0-9.json => 0 to 9 hand wirte digits recognition with softmax and CCE loss
  • --gen-data1d _____ regre-relu.json => 1d random generated data with RELU activation and MSE loss
  • --gen-data1d _____ regre-sigmoid.json => 1d random generated data with Sigmoid activation and MSE loss

Class

Below is a simple example of using the MultiLayerPerceptron class to train a regression model.

from mlp.neural_network import MultiLayerPerceptron, relu
from mlp.data_process import generate_data_rand

def main():
    try:
        # Define network structure: input → 64 → 32 → output
        net_shape = (1, 64, 32, 1)
        activation_funcs = (relu, relu, None)  # last layer is linear (regression)
        weights_init = ("he", "he", "he")  # Define the weights initialization methods

        # Initialize neural network
        nn = MultiLayerPerceptron(net_shape, activation_funcs, weights_init)

        # Generate training data
        inputs, truths = generate_data_rand(142, 500, 0.02)
        test_inputs, test_truths = generate_data_rand(123, 50, 0.02)
        nn.train(inputs, truths, 10000, 0.005, batch_size=20, animation="plot")

        # Test with new data
        nn.test(test_inputs, test_truths)

        # Save graphs
        nn.save_plots()

    except Exception as e:
        print("Error:", e)


if __name__ == "__main__":
    main()

Generalization

The figure shows that multilayer perceptron is capable of learning non-linear patterns from different datasets and can generalize beyond the training data, making them applicable to a wide range of tasks.

Regression demo

Hyperparameter Tuning: Batch Size

Batch size is a key hyperparameter that influences both training speed and model performance. To study its effect, I kept all other parameters constant and varied only the batch size, trained the model on the MNIST dataset. In this experiment, I used CuPy to replace NumPy in order to leverage GPU acceleration and improve computational efficiency during training.

{
    "shape": [784, 392, 128, 64, 32, 10],
    "activation_funcs": ["relu", "relu", "relu", "relu", "softmax"],
    "weights_init": ["he", "he", "he", "he", "xavier"],
    "loss": "CategoricalCrossEntropy"
}
Batch Size Time (s) Accuracy Loss
1 2040 0.9814 0.1580
10 180 0.9769 0.1197
50 45 0.9719 0.1371
100 31 0.9676 0.1153
200 25 0.9593 0.1362
500 22 0.9414 0.1937
1000 21 0.9414 0.1937

Analysis

  • Training Time: As batch size increases, training time decreases dramatically, reflecting better computational efficiency.

  • Accuracy: Accuracy drops noticeably when batch size exceeds 200, indicating that very large batches may reduce generalization capability. The drop in accuracy for large batches can sometimes be reduced by slightly increasing the learning rate.

  • Loss: Smaller batches tend to achieve lower loss values, suggesting better fine-grained learning at the cost of longer training time.

Conclusion

In this experiment, batch sizes between 50 and 100 provide the best balance between training efficiency and model performance. Smaller batches has higher accuracy but are slower, while larger batches are faster but may sacrifice precision.


License

This project is licensed under the MIT License - see the LICENSE file for details.

Metadata

Release files for sw-mlp 1.0.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for sw-mlp 1.0.0
File Size Uploaded Provenance Actions
sw_mlp-1.0.0.tar.gz 16.7 kB Without provenance Details for sw_mlp-1.0.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for sw-mlp 1.0.0
File Interpreter ABI Platform Actions
sw_mlp-1.0.0-py3-none-any.whl Python 3 none any Details for sw_mlp-1.0.0-py3-none-any.whl

Total release size: 37.6 kB

Release files / sw_mlp-1.0.0.tar.gz

Download URL sw_mlp-1.0.0.tar.gz
Size 16.7 kB
Tags Source
SHA-256 checksum
How to use checksums
3cd84ba6cd93487306ed9a4e82e1f917e83e6b4e065f53723b94774197d1875a
BLAKE2b-256 checksum
How to use checksums
15f3d49ed54f174007018b1f4ea2be9b8286acfec9730ef12b5110efaa684686
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.11.29 {"installer":{"name":"uv","version":"0.11.29","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Release files / sw_mlp-1.0.0-py3-none-any.whl

Download URL sw_mlp-1.0.0-py3-none-any.whl
Size 20.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
28be24ee7b944940c052ff8984f78588d9f285a0e076bbd24a53a8bc116b1f2c
BLAKE2b-256 checksum
How to use checksums
fedda33d2de9cac93e9a9007aca133d0bee358d3c7b339f74f1acb4efc4a2625
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.11.29 {"installer":{"name":"uv","version":"0.11.29","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Security for sw-mlp 1.0.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page