Multilayer Perceptron
Multilayer perceptron (MLP) is a simple neural network (NN), can solve both regression and classification tasks. The project provides a reusable and modular implementation.
Requirements
The project is built from scratch with Python, with only Numpy, Pandas, Matplotlib, without high-level ML libraries.
Use Python 3.12 and uv to manage the environment and dependencies. Install uv before continuing. Run the following commands from the project root:
git clone https://github.com/swangarch/multilayer_perceptron
cd multilayer_perceptron
# Create .venv and install the locked dependencies and the project
uv sync --locked
# Alternatively, use the Makefile (runs uv sync)
make setup
Commands below use uv run, so activating the virtual environment is not required.
To activate it manually on macOS/Linux, use source .venv/bin/activate.
Usage
Run these commands from the project root. Makefile targets use uv run -m mlp.
uv run -m mlp --help
For example dataset classification
# To split the dataset
make split
# To train with training configuration file and training dataset
make train
# To test with test dataset
make test
Breast cancer diagnosis dataset, a binary classification based on 30 features of a cell.
MNIST handwrite digits classification (0–9)
# To split, train and test the dataset
make digit
For the handwritten digit classification task (0–9), the model achieved 97.19% test accuracy using 33,000 training samples of 28 × 28 images in the optimal configuration.
For the handwritten character recognition task with 94 categories, the model reached 91% accuracy based on 175,000 of 28 × 28 images training samples.
For random generated 1d data regression
# To train and test generated data, with relu activation
make regre-relu
# To train and test generated data, with sigmoid activation
make regre-sigmoid
Packaging
Build the wheel and source distribution with uv:
uv build
# Alternatively
make build
The output files are written to dist/:
sw_mlp-1.0.0-py3-none-any.whlsw_mlp-1.0.0.tar.gz
To install the wheel into a Python 3.12 virtual environment:
uv venv --python 3.12
uv pip install dist/sw_mlp-1.0.0-py3-none-any.whl
# macOS/Linux: run the installed package
.venv/bin/python -m mlp --help
The wheel contains the Python package. Example configurations and datasets in the
repository's config/ and data/ directories must be distributed separately.
Pass their paths when running the installed package. The split command writes to
data/train.csv and data/test.csv, so the working directory must contain a
data/ directory.
Features
-
Training methods → supports Stochastic Gradient Descent (SGD) and mini-batch training.
-
Visualization → includes real-time animations to track the learning process.
-
Configurable architecture → adjustable network shape, activation functions, initialization methods, max iterations, learning rate, batch size, etc via a configuration file.
-
Data handling → data loading and preprocessing.
-
Support multiple activation functions: ReLU, Sigmoid, Softmax, Leaky ReLU
-
Support multiple loss functions:
- Categorical Cross Entropy (CCE) with automatic one-hot encoding for categorical labels
- Binary Cross Entropy (BCE)
- Mean Squared Error (MSE)
-
Support Fine-tuning: continue training from saved weights for further optimization
Methodology
A Multilayer Perceptron is a basic artificial neural network that learns to map input data to the correct output by adjusting internal parameters called weights and biases. It is composed of multiple layers of artificial neurons, where each neuron performs a simple computation — combining inputs, multiplying weights, adding a bias, and passing the result through an activation function to introduce non-linearity. Then using back propagation and gradient descent to find ideal weights and biases.
Learning Process
The learning process consists of three main stages:
- Feedforward — compute predictions by passing data through all layers.
- Back propagation — calculate gradients of the loss with respect to the weights, using chain rule.
- Gradient Descent — update weights and biases to minimize the loss over time.
Through repeated iterations of these steps, the MLP gradually improves its predictions and learns to generalize from the training data.
Feedfoward
In the feedforward step, the input data passes through several layers of neurons. Each neuron performs a weighted sum of its inputs, adds a bias, and then applies an activation function such as ReLU, Sigmoid, or Softmax.
When neurons are grouped into a layer, their computations can be efficiently expressed using matrix multiplication. This allows the model to process many neurons at once, using vectorized operations in NumPy. Each layer's output becomes the input to the next layer, forming a feedforward chain that passes information through the network.
$$Z=WX+b$$
$$A=f(Z)$$
The final layer produces the output — either a value (for regression) or class probabilities (for classification).
Back probagation
In backpropagation, the model calculates how far its predictions are from the correct answers using a loss function (like MSE or Cross-Entropy). Then, it computes the gradient of this loss to each weight and bias using the chain rule. This process tells the model which direction each weight and bias should move to reduce the overall error.
For the output layer, the gradient simplification is elegant:
MSE Regression: The output layer typically uses no activation function (linear output). The gradient of the loss with respect to the output is:
$$\frac{\partial L}{\partial \hat{y}} = \hat{y} - y$$
Cross-Entropy Classification: The output layer uses Softmax (multi-class) or Sigmoid (binary) activation. Remarkably, when combined with Cross-Entropy loss, the gradient simplifies to the same form:
$$\frac{\partial L}{\partial z} = \hat{y} - y$$
If you're interested in the details, this article by Shivam Mehta explains in depth how this simplification was derived. This mathematical convenience makes backpropagation efficient for both regression and classification tasks.
Gradient descent
Gradient descent is the optimization process that updates the weights and biases. It moves the weights and biases in the direction that reduces the loss the most, step by step.
$$ W := W - 𝜂 \frac{\partial J}{\partial W} $$
$$ b := b - 𝜂 \frac{\partial J}{\partial b} $$
The learning rate controls how big each step is — too high can make learning unstable, too low can make it too slow.
There are two main modes implemented in this project:
Stochastic Gradient Descent (SGD): updates weights for each training sample, when batch size is set to 1.
Mini-batch Gradient Descent: updates weights after a small batch of samples, balancing speed and stability.
Non linear classification
This diagram shows how an MLP can separate data that is not linearly separable — for instance, when data points form circular or spiral patterns. With multiple layers and non-linear activations, the network learns complex boundaries that a simple linear model cannot.
Programs
In addition to the provided Makefile, the following section explains the usage in detail and describes how you can customize the training process to fit your specific needs.
Configuration
An json config file to pass network structure and training configuration is required.
{
"shape": [784, 64, 32, 10],
// Network shape, number of neurons in each layer
"activation_funcs": ["relu", "relu", "softmax"],
// Activation functions for each layer
"weights_init": ["he", "he", "xavier"],
// Layer weights initialization methods, e.g. he / xavier / zero
"loss": "CategoricalCrossEntropy",
// Loss function, supporting MeanSquareError and CrossEntropy
"max_epoch": 1000,
// Maximum number of training epochs
"learning_rate": 0.01,
// Learning rate for gradient descent
"batch_size": 50,
// Batch size for mini-batch training, 1 for SGD
"classification": true,
// Whether to calculate accuracy (true for classification tasks)
"animation": "none",
// To show training animation, recommended only for 1D data
"train_ratio": 0.8,
// Ratio to split dataset into training and validation
"threshold": false,
// Threshold for early stopping, false means disabled
"index": false
// Whether the input CSV file contains an index column
}
Train and predict
We recommand you to use --help option before the first usage.
Usage:
uv run -m mlp <option> <config.json> <data.csv> [params.json]
Options:
-s: Split dataset into (train.csv) and (test.csv), no params required
-t: Train dataset, if params.json is provided, it performs fine tuning.
-p: Test on a test dataset, if params.json is provided, it performs prediction based on trained model.
--help: Show help messages.
-More features to come.
Note:
There is built in dataset generated by script, use --gen-data1d to replace dataset
The program will:
-
Defines the network structure (layers, activation functions).
-
Trains using backpropagation and gradient descent.
-
Visualize loss and prediction:
-
By default, regression task will visualize the loss, and the data distribution for first dimension.
-
In addition, classification task will add an accuracy plot.
Demo
The example graph on both classification and regression.
Datasets
Datasets are provided in this repository, one example corresponding to breast cancer diagnosis classification. It contains 32 columns, where the diagnosis column is the label to predict. The label can take two values:
M → Malignant (cancerous)
B → Benign (non-cancerous)
The remaining 30 numerical features describe various characteristics of a cell nucleus extracted with fine-needle aspiration (FNA), such as radius, texture, perimeter, area, and smoothness. These measurements are widely used for early breast cancer detection.
To split data:
It is split into two parts: one for training and validation one for testing.
uv run -m mlp -s config/data.json data/data.csv
The objective is to classify whether a tumor is malignant or benign based on the given features.
To train
# Train a new model
uv run -m mlp -t config/data.json data/train.csv
# Fine-tune an existing model
uv run -m mlp -t config/data.json data/train.csv params.json
After training, the params.json will be saved, which contain weights, biases and network structure. If params.json is provided, the program will perform fine tuning.
To test
uv run -m mlp -p config/data.json data/test.csv params.json
The loss function is selected explicitly by the loss field in the configuration.
Use BinaryCrossEntropy with a sigmoid output for binary classification,
CategoricalCrossEntropy with a softmax output for multi-class classification,
or MeanSquareError with a linear output ("none") for regression.
The output activation alone does not select the loss function.
The program will make prediction, an accuracy will be calculated and prediction will be saved in predictions.json.
Other datasets and configurations provided
- 0-9.csv _____ 0-9.json => 0 to 9 hand wirte digits recognition with softmax and CCE loss
- --gen-data1d _____ regre-relu.json => 1d random generated data with RELU activation and MSE loss
- --gen-data1d _____ regre-sigmoid.json => 1d random generated data with Sigmoid activation and MSE loss
Class
Below is a simple example of using the MultiLayerPerceptron class to train a regression model.
from mlp.neural_network import MultiLayerPerceptron, relu
from mlp.data_process import generate_data_rand
def main():
try:
# Define network structure: input → 64 → 32 → output
net_shape = (1, 64, 32, 1)
activation_funcs = (relu, relu, None) # last layer is linear (regression)
weights_init = ("he", "he", "he") # Define the weights initialization methods
# Initialize neural network
nn = MultiLayerPerceptron(net_shape, activation_funcs, weights_init)
# Generate training data
inputs, truths = generate_data_rand(142, 500, 0.02)
test_inputs, test_truths = generate_data_rand(123, 50, 0.02)
nn.train(inputs, truths, 10000, 0.005, batch_size=20, animation="plot")
# Test with new data
nn.test(test_inputs, test_truths)
# Save graphs
nn.save_plots()
except Exception as e:
print("Error:", e)
if __name__ == "__main__":
main()
Generalization
The figure shows that multilayer perceptron is capable of learning non-linear patterns from different datasets and can generalize beyond the training data, making them applicable to a wide range of tasks.
Hyperparameter Tuning: Batch Size
Batch size is a key hyperparameter that influences both training speed and model performance. To study its effect, I kept all other parameters constant and varied only the batch size, trained the model on the MNIST dataset. In this experiment, I used CuPy to replace NumPy in order to leverage GPU acceleration and improve computational efficiency during training.
{
"shape": [784, 392, 128, 64, 32, 10],
"activation_funcs": ["relu", "relu", "relu", "relu", "softmax"],
"weights_init": ["he", "he", "he", "he", "xavier"],
"loss": "CategoricalCrossEntropy"
}
| Batch Size | Time (s) | Accuracy | Loss |
|---|---|---|---|
| 1 | 2040 | 0.9814 | 0.1580 |
| 10 | 180 | 0.9769 | 0.1197 |
| 50 | 45 | 0.9719 | 0.1371 |
| 100 | 31 | 0.9676 | 0.1153 |
| 200 | 25 | 0.9593 | 0.1362 |
| 500 | 22 | 0.9414 | 0.1937 |
| 1000 | 21 | 0.9414 | 0.1937 |
Analysis
-
Training Time: As batch size increases, training time decreases dramatically, reflecting better computational efficiency.
-
Accuracy: Accuracy drops noticeably when batch size exceeds 200, indicating that very large batches may reduce generalization capability. The drop in accuracy for large batches can sometimes be reduced by slightly increasing the learning rate.
-
Loss: Smaller batches tend to achieve lower loss values, suggesting better fine-grained learning at the cost of longer training time.
Conclusion
In this experiment, batch sizes between 50 and 100 provide the best balance between training efficiency and model performance. Smaller batches has higher accuracy but are slower, while larger batches are faster but may sacrifice precision.
License
This project is licensed under the MIT License - see the LICENSE file for details.
Metadata
Release files for sw-mlp 1.0.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| sw_mlp-1.0.0.tar.gz | 16.7 kB | Without provenance | Details for sw_mlp-1.0.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| sw_mlp-1.0.0-py3-none-any.whl | Python 3 | none | any | Details for sw_mlp-1.0.0-py3-none-any.whl |
Total release size: 37.6 kB
Release files / sw_mlp-1.0.0.tar.gz
| Download URL | sw_mlp-1.0.0.tar.gz |
|---|---|
| Size | 16.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
3cd84ba6cd93487306ed9a4e82e1f917e83e6b4e065f53723b94774197d1875a
|
|
BLAKE2b-256 checksum How to use checksums |
15f3d49ed54f174007018b1f4ea2be9b8286acfec9730ef12b5110efaa684686
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.29 {"installer":{"name":"uv","version":"0.11.29","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / sw_mlp-1.0.0-py3-none-any.whl
| Download URL | sw_mlp-1.0.0-py3-none-any.whl |
|---|---|
| Size | 20.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
28be24ee7b944940c052ff8984f78588d9f285a0e076bbd24a53a8bc116b1f2c
|
|
BLAKE2b-256 checksum How to use checksums |
fedda33d2de9cac93e9a9007aca133d0bee358d3c7b339f74f1acb4efc4a2625
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.29 {"installer":{"name":"uv","version":"0.11.29","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for sw-mlp 1.0.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.