NeuralNetwork

A small, dependency-free, from-scratch neural network library written in C++17, with Python bindings via pybind11.

It implements fully-connected (dense) feed-forward networks with backpropagation, three optimizers (SGD, Momentum, Adam), three loss functions, six activation functions, and accuracy metrics. Everything is implemented manually — no external math/ML library is used.

NeuralNetwork/
├── CMakeLists.txt            # builds the C++ executable + Python module
├── main.cpp                  # C++ example (quadratic regression)
├── setup.py                  # pip install . (Python bindings)
├── pyproject.toml            # build-system metadata for pip
├── include/                  # public headers
│   ├── Activation.h
│   ├── Layer.h
│   ├── Loss.h
│   ├── Metrics.h
│   ├── NeuralNetwork.h
│   ├── Neuron.h
│   ├── Optimizer.h
│   └── Random.h
├── src/                      # implementations
│   ├── Activation.cpp
│   ├── Layer.cpp
│   ├── Loss.cpp
│   ├── Metrics.cpp
│   ├── NeuralNetwork.cpp
│   ├── Neuron.cpp
│   ├── Optimizer.cpp
│   └── Random.cpp
└── python/                   # Python package
    ├── bindings.cpp          # pybind11 bindings
    ├── example.py            # Python example (XOR)
    └── neuralnetwork/        # importable package
        └── __init__.py

---

Table of contents

• Concepts
• Building
  • C++ (executable)
  • Python module
• C++ API reference
  • Random
  • Activation
  • Neuron
  • Layer
  • Loss
  • Metrics
  • Optimizer
  • NeuralNetwork
• Python API reference
• Examples
  • C++: XOR
  • C++: multi-class classification
  • Python: XOR
  • Python: regression
  • Python: multi-class classification
• How it works
• Tips & troubleshooting

---

Concepts

• Inputs / outputs are std::vector<double> (Python: list[float]). A dataset is std::vector<std::vector<double>> (Python: list[list[float]]).
• A network is a sequence of fully-connected layers. Each layer contains neurons, each with a weight vector and a bias.
• NeuralNetwork is constructed with the input size, the loss type, an optimizer (shared pointer in C++), and a learning rate. Layers are added with addLayer() and materialized with build().
• Forward pass: z = W·x + b, then output = activation(z) (softmax is applied across the whole layer).
• Backward pass: gradients are computed via backpropagation. The loss gradient is propagated layer by layer; the softmax layer uses the exact Jacobian.
• Training: trainSample() does a forward pass, a backward pass, then one optimizer update. trainBatch() averages gradients over the batch, then applies one update.
• Weights are initialized randomly (He init for ReLU/LeakyReLU, Xavier otherwise); biases start at 0.

---

Building

Install from PyPI

pip install neuralnetwork-cpp
python3 -c "import neuralnetwork; print(neuralnetwork.__version__)"

Prebuilt wheels are provided for Linux, macOS, and Windows on Python 3.9–3.14. If no wheel matches your platform, pip builds the module from source, which requires a C++17 compiler (pybind11 is fetched automatically).

Build from source

Requirements: a C++17 compiler, CMake ≥ 3.14, and (for the Python module) Python 3 with a working compiler toolchain.

C++ (executable)

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build
./build/neural_network          # runs main.cpp (quadratic regression demo)

Python module

Two ways:

1. Build with CMake (recommended if you do not have pip): the module is produced directly inside the package folder.

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build
# result: python/neuralnetwork/_core.cpython-<py>-<arch>.so

Then import from anywhere by putting the python/ directory on PYTHONPATH:

export PYTHONPATH="$PWD/python"
python3 -c "import neuralnetwork; print(neuralnetwork.__version__)"

2. pip install . (builds and installs the module, e.g. into a virtualenv):

pip install .
python3 -c "import neuralnetwork"

To skip the Python bindings during a CMake build:

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DBUILD_PYTHON_MODULE=OFF

---

C++ API reference

Everything lives in namespace nn. Include <NeuralNetwork.h> for the high-level API.

Random

Header: include/Random.h

Low-level random number helpers. All use one global, device-seeded std::mt19937.

Function	Description
static std::mt19937& Random::generator()	Reference to the global seeded RNG.
static double Random::uniform(double min, double max)	Uniform double in [min, max).
static double Random::gaussian(double mean, double stddev)	Gaussian sample with given mean and stddev.

double a = nn::Random::uniform(0.0, 1.0);     // uniform [0,1)
double b = nn::Random::gaussian(0.0, 1.0);    // standard normal

Activation

Header: include/Activation.h

enum class ActivationType { None, ReLU, LeakyReLU, Sigmoid, Tanh, Softmax };

None is the identity (linear) activation, used on the output layer for regression.

Function	Description
std::string activationName(ActivationType)	Human-readable name.
static double Activation::f(type, double x)	Applies the activation to a single scalar.
static double Activation::derivative(type, double z, double output)	Derivative w.r.t. the pre-activation z; output is the activated value (used for Sigmoid/Tanh shortcuts).
static std::vector<double> Activation::apply(type, const std::vector<double>& z)	Element-wise activation, except Softmax which normalizes the whole vector.

double s = nn::Activation::f(nn::ActivationType::Sigmoid, 0.0);        // 0.5
double d = nn::Activation::derivative(nn::ActivationType::Sigmoid, 0.0, 0.5); // 0.25
auto probs = nn::Activation::apply(nn::ActivationType::Softmax, {1.0, 2.0, 3.0});

Neuron

Header: include/Neuron.h

Represents one unit: a weight vector + bias. During the forward pass it stores its preactivation (z), output, and during backprop its delta and gradients.

Member	Type	Description
weights	std::vector<double>	Synaptic weights.
weightGradients	std::vector<double>	Accumulated dLoss/dWeight.
bias	double	Bias term.
biasGradient	double	Accumulated dLoss/dBias.
preactivation	double	z = W·x + b from the last forward pass.
output	double	Activated value from the last forward pass.
delta	double	dLoss/dz from the last backward pass.
activation	ActivationType	Activation used by this neuron.
velocityW / velocityB	vector / double	Momentum state.
mW, vW, mB, vB	vector / doubles	Adam moment estimates.
timestep	size_t	Adam bias-correction step counter.

Member function	Description
Neuron(size_t numInputs, ActivationType act)	Creates weights (He/Xavier initialized), bias = 0.
double forward(const std::vector<double>& inputs)	Computes z and stores preactivation / output; returns output.
double derivative() const	Activation::derivative(activation, preactivation, output).
void zeroGradients()	Resets weight/bias gradients to 0.
void applyGradients(Optimizer& opt, double lr)	Delegates one update step to the optimizer.

nn::Neuron n(3, nn::ActivationType::ReLU);     // 3 inputs
double out = n.forward({0.5, -1.0, 2.0});
n.weightGradients[0] = 0.01;                   // filled by backprop normally

Layer

Header: include/Layer.h

A dense layer owning a std::vector<Neuron>.

Member function	Description
Layer(size_t inputSize, size_t numNeurons, ActivationType act)	Builds the layer's neurons.
std::vector<double> forward(const std::vector<double>& input)	Computes outputs (softmax is applied over the whole layer) and stores inputs / outputs.
std::vector<double> backward(const std::vector<double>& outputGradients, bool accumulateGradients)	Computes each neuron's delta, accumulates weightGradients/biasGradient, and returns the gradients w.r.t. the layer input.
void zeroGradients()	Zeroes every neuron's gradients.

Public fields: neurons, inputs, outputs, activation.

nn::Layer hidden(2, 4, nn::ActivationType::Tanh);     // 2 inputs -> 4 tanh units
auto out = hidden.forward({0.3, -0.7});
auto inGrad = hidden.backward({0.1, 0.2, -0.1, 0.3}, false);

Loss

Header: include/Loss.h

enum class LossType { MSE, CrossEntropy, BinaryCrossEntropy };

Function	Description
std::string lossName(LossType)	Human-readable name.
static double Loss::compute(type, const std::vector<double>& prediction, const std::vector<double>& target)	Scalar loss value.
static std::vector<double> Loss::gradient(type, prediction, target)	dLoss/dprediction per output element.

Formulas (inputs have n elements, p = prediction, t = target):

• MSE: (1/n) · Σ(pᵢ − tᵢ)²
• CrossEntropy: −Σ tᵢ·log(pᵢ)
• BinaryCrossEntropy: −(1/n) · Σ [tᵢ·log(pᵢ) + (1−tᵢ)·log(1−pᵢ)]

double l = nn::Loss::compute(nn::LossType::MSE, {0.8, 0.2}, {1.0, 0.0});
auto g  = nn::Loss::gradient(nn::LossType::MSE, {0.8, 0.2}, {1.0, 0.0});

Metrics

Header: include/Metrics.h

Function	Description
static size_t Metrics::predictedClass(const std::vector<double>& output)	Index of the largest output (argmax).
static double Metrics::accuracy(prediction, target)	1.0 if correct else 0.0; single-output vectors are thresholded at 0.5, multi-output use argmax.
static double Metrics::accuracy(predictions, targets)	Mean accuracy over a dataset.

double acc = nn::Metrics::accuracy({{0.9, 0.1}, {0.2, 0.8}}, {{1, 0}, {0, 1}}); // 1.0

Optimizer

Header: include/Optimizer.h

Abstract base Optimizer with virtual std::string name() const and virtual void update(Neuron&, double learningRate). Neural networks hold the optimizer as a std::shared_ptr<Optimizer>.

Class	Constructor	Update rule
SGD	SGD()	w ← w − lr·grad
MomentumSGD	MomentumSGD(double momentum = 0.9)	v ← μ·v + lr·grad; w ← w − v
Adam	Adam(double beta1 = 0.9, double beta2 = 0.999, double epsilon = 1e-8)	Adaptive moment estimation with bias correction

auto opt = std::make_shared<nn::Adam>(0.9, 0.999, 1e-8);
std::cout << opt->name() << "\n";   // "Adam"

NeuralNetwork

Header: include/NeuralNetwork.h

struct EvaluationResult {
    double loss;
    double accuracy;
};

Member	Description
NeuralNetwork(size_t inputSize, LossType lossType, std::shared_ptr<Optimizer> optimizer, double learningRate)	Constructs an empty network.
void addLayer(size_t units, ActivationType activation)	Registers a hidden/output layer configuration.
void build()	Materializes all layers (must be called before any forward/train).
std::vector<double> forward(const std::vector<double>& input)	Runs a forward pass, returns the output vector.
std::vector<double> predict(const std::vector<double>& input)	Same as forward (convenience alias).
double backprop(const std::vector<double>& target)	Runs backward pass for the most recent forward pass; returns the loss. Does not update weights.
double trainSample(const std::vector<double>& input, const std::vector<double>& target)	Forward + backward + one optimizer update; returns loss.
double trainBatch(const std::vector<std::vector<double>>& inputs, const std::vector<std::vector<double>>& targets)	Accumulates gradients over the batch (averaged), applies one optimizer update; returns mean loss.
EvaluationResult evaluate(const std::vector<std::vector<double>>& inputs, const std::vector<std::vector<double>>& targets)	Mean loss + classification accuracy over a dataset (no weight updates).
size_t inputSize() const	Configured input dimension.
size_t layerCount() const	Number of materialized layers.
const std::vector<Layer>& layers() const	Read access to the materialized layers.

Usage pattern:

#include "NeuralNetwork.h"

using namespace nn;

NeuralNetwork net(2, LossType::MSE, std::make_shared<Adam>(), 0.1);
net.addLayer(4, ActivationType::Tanh);       // hidden
net.addLayer(1, ActivationType::Sigmoid);    // output
net.build();

// single sample training
for (int e = 0; e < 1000; ++e)
    net.trainSample({0, 1}, {1});

// batch training
std::vector<std::vector<double>> X = {{0,0},{0,1},{1,0},{1,1}};
std::vector<std::vector<double>> Y = {{0},{1},{1},{0}};
double loss = net.trainBatch(X, Y);          // mean loss over the batch

// evaluation
EvaluationResult res = net.evaluate(X, Y);
std::cout << "loss=" << res.loss << " acc=" << res.accuracy << "\n";

// inference
auto out = net.predict({1, 0});

Note: backprop() uses the inputs/outputs cached by the most recent forward() call — call forward() (or trainSample) immediately before it.

---

Python API reference

Import the package:

import neuralnetwork as nn

The names map 1:1 to the C++ API (snake_case for methods, and ActivationType.None is renamed Linear because None is a Python keyword).

Enums

nn.ActivationType.Linear        # identity (was C++ None)
nn.ActivationType.ReLU
nn.ActivationType.LeakyReLU
nn.ActivationType.Sigmoid
nn.ActivationType.Tanh
nn.ActivationType.Softmax

nn.LossType.MSE
nn.LossType.CrossEntropy
nn.LossType.BinaryCrossEntropy

Optimizers

nn.SGD()                                # plain SGD
nn.MomentumSGD(momentum=0.9)            # momentum SGD
nn.Adam(beta1=0.9, beta2=0.999, epsilon=1e-8)  # Adam

All expose .name() returning e.g. "Adam".

NeuralNetwork

Python method	C++ equivalent	Description
nn.NeuralNetwork(input_size, loss_type, optimizer, learning_rate)	constructor	Create network.
net.add_layer(units, activation)	addLayer	Register a layer.
net.build()	build	Materialize layers.
net.forward(input)	forward	Forward pass, returns list[float].
net.predict(input)	predict	Inference alias.
net.backprop(target)	backprop	Backward pass for last forward; returns loss.
net.train_sample(input, target)	trainSample	One sample update; returns loss.
net.train_batch(inputs, targets)	trainBatch	Batch update; returns mean loss.
net.evaluate(inputs, targets)	evaluate	Returns EvaluationResult.
net.input_size()	inputSize	Input dimension.
net.layer_count()	layerCount	Number of layers.

EvaluationResult has read-only fields .loss and .accuracy and a repr.

Metrics

nn.Metrics.predicted_class(output)                        # argmax index
nn.Metrics.accuracy(prediction, target)                   # 0.0 or 1.0
nn.Metrics.accuracy(predictions, targets)                 # mean accuracy

Minimal Python usage:

import neuralnetwork as nn

net = nn.NeuralNetwork(2, nn.LossType.MSE, nn.Adam(), 0.1)
net.add_layer(4, nn.ActivationType.Tanh)
net.add_layer(1, nn.ActivationType.Sigmoid)
net.build()

for _ in range(1000):
    net.train_sample([0, 1], [1])

pred = net.predict([0, 1])   # [0.998...]

---

Examples

C++: XOR

#include <iostream>
#include "NeuralNetwork.h"

int main() {
    using namespace nn;

    NeuralNetwork net(2, LossType::MSE, std::make_shared<Adam>(), 0.1);
    net.addLayer(4, ActivationType::Tanh);
    net.addLayer(1, ActivationType::Sigmoid);
    net.build();

    const std::vector<std::vector<double>> X = {{0,0},{0,1},{1,0},{1,1}};
    const std::vector<std::vector<double>> Y = {{0},{1},{1},{0}};

    for (int e = 1; e <= 2000; ++e) {
        double loss = 0.0;
        for (size_t i = 0; i < X.size(); ++i) loss += net.trainSample(X[i], Y[i]);
        if (e % 500 == 0)
            std::cout << "epoch " << e << " loss=" << loss / 4 << "\n";
    }

    for (size_t i = 0; i < X.size(); ++i)
        std::cout << X[i][0] << " XOR " << X[i][1] << " = "
                  << (net.predict(X[i])[0] > 0.5 ? 1 : 0) << "\n";
    return 0;
}

C++: multi-class classification

Softmax output + CrossEntropy loss on a 3-class problem.

#include <iostream>
#include "NeuralNetwork.h"

int main() {
    using namespace nn;

    NeuralNetwork net(2, LossType::CrossEntropy, std::make_shared<Adam>(), 0.01);
    net.addLayer(16, ActivationType::ReLU);
    net.addLayer(3, ActivationType::Softmax);
    net.build();

    // Points near (1,1) -> class 0; near (-1,1) -> class 1; near (-1,-1) -> class 2
    const std::vector<std::vector<double>> X = {
        {1.1, 0.9}, {0.9, 1.0}, {1.0, 1.1},    // class 0
        {-1.1, 1.0}, {-0.9, 0.9}, {-1.0, 1.1}, // class 1
        {-1.0, -1.1}, {-1.1, -0.9}, {-0.9, -1.0}}; // class 2
    const std::vector<std::vector<double>> Y = {
        {1,0,0}, {1,0,0}, {1,0,0},
        {0,1,0}, {0,1,0}, {0,1,0},
        {0,0,1}, {0,0,1}, {0,0,1}};

    for (int e = 1; e <= 500; ++e) {
        double loss = 0.0;
        for (size_t i = 0; i < X.size(); ++i) loss += net.trainSample(X[i], Y[i]);
        if (e % 100 == 0)
            std::cout << "epoch " << e << " loss=" << loss / X.size() << "\n";
    }

    auto res = net.evaluate(X, Y);
    std::cout << "test accuracy = " << res.accuracy << "\n";
    return 0;
}

Python: XOR

Run python3 python/example.py, or:

import neuralnetwork as nn

net = nn.NeuralNetwork(2, nn.LossType.MSE, nn.Adam(), 0.1)
net.add_layer(4, nn.ActivationType.Tanh)
net.add_layer(1, nn.ActivationType.Sigmoid)
net.build()

X = [[0, 0], [0, 1], [1, 0], [1, 1]]
Y = [[0], [1], [1], [0]]

for epoch in range(1, 2001):
    loss = sum(net.train_sample(X[i], Y[i]) for i in range(4)) / 4
    if epoch % 500 == 0:
        res = net.evaluate(X, Y)
        print(f"epoch {epoch:4d}  loss = {loss:.6f}  accuracy = {res.accuracy:.0%}")

print([round(net.predict(x)[0], 4) for x in X])
# e.g. [0.0008, 0.999, 0.999, 0.0009]

Python: regression

Normalize inputs to keep activations unsaturated (the hidden units are ReLU here).

import random
import neuralnetwork as nn

random.seed(1)
X = [[random.uniform(-1, 1)] for _ in range(2000)]
Y = [[x[0] * x[0] + 3 * x[0] - 2] for x in X]     # targets in [-4, 2]

net = nn.NeuralNetwork(1, nn.LossType.MSE, nn.Adam(), 0.01)
net.add_layer(32, nn.ActivationType.ReLU)
net.add_layer(1, nn.ActivationType.Linear)
net.build()

for epoch in range(1, 51):
    loss = 0.0
    for start in range(0, 2000, 128):
        loss += net.train_batch(X[start:start + 128], Y[start:start + 128])
    if epoch % 10 == 0:
        print(f"epoch {epoch:3d}  loss = {loss / 16:.5f}")

print(round(net.predict([0.5])[0], 4))   # ~ -0.25 (true: 0.25 + 1.5 - 2)

Python: multi-class classification

import neuralnetwork as nn

net = nn.NeuralNetwork(2, nn.LossType.CrossEntropy, nn.Adam(), 0.01)
net.add_layer(16, nn.ActivationType.ReLU)
net.add_layer(3, nn.ActivationType.Softmax)
net.build()

X = [[1.1, 0.9], [0.9, 1.0], [-1.1, 1.0], [-0.9, 0.9], [-1.0, -1.1], [-1.1, -0.9]]
Y = [[1, 0, 0], [1, 0, 0], [0, 1, 0], [0, 1, 0], [0, 0, 1], [0, 0, 1]]

for _ in range(300):
    for i in range(len(X)):
        net.train_sample(X[i], Y[i])

res = net.evaluate(X, Y)
print(f"accuracy: {res.accuracy:.0%}")            # 100%
print(net.predict([1.05, 1.05]))                  # e.g. [0.97, 0.02, 0.01]

---

How it works

Forward pass per neuron:

z  = b + Σᵢ wᵢ·xᵢ
a  = activation(z)        (softmax is computed across the layer)

Backward pass — the chain rule in reverse:

1. δ_output = dLoss/doutput from Loss::gradient.
2. Each layer converts that into per-neuron delta = dLoss/dz:
   • element-wise activations: delta = dLoss/da · activation′(z)
   • softmax: delta = aᵢ · (dLoss/daᵢ − Σⱼ dLoss/daⱼ·aⱼ) (exact Jacobian)
3. Gradients are accumulated per neuron: ∂Loss/∂wᵢ = delta·xᵢ, ∂Loss/∂b = delta.
4. The input gradient ∂Loss/∂xᵢ = Σ delta·wᵢ is passed to the previous layer.

Weight initialization — He for ReLU/LeakyReLU (σ = √(2/fan_in)), Xavier otherwise (σ = √(1/fan_in)); biases are 0.

Optimizer updates — see the Optimizer table. Adam keeps per-parameter first/second moments with bias correction.

---

Tips & troubleshooting

• Call build() after addLayer() and before any forward/train/evaluate call.
• Input sizes must match input_size; mismatches throw std::runtime_error (C++) / RuntimeError (Python).
• Regression: use MSE + ActivationType.Linear on the output layer, and normalize inputs (and often targets) so activations do not saturate.
• Multi-class: use CrossEntropy + Softmax output; targets are one-hot vectors.
• Binary classification: BinaryCrossEntropy + Sigmoid output.
• Slow convergence or saturation: reduce the learning rate, normalize inputs to roughly [-1, 1], or prefer ReLU hidden layers.
• Python keyword clash: the identity activation is nn.ActivationType.Linear in Python, but ActivationType::None in C++.
• Re-import after rebuild: if you rebuild the module, restart the Python process — the extension is loaded once.