NeuroCran — Mathematical Foundations

This section explains the mathematical operations behind NeuroCran, from individual neurons to neural network training.

The formulas describe standard neural network mathematics. Their implementation status depends on the features currently supported by NeuroCran.

Table of Contents

1. Mathematical Notation
2. Neuron Computation
3. Activation Functions
4. Activation Functions Assigned to Individual Neurons
5. Layer Computation
6. Forward Propagation
7. Weight Initialization
8. Loss Functions
9. Backpropagation
10. Gradient Descent and Optimization
11. Regularization
12. Numerical Stability

---

1. Mathematical Notation

The following symbols are used throughout NeuroCran's mathematical documentation.

Symbol	Meaning
x	An input value or input vector
w	A weight connecting neurons
b	A bias value
z	A neuron's value before activation
a	A neuron's output after activation
f(x)	An activation function
W	A weight matrix
b	A bias vector
L	A loss value or loss function, depending on context
η (eta)	Learning rate
∇ (nabla)	Gradient operator
n	Number of neurons or samples, depending on context
m	Number of input features in the neuron equations below

The meaning of a symbol can depend on its context. For example, lowercase a can represent a neuron's activation output, while a list of activation functions is written as A.

2. Neuron Computation

A neuron performs two main operations:

1. It calculates a weighted sum of its inputs and adds a bias.
2. It applies an activation function to the result.

Step 1: Calculate the weighted sum

Mathematical formula

z = w₁x₁ + w₂x₂ + ⋯ + wₘxₘ + b

The same equation can be written using summation notation:

z = Σᵢ₌₁ᵐ wᵢxᵢ + b

Meaning

• xᵢ is input feature i.
• wᵢ is the weight associated with input feature i.
• b is the bias.
• z is the value calculated before activation.

Step 2: Apply the activation function

a = f(z)

Here, f is the selected activation function and a is the final output of the neuron.

For a ReLU neuron:

a = max(0, z)

Python example

inputs = [2.0, 3.0]
weights = [0.5, 0.2]
bias = 0.1

z = sum(x * w for x, w in zip(inputs, weights)) + bias

# ReLU activation
output = max(0.0, z)

print("Weighted sum:", z)
print("Neuron output:", output)

The weighted sum is 1.7, and the ReLU output is also 1.7.

---

3. Activation Functions

An activation function transforms a neuron's weighted sum into its output.

3.1 ReLU — Rectified Linear Unit

Formula

ReLU(x) = max(0, x)

Behavior

• If x is positive, the output equals x.
• If x is negative, the output is zero.
• If x is zero, the output is zero.

Example: ReLU(-3) = 0 and ReLU(4) = 4.

3.2 Sigmoid

Formula

σ(x) = 1 / (1 + e⁻ˣ)

Behavior

The output lies between 0 and 1 for finite real inputs.

Sigmoid is often used in binary classification output layers.

3.3 Tanh

Formula

tanh(x) = (eˣ − e⁻ˣ) / (eˣ + e⁻ˣ)

Behavior

The output lies between -1 and 1.

3.4 Softplus

Formula

Softplus(x) = ln(1 + eˣ)

Behavior

Softplus is a smooth alternative to ReLU and produces positive outputs.

3.5 Softmax

Softmax converts a vector of scores into normalized values.

For class i:

Softmax(zᵢ) = eᶻⁱ / Σⱼ eᶻʲ

The resulting values sum to 1 and can represent class probabilities in a classification model.

Softmax acts on a vector of scores rather than independently transforming each score.

3.6 Logistic

The logistic activation function is another name for the standard sigmoid function:

Logistic(x) = 1 / (1 + e⁻ˣ)

Registering both names can be useful for API compatibility, but they represent the same mathematical function under these definitions.

---

4. Activation Functions Assigned to Individual Neurons

This is the central design idea behind NeuroCran.

A layer can be configured so that each neuron uses its own activation function.

For a layer containing four neurons, define the activation configuration as:

Activation configuration

A = [ReLU, Sigmoid, Tanh, Softplus]

This means:

• Neuron 1 uses ReLU.
• Neuron 2 uses Sigmoid.
• Neuron 3 uses Tanh.
• Neuron 4 uses Softplus.

Let zᵢ represent the weighted sum for neuron i, and let fᵢ represent its assigned activation function.

The output of neuron i is:

aᵢ = fᵢ(zᵢ)

For a layer containing n neurons, the output vector is:

a = [f₁(z₁), f₂(z₂), ..., fₙ(zₙ)]

Each neuron can therefore apply a different function to its own weighted sum.

Python representation

activations = [
    "relu",
    "sigmoid",
    "tanh",
    "softplus",
]

The list represents the desired configuration. The actual NeuroCran implementation must validate that the list length matches the number of neurons and that each activation name is supported.

Why this matters

A heterogeneous activation layer allows developers to experiment with different activation functions within the same layer.

It does not automatically guarantee better accuracy or faster training. The effect should be measured experimentally.

---

5. Layer Computation

A fully connected layer calculates weighted sums for all neurons.

Matrix formula

Z = WX + b

Then the layer applies its activation functions:

A = F(Z)

Where:

• X represents the input vector or input batch.
• W is the weight matrix.
• b is the bias vector.
• Z contains the weighted sums.
• F applies the configured activation functions.
• A contains the layer outputs.

For a layer with different activation functions per neuron, F applies the corresponding function to each neuron's weighted sum.

For a batch of input samples, the bias vector is broadcast across the batch dimension.

Dimensions

For a single sample:

• Input vector X has shape (m,).
• Weight matrix W has shape (n, m).
• Bias vector b has shape (n,).
• Output vector A has shape (n,).

Here, m is the number of input features and n is the number of neurons.

Matching these dimensions is essential for correct tensor computation.

---

6. Forward Propagation

Forward propagation calculates predictions by passing data through the network.

For layer l:

Z⁽ˡ⁾ = W⁽ˡ⁾A⁽ˡ⁻¹⁾ + b⁽ˡ⁾

A⁽ˡ⁾ = F⁽ˡ⁾(Z⁽ˡ⁾)

The output of one layer becomes the input to the next layer.

For the first layer, the input data is used instead of a previous layer's output.

The final layer produces the network prediction, often denoted by ŷ (y-hat).

Example architecture

Suppose a network contains:

• 3 input features.
• 4 neurons in the first hidden layer.
• 2 output neurons.

The first layer's weight matrix has shape (4, 3), and its bias vector has shape (4,).

The output layer's weight matrix has shape (2, 4), and its bias vector has shape (2,).

The resulting prediction contains two output values per input sample.

---

7. Weight Initialization

Weight initialization determines the starting values of trainable parameters.

Good initialization can help maintain reasonable activation and gradient scales during training.

7.1 Zero initialization

wᵢⱼ = 0

Zero initialization can cause neurons in the same layer to learn identical features when they have identical initialization and symmetric updates. It is generally unsuitable for initializing all weights of a conventional hidden layer.

7.2 Xavier / Glorot initialization

A common uniform Xavier initialization is:

W ~ Uniform(-√(6 / (fan_in + fan_out)), +√(6 / (fan_in + fan_out)))

Where:

• fan_in is the number of inputs to the layer.
• fan_out is the number of outputs from the layer.

Xavier initialization is commonly used with approximately symmetric activations such as tanh.

7.3 He / Kaiming initialization

A common normal He initialization is:

W ~ Normal(0, √(2 / fan_in))

Here, the second parameter is the standard deviation.

He initialization is commonly used with ReLU-like activations.

7.4 LeCun initialization

A common normal LeCun initialization is:

W ~ Normal(0, 1 / √fan_in)

Here, the second parameter is the standard deviation.

7.5 Orthogonal initialization

Orthogonal initialization constructs a weight matrix whose rows or columns are orthogonal, as permitted by its dimensions.

For a square orthogonal matrix:

WᵀW = I

Here, Wᵀ is the transpose of W and I is the identity matrix.

Important note

The initialization strategy should match the layer dimensions and activation behavior. In a layer where different neurons use different activation functions, a single initialization strategy may not be equally suitable for every neuron.

---

8. Loss Functions

A loss function measures the difference between a model's predictions and the target values.

8.1 Mean Squared Error (MSE)

MSE = (1 / N) Σᵢ₌₁ᴺ (yᵢ − ŷᵢ)²

Where:

• N is the number of samples.
• yᵢ is the actual target.
• ŷᵢ is the predicted value.

MSE is commonly used for regression tasks.

8.2 Mean Absolute Error (MAE)

MAE = (1 / N) Σᵢ₌₁ᴺ |yᵢ − ŷᵢ|

MAE measures the average absolute difference between predictions and targets.

8.3 Binary Cross-Entropy

For binary classification:

BCE = −(1 / N) Σᵢ₌₁ᴺ [yᵢ ln(ŷᵢ) + (1 − yᵢ) ln(1 − ŷᵢ)]

Here, yᵢ is the binary target and ŷᵢ is the predicted probability.

In numerical implementations, predicted probabilities must be handled safely near 0 and 1.

8.4 Categorical Cross-Entropy

For a classification task with C classes:

CE = −Σⱼ₌₁ᶜ yⱼ ln(pⱼ)

Where yⱼ is the target indicator for class j and pⱼ is the predicted probability for class j.

For a single one-hot encoded target, this simplifies to the negative logarithm of the probability assigned to the correct class.

---

9. Backpropagation

Backpropagation calculates how the loss changes with respect to the network's trainable parameters.

These derivatives are used to update weights and biases during training.

9.1 Derivative of the loss

For a weight w, the gradient is:

∂L / ∂w

It measures how sensitive the loss L is to a small change in w.

9.2 Chain rule

For a function that depends on another function:

dL / dx = (dL / dy) × (dy / dx)

The chain rule allows gradients to be propagated backward through successive operations.

9.3 Activation derivatives

For ReLU, away from zero:

ReLU'(x) = 1 when x > 0

ReLU'(x) = 0 when x < 0

At x = 0, an implementation must choose a derivative convention.

For sigmoid:

σ'(x) = σ(x)(1 − σ(x))

For tanh:

tanh'(x) = 1 − tanh²(x)

For softplus:

Softplus'(x) = σ(x)

For a neuron-specific activation, the derivative must correspond to that neuron's selected function.

9.4 Output-layer error

For a simple scalar-output neuron:

δ = (∂L / ∂a) × f'(z)

Where:

• δ is the local gradient with respect to the pre-activation value.
• L is the loss.
• a is the neuron output.
• z is the weighted sum.
• f'(z) is the activation derivative.

The exact expression depends on the chosen loss and output activation. For example, softmax combined with categorical cross-entropy has a simplified gradient.

9.5 Weight gradient

For a neuron with input xⱼ:

∂L / ∂wⱼ = δxⱼ

And the bias gradient is:

∂L / ∂b = δ

These gradients are used by the optimizer to update parameters.

---

10. Gradient Descent and Optimization

Gradient descent updates parameters in the direction that reduces the loss locally.

10.1 Basic gradient descent

w_new = w_old − η(∂L / ∂w)

b_new = b_old − η(∂L / ∂b)

Where η is the learning rate.

A learning rate that is too large can cause unstable training, while one that is too small can make training slow.

10.2 Stochastic gradient descent

Stochastic gradient descent estimates the gradient using one training example or a small mini-batch.

For a mini-batch B:

w_new = w_old − η∇w L_B

Here, L_B is the loss computed over the mini-batch.

10.3 Adam optimizer

Adam maintains moving averages of gradients and squared gradients.

First moment estimate:

mₜ = β₁mₜ₋₁ + (1 − β₁)gₜ

Second moment estimate:

vₜ = β₂vₜ₋₁ + (1 − β₂)gₜ²

Bias-corrected estimates:

m̂ₜ = mₜ / (1 − β₁ᵗ)

v̂ₜ = vₜ / (1 − β₂ᵗ)

Parameter update:

wₜ = wₜ₋₁ − ηm̂ₜ / (√v̂ₜ + ε)

Here, gₜ is the gradient at step t, β₁ and β₂ control the moving averages, and ε is a small numerical-stability constant.

Adam is one possible future optimization component; its availability depends on the implemented NeuroCran API.

---

11. Regularization

Regularization adds a penalty to the training objective to discourage certain forms of model complexity.

11.1 L1 regularization

L_total = L_data + λΣᵢ |wᵢ|

L1 regularization penalizes the absolute values of weights and can encourage sparse solutions.

11.2 L2 regularization

L_total = L_data + (λ / 2)Σᵢ wᵢ²

L2 regularization penalizes large weights.

Here, λ controls the strength of the regularization penalty.

Regularization should be implemented consistently with the loss calculation and gradient computation.

---

12. Numerical Stability

Neural network operations can encounter overflow, underflow, division by zero, or invalid values.

12.1 Stable sigmoid

Instead of evaluating the exponential expression naively for all input values, use a numerically stable implementation that handles positive and negative inputs safely.

12.2 Stable softmax

For a vector of scores z, subtract the maximum score before exponentiation:

Softmax(zᵢ) = eᶻⁱ⁻ᵐᵃˣ(z) / Σⱼ eᶻʲ⁻ᵐᵃˣ(z)

Subtracting the same constant from every score does not change the softmax probabilities and helps prevent exponential overflow.

12.3 Stable loss calculations

Loss implementations should avoid taking the logarithm of zero and should handle extreme predictions safely. Where possible, use stable combined loss functions instead of separately calculating probabilities and logarithms.

12.4 Shape validation

NeuroCran should validate that matrix and vector dimensions are compatible before performing operations.

For example, a weight matrix with shape (n, m) requires an input vector with m features for the matrix product to produce n outputs.

---

Mathematical Documentation Policy

NeuroCran distinguishes between mathematical definitions and implemented functionality.

• A formula describes the intended mathematical operation.
• A Python example illustrates how an operation might be represented in code.
• A feature is considered supported only when it exists in the current implementation and has been tested.
• Public API documentation must match the actual function signatures and behavior.
• Numerical edge cases and supported input shapes should be documented wherever relevant.

This distinction helps keep NeuroCran's documentation accurate as the project evolves.