ann-engine 0.1.0

Neural Network framework built from scratch using NumPy

ANN Engine – A Minimal Neural Network Training Framework from Scratch

OverView

ANN Engine is a lightweight neural network training framework built from scratch using NumPy.

The objective of this project is to understand and implement the core mechanics of deep learning frameworks such as:

• Forward propagation
• Backpropagation
• Automatic gradient computation
• Optimization algorithms
• Modular layer design
• Training loop abstraction

This project focuses on clarity, modularity, and mathematical correctness rather than performance.

Motivation

Modern deep learning frameworks abstract away gradient computation and optimization.

To deeply understand how neural networks train internally, this project re-implements:

• Linear layers
• Activation functions
• Loss functions
• Optimizers
• Backpropagation logic
• Model class with fit/predict interface

The goal is to bridge the gap between theory and production frameworks.

Installation

From PyPI (Recommended)

bash
pip install ann-engine

From Source

bash
# Clone the repository
git clone https://github.com/biswas-bit/ANN-Engine.git
cd ANN-Engine

# Install in development mode
pip install -e .

Build from Source

bash
# Build the package
python -m build

# Install the built package
pip install dist/ann_engine-0.1.0-py3-none-any.whl

Quick Start

Here's a simple example to get started with the ANN Engine:

python
import numpy as np
from ann_engine.layers import Sequential, Dense
from ann_engine.losses import CrossEntropyLoss
from ann_engine.optimizers import Adam

# Create a simple neural network
model = Sequential([
    Dense(128, activation='relu', input_dim=784),
    Dense(64, activation='relu'),
    Dense(10, activation='softmax')
])

# Compile the model with loss and optimizer
model.compile(loss='cross_entropy', optimizer=Adam(learning_rate=0.001))

# Train the model
# X_train: training data (n_samples, 784)
# y_train: training labels (n_samples,)
model.fit(X_train, y_train, epochs=10, batch_size=32)

# Make predictions
predictions = model.predict(X_test)

Features

🧠 Neural Network Layers

Layer	Description
Dense	Fully connected linear layer
Sequential	Container for stacking layers

⚡ Activation Functions

Activation	Description
ReLU	Rectified Linear Unit: max(0, x)
Sigmoid	Sigmoid function: 1 / (1 + exp(-x))
Tanh	Hyperbolic tangent
LeakyReLU	Leaky ReLU with small negative slope
ELU	Exponential Linear Unit
Softmax	Softmax function for multi-class classification
LogSoftmax	Log of softmax function
Softplus	Softplus function: log(1 + exp(x))
Swish	Swish activation: x * sigmoid(x)
GELU	Gaussian Error Linear Unit
Identity	Identity function (no activation)

📉 Loss Functions

Loss Function	Description	Use Case
MSELoss	Mean Squared Error	Regression
CrossEntropyLoss	Cross Entropy Loss	Multi-class Classification
NLLLoss	Negative Log Likelihood	Classification
BCELoss	Binary Cross Entropy	Binary Classification
BCEWithLogitsLoss	BCE with sigmoid built-in	Binary Classification
HuberLoss	Huber loss (smooth L1)	Regression

🔧 Optimizers

Optimizer	Description
SGD	Stochastic Gradient Descent
SGDWithMomentum	SGD with Momentum
NAG	Nesterov Accelerated Gradient
AdaGrad	Adaptive Gradient
Adam	Adaptive Moment Estimation
RMSProp	Root Mean Square Propagation

API Reference

Creating a Model

Method 1: Using Sequential API

python
from ann_engine.layers import Sequential, Dense, ReLU, Softmax

# Create a sequential model
model = Sequential([
    Dense(256, activation='relu', input_dim=784),
    Dense(128, activation='relu'),
    Dense(10, activation='softmax')
])

Method 2: Using Model Class (Custom Architecture)

python
from ann_engine.engine.models import Model
from ann_engine.layers import Dense, ReLU, Softmax

class MyModel(Model):
    def __init__(self):
        super().__init__()
        self.fc1 = Dense(256, activation='relu')
        self.fc2 = Dense(128, activation='relu')
        self.fc3 = Dense(10, activation='softmax')
    
    def forward(self, x):
        x = self.fc1(x)
        x = self.fc2(x)
        x = self.fc3(x)
        return x

model = MyModel()

Compiling the Model

python
from ann_engine.optimizers import Adam, SGD, RMSProp

# Using Adam optimizer (recommended default)
model.compile(
    loss='cross_entropy',  # or 'mse', 'bce', etc.
    optimizer=Adam(learning_rate=0.001)
)

# Using SGD with momentum
model.compile(
    loss='mse',
    optimizer=SGD(learning_rate=0.01, momentum=0.9)
)

# Using RMSProp
model.compile(
    loss='cross_entropy',
    optimizer=RMSProp(learning_rate=0.001)
)

Training the Model

python
import numpy as np

# X_train: (n_samples, n_features)
# y_train: (n_samples,) - class labels for classification
#          or (n_samples, n_outputs) for regression

# Basic training
model.fit(X_train, y_train, epochs=10, batch_size=32)

# Training with validation
model.fit(
    X_train, y_train,
    epochs=20,
    batch_size=64,
    validation_data=(X_val, y_val)
)

# Training with verbose output
model.fit(
    X_train, y_train,
    epochs=10,
    batch_size=32,
    verbose=1
)

Making Predictions

python
# Predict class labels
predictions = model.predict(X_test)
# Returns: array of class indices

# Get prediction probabilities
probabilities = model.predict_proba(X_test)
# Returns: array of shape (n_samples, n_classes)

Evaluating the Model

python
# Evaluate on test data
loss, accuracy = model.evaluate(X_test, y_test)
print(f"Test Loss: {loss:.4f}")
print(f"Test Accuracy: {accuracy:.4f}")

Examples

MNIST Classification

python
import numpy as np
from ann_engine.layers import Sequential, Dense
from ann_engine.losses import CrossEntropyLoss
from ann_engine.optimizers import Adam

# Load and preprocess MNIST data
# X_train: (60000, 784), y_train: (60000,)
# X_test: (10000, 784), y_test: (10000,)

# Flatten images: (28, 28) -> (784,)
X_train = X_train.reshape(-1, 784)
X_test = X_test.reshape(-1, 784)

# Normalize to [0, 1]
X_train = X_train / 255.0
X_test = X_test / 255.0

# Create model
model = Sequential([
    Dense(256, activation='relu', input_dim=784),
    Dense(128, activation='relu'),
    Dense(10, activation='softmax')
])

# Compile and train
model.compile(
    loss='cross_entropy',
    optimizer=Adam(learning_rate=0.001)
)

model.fit(X_train, y_train, epochs=10, batch_size=128, verbose=1)

# Evaluate
loss, accuracy = model.evaluate(X_test, y_test)
print(f"Test Accuracy: {accuracy * 100:.2f}%")

Binary Classification

python
from ann_engine.layers import Sequential, Dense, Sigmoid
from ann_engine.losses import BCELoss
from ann_engine.optimizers import Adam

# Create model for binary classification
model = Sequential([
    Dense(64, activation='relu', input_dim=20),
    Dense(32, activation='relu'),
    Dense(1, activation='sigmoid')
])

model.compile(
    loss='bce',  # Binary Cross Entropy
    optimizer=Adam(learning_rate=0.001)
)

model.fit(X_train, y_train, epochs=50, batch_size=32)

# Predictions are probabilities between 0 and 1
predictions = model.predict(X_test)

Regression

python
from ann_engine.layers import Sequential, Dense, Identity
from ann_engine.losses import MSELoss, HuberLoss
from ann_engine.optimizers import SGD

# Create model for regression
model = Sequential([
    Dense(128, activation='relu', input_dim=13),
    Dense(64, activation='relu'),
    Dense(1, activation='identity')  # No activation for regression
])

model.compile(
    loss='mse',  # or 'huber' for Huber loss
    optimizer=SGD(learning_rate=0.01, momentum=0.9)
)

model.fit(X_train, y_train, epochs=100, batch_size=32)

# Predict continuous values
predictions = model.predict(X_test)

Custom Layer Configuration

python
from ann_engine.layers import Dense, ReLU

# Customizing Dense layer parameters
layer = Dense(
    units=512,           # Number of neurons
    activation='relu',   # Activation function
    use_bias=True,       # Whether to use bias
    kernel_initializer='glorot',  # Weight initialization
    bias_initializer='zeros'
)

Advanced Usage

Learning Rate Scheduling

python
from ann_engine.optimizers import Adam

# Using a smaller learning rate
optimizer = Adam(learning_rate=0.0001)
model.compile(loss='cross_entropy', optimizer=optimizer)

Early Stopping (Manual Implementation)

python
best_loss = float('inf')
patience = 5
no_improvement_count = 0

for epoch in range(100):
    model.fit(X_train, y_train, epochs=1, batch_size=32, verbose=0)
    val_loss = model.evaluate(X_val, y_val)[0]
    
    if val_loss < best_loss:
        best_loss = val_loss
        no_improvement_count = 0
        # Save best model weights
        best_weights = model.get_weights()
    else:
        no_improvement_count += 1
        if no_improvement_count >= patience:
            print(f"Early stopping at epoch {epoch}")
            break

Saving and Loading Weights

python
# Get model weights
weights = model.get_weights()

# Set model weights
model.set_weights(weights)

Architecture Details

Forward Propagation

Input → Dense (linear) → Activation → ... → Output

Backpropagation

The framework automatically computes gradients through automatic differentiation:

1. Forward pass: Compute outputs and cache intermediate values
2. Backward pass: Compute gradients using chain rule
3. Update weights: Use optimizer to update parameters

Supported Input Formats

• Training Data: (n_samples, n_features) - 2D numpy array
• Labels:
  • Classification: (n_samples,) - 1D array of class indices
  • Regression: (n_samples, n_outputs) - 2D array

Performance Tips

1. Batch Size: Start with 32-128. Larger batches train faster but may require learning rate adjustment.

2. Learning Rate:

   • Adam: 0.001 (default)
   • SGD: 0.01-0.1
   • Adjust based on convergence behavior

3. Network Architecture:

   • Start with simpler networks
   • Add complexity gradually
   • Monitor for overfitting

4. Data Preprocessing:

   • Normalize/standardize inputs
   • Shuffle data between epochs

Troubleshooting

NaN Losses

• Reduce learning rate
• Check for division by zero
• Verify data preprocessing
• Consider gradient clipping

Poor Convergence

• Increase model capacity
• Adjust learning rate
• Check activation function choices
• Verify data quality

Memory Issues

• Reduce batch size
• Use smaller network
• Process data in chunks

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

MIT License - See LICENSE file for details.

Acknowledgments

This project is inspired by:

• PyTorch
• TensorFlow
• Keras

Built with ❤️ using NumPy.

Folder Structure

ann_engine/
├── __init__.py
│
├── core/
│   ├── __init__.py
│   ├── parameter.py
│   └── tensor.py
│
├── datasets/
│   ├── __init__.py
│   ├── base.py
│   └── datasets.py
│
├── engine/
│   ├── __init__.py
│   └── models.py
│
├── layers/
│   ├── __init__.py
│   ├── activations.py
│   ├── base.py
│   ├── dense.py
│   ├── layers.py
│   └── sequential.py
│
├── losses/
│   ├── __init__.py
│   ├── base.py
│   └── loss.py
│
├── optimizers/
│   ├── __init__.py
│   ├── adagrad.py
│   ├── adam.py
│   ├── base.py
│   ├── nag.py
│   ├── rsmprop.py
│   └── sgd.py
│
└── utils/
    └── __init__.py

Version History

• 0.1.0: Initial release
  • Basic Dense and Sequential layers
  • Common activation functions
  • Core optimizers (SGD, Adam, etc.)
  • Loss functions for classification and regression