Building Neural Networks from scratch.
Project description
NN-Without-Frameworks
This project aims to implement different Neural Network configuration without using scientific frameworks like TensorFlow or PyTorch.
Each network/config is implemented in 4 formats while trying to mimic PyTorch's Subclassing procedure:
- In Python using NumPy
- In Python without taking advantage of NumPy
- In java
- In C++
What is currently supported? (equally in all languages and formats)
Layers:
- Fully Connected
Activations:
- Linear
- ReLU
Loss functions:
- MSE
- Cross Entropy
Weight Initializers:
- Xavier Uniform (aka Glorot)
- He Normal (aka Kaiming Normal)
bias initializer:
- Constant (zero)
Optimizers:
- SGD
- SGD + Momentum
- RMSProp
- AdaGrad
- Adam
Regularizer:
- l1
- l2
Normalization:
- BatchNorm1d
Examples
Each directory contains a train.*
that performs tests of correctness and functionality according to its corresponding format and language. You can run it to get a sense of what is going on.
Snippet
Define your network
- Python:
import python_nn.numpy_nn as nn # import python_nn.pure_nn as nn
class MyNet(nn.Module):
def __init__(self, input_dim, out_dim):
super().__init__()
self.input_dim = input_dim
self.out_dim = out_dim
self.hidden1 = nn.layers.Dense(in_features=self.input_dim,
out_features=100,
activation=nn.acts.ReLU(),
weight_initializer=nn.inits.HeNormal(nn.acts.ReLU()),
bias_initializer=nn.inits.Constant(0.),
regularizer_type="l2",
lam=1e-3
)
self.output = nn.layers.Dense(in_features=100,
out_features=self.out_dim,
weight_initializer=nn.inits.XavierUniform(),
bias_initializer=nn.inits.Constant(0.),
regularizer_type="l2",
lam=1e-3
)
def forward(self, x):
x = self.hidden1(x)
return self.output(x)
- Java:
import Layers.Dense;
class MyNet extends Module{
int in_features = 0, out_features = 0;
Dense hidden1, output;
public MyNet(int in_features, int out_features){
this.in_features = in_features;
this.out_features = out_features;
this.hidden1 = new Dense(this.in_features,
100,
"relu",
"he_normal",
"zeros", // bias initializer
"l2",
0.001F);
this.layers.add(this.hidden1); // the Crucial and only different part to PyTorch's subclassing
this.output = new Dense(100,
out_features,
"linear",
"xavier_uniform",
"zeros", // bias initializer
"l2",
0.001F);
this.layers.add(this.output); // Crucial and different part to PyTorch's subclassing
}
public float[][] forward(float[][] x){
x = this.hidden1.forward(x);
x = this.output.forward(x);
return x;
}
}
- C++:
#include <iostream>
#include <module.h>
#include <layers.h>
using namespace std;
class MyNet : public Module{
public:
int in_features = 0, out_features = 0;
Dense *hidden, *output; // Layers should be predifned especially, they should be pointers
MyNet(int in_features, int out_features){
this->in_features = in_features;
this->out_features = out_features;
this->hidden = new Dense{this->in_features,
100,
"relu",
"he_normal",
"zeros", // bias initializer
"l2",
0.001};
this->parameters.push_back(this->hidden); // same as java
this->output = new Dense{100,
this->out_features,
"linear",
"xavier_uniform",
"zeros", // bias initializer
"l2",
0.001};
this->parameters.push_back(this->output); same as java
}
float_batch forward(const float_batch &input){ // float_batch =: vector<vector<float> >
float_batch x = this->hidden->forward(input);
x = this->output->forward(x);
return x;
}
};
Train your network
- Python
my_net = MyNet(num_features, num_classes)
ce_loss = nn.losses.CrossEntropyLoss()
opt = nn.optims.SGD(my_net.parameters, lr=1.)
for step in range(epoch):
y = my_net(x)
loss = ce_loss(y, t)
my_net.backward(loss)
opt.apply()
- Java:
import Losses.*;
import Optimizers.*;
MyNet my_net = new MyNet(num_features, num_classes);
CrossEntropyLoss celoss = new CrossEntropyLoss();
SGD opt = new SGD(1.0F, my_net.layers);
for (int epoch = 0; epoch < num_epoch; epoch++) {
y = my_net.forward(x);
Loss loss = celoss.apply(y, t);
my_net.backward(loss);
opt.apply();
}
- C++:
#include <losses.h>
#include <optimizers.h>
MyNet my_net = MyNet{num_features, num_classes};
CrossEntropyLoss celoss;
SGD opt(1, my_net.parameters);
float_batch y; // float_batch =: vector<vector<float> >
for(int step = 0; step < num_epoch; step++){
y= my_net.forward(x);
Loss loss = celoss.apply(y, t);
my_net.backward(loss);
opt.apply();
}
Acknowledgement
- Current code is inspired by the elegant and simple repository Simple Neural Networks by @MorvanZhou .
- Mathematical foundation of different parts is based on slides of CS W182 / 282A and CS231n courses.
Contributing
- The current code is far from done and any fix, suggestion, pull request, issue, etc is highly appreciated and welcome. 🤗
- Current work focuses on discovering what is under the hood rather optimization involved in implementing ideas so, feel free to conduct sanity-checks behind the math and correctness of each part and/or if you come up with a better or optimized solution, please don't hesitate to bring up a PR. [thank you in advance. 😊]
- You can take a look at
todo.md
.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for nn_without_frameworks-0.1.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9209fa0a18d9ddf217c4a74760c6544aca9c9ff80160c2e67021367f75a177f4 |
|
MD5 | 1154c9850158f4c52f6c6ff4ff2efdb0 |
|
BLAKE2b-256 | f965a445da5d2eda01ec3320115ab8e1fbe144d2159e51544e9e7734aec040de |
Close
Hashes for nn_without_frameworks-0.1.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | af6b92e5202424009c8f1fed1f1cedd45eaf62ee2e093506b86adc969bdb2190 |
|
MD5 | 016a61e1728210584697aba04647664a |
|
BLAKE2b-256 | c79f78de8a270714e6ec7b2a55b10077e3d81d9af2cc07d505713643487d975f |