PyDyNet：NumPy-based Dynamic Deep Learning Framework

Chinese README: cnREADME.md

Towards Large Language Model

In the summer of 2025, I restart the development of PyDyNet after two years. PyDyNet implemented a pure inference version of Llama3 (6-layer Transformer, vocab-size=32000). The implementation is inspired by the NumPy version and dataset available here. To run it, download the dataset into the llm/llama folder and execute:

>>> python -m llm.llama.infer

There was a boy named Timmy. He loved to play with hi toy and run around outside. One day, Timmy' mom asked him to help her with the laundry. Timmy didn't want to help because he wanted to play. But hi mom said, "Timmy, you need to help me. It' important to help out."
Timmy didn't want to help, but he knew he had to. So, he put on hi shoe and went outside to help hi mom. A they were folding the clothe, Timmy saw a big pile of laundry on the floor. He wanted to help, so he started to pick it up. But then, he accidentally knocked over a pile of clothe and they fell on him. Timmy wa okay, but he felt bad.
Hi mom saw what happened and said, "Timmy, you need to be more careful. You could have hurt yourself." Timmy felt bad and said sorry. Hi mom hugged him and said, "It' okay, accident happen. Let' clean up the laundry together." Timmy learned that it' important to be careful and help out when you need it.

Token count: 262, elapsed: 0.87s, 300 tokens/s

For parameter fine-tuning, run:

python -m llm.llama.finetune --text "A short domain sample for adaptation" --steps 30 --lr 1e-4 --trainable lm_head --save llm/llama/data/finetuned_params.npz

Then load the tuned weights for generation:

python -m llm.llama.infer --prompt "A short domain sample" --finetuned llm/llama/data/finetuned_params.npz

We also implemented a pure inference version of CLIP, inspired by the NumPy version and dataset available NPCLIP. To run it, imigrate data folder of MPCLIP into llm/clip folder and execute:

>>> python -m llm.clip.infer
Label probs: [0.000953   0.48176003 0.51728696]

For parameter fine-tuning on CLIP, run:

python -m llm.clip.finetune --image llm/clip/picture.png --labels "a fish,a dog,a cat" --target 2 --steps 20 --lr 1e-5 --trainable text_encoder.proj,image_encoder.proj --save llm/clip/data/finetuned_clip_params.npz

Then load tuned parameters for inference:

python -m llm.clip.infer --image llm/clip/picture.png --labels "a fish,a dog,a cat" --finetuned llm/clip/data/finetuned_clip_params.npz

for the following image and query ["a fish", "a dog", "a cat"]

Overview

PyDyNet is a neural network framework implemented entirely in NumPy (with CuPy support since version 0.0.7, using the same API). Its syntax is inspired by PyTorch, and its structure is as follows:

graph LR
   N(numpy/cupy.ndarray)--Backend--> A(Tensor) --> ds(Dataset) ---> Data(DataLoader)---> Mission
   A  --Eager execution--> B(Basic operators:<br> add, exp, etc)
   B -.Autograd-.-> A

   B --> CO(Complex<br>operators)
   --> f(Function:<br>img2col, etc) 
   --> M(Basic Module:<br>Linear, etc)
   --> CM(Advanced Module: CNN, RNN, Transformer, etc)
   --> Mission(Learning task)
   A --> GD(Optimizer:<br> SGD, Adam, etc) ---> LS(lr_scheduler: <br>StepLR, etc)---> Mission

Dashed lines indicate that users can disable automatic differentiation using no_grad.

Install

Just

pip install pydynet

or

git clone https://github.com/Kaslanarian/PyDyNet
cd PyDyNet
python setup.py install

Example

Examples can be found in the examples/pydynet directory, with equivalent PyTorch implementations in examples/pytorch. To run an example, use:

python -m examples.pydynet.xxx

Automatic Differentiation

The example autodiff1d.py demonstrates automatic differentiation by performing gradient descent on a one-dimensional convex function:

A multi-variable convex function example is provided in autodiff2d.py:

MLP & LeNet

The example mlp_cnn.py uses MLP and LeNet to classify MNIST digits. The training and testing accuracies are shown below:

Dropout & Batch Normalization

The example mlp_dropout_bn.py compares the performance of three networks on the fetch_olivetti_faces dataset (64×64 pixel images):

1. Three-layer MLP;
2. Three-layer MLP with Dropout;
3. Three-layer MLP with Batch Normalization.

Recurrent Neural Network (RNN)

The example ts_prediction.py demonstrates time series prediction using a GRU:

Transformer

The example transformer.py shows how to train a text classification model using a Transformer. The training results are as follows:

Dataset (CoLA) link: https://nyu-mll.github.io/CoLA/cola_public_1.1.zip

Cuda Acceleration

PyDyNet supports CUDA acceleration through CuPy. To use it, simply install CuPy and use the same API as NumPy. We compare the performance of PyDyNet with CuPy and NumPy as follows on Nvidia GeForce RTX 4090: