pokepy-generator 1.0.5

A character-level language model built from scratch using only numpy.

pokepy :)

a character-level language model built completely from scratch using only numpy that generates pokémon sounding names. no PyTorch, no autograd, no deep learning frameworks — every forward pass, backward pass, and gradient update is manually implemented.

this project started as a way to understand how neural networks actually work under the hood. I first built a simple MLP, then expanded it into a WaveNet-style architecture to explore how increasing context length changes what a model can learn.

demo

this project is fully distributed on PyPI. follow these steps to set up an isolated sandbox environment and run the demo across macOS, Linux, or Windows.

initialize a venv

to bypass native system file locks on macOS and ensure dependencies install safely, create and activate a localized virtual environment:

On macOS/Linux:

python3 -m venv .venv
source .venv/bin/activate 

On windows:
python -m venv .venv
.venv\Scripts\activate

### install the package
pip install pokepy-generator

### launch the app

Run the terminal execution shortcut from anywhere in the activated enviornment:
pokepy

Then open your browser and head to the local link:



---

# what is this

I wanted to understand the foundations behind language models, so I built a mini character-level text generator completely from scratch.

instead of relying on existing machine learning libraries, I manually implemented:

* embeddings
* linear layers
* batch normalization
* tanh activations
* softmax
* cross entropy loss
* backpropagation
* gradient descent

everything runs only with numpy.

the model learns character patterns from pokémon names and generates new names based on the relationships it discovers.

---

# model 1 — MLP

## architecture

the first version was a simple multilayer perceptron:

* character embedding layer (10-dimensional vectors)
* linear layer
* batch normalization
* tanh activation
* output linear layer
* softmax + cross entropy loss

hidden size:

200 neurons

context length:

3 characters

this means the model only looks at the previous 3 characters to predict the next one.

example:

pik → a ika → next character

the context continuously shifts as the model generates.

---

## MLP results

training:

train loss: 1.294 validation loss: 3.504

generated names:

blipedeedo rosalini lect dartic star vigus swannon hippowdon the larvinerao

the MLP learned basic character relationships, but the limited context window made it difficult to understand longer patterns inside names.

---

# model 2 — WaveNet

## why I built this

the biggest limitation of the MLP was context length.

with only 3 characters of context, the model could only see a small part of each name.

for example:

charizard

cha har ari riz

the model does not understand the larger structure of the word.

WaveNet improves this by gradually combining groups of characters, allowing the model to build larger representations without massively increasing the number of parameters.

---

## architecture

WaveNet-style architecture:

* character embedding layer (10 dimensions)
* FlattenConsecutive layers
* multiple linear layers
* batch normalization
* tanh activations
* final output layer
* softmax + cross entropy loss

context length:

8 characters

the model builds information hierarchically:

characters

↓

combined character groups

↓

higher level features

↓

next character prediction

---

## WaveNet results

training:

train loss: 1.949 validation loss: 2.748

generated names:

gropinig pyghislacat poloun hoongel spuspiniyan ongtover kasato xel felspipon linmatie asherron beatdiqdule madstutf drudona rouzslra liwsywunk galeon magnoslaws araidono lickopt

WaveNet produced longer and more structured generations because it had access to a larger context window.

---

# MLP vs WaveNet

|                  | MLP                 | WaveNet             |
| ---------------- | ------------------- | ------------------- |
| Context size     | 3 characters        | 8 characters        |
| Hidden size      | 200                 | 32                  |
| Training steps   | 300,000             | 10,000              |
| Architecture     | Single hidden layer | Hierarchical layers |
| Feature learning | Direct              | Progressive         |
| Main advantage   | Simple baseline     | Larger context      |

---

# challenges

## context length

one of the biggest lessons from this project was understanding why context matters.

a model with a smaller context window can only learn local patterns, while larger context allows it to understand longer relationships.

the MLP used:

3 character context

while WaveNet increased this to:

8 character context

which allowed it to capture more structure from names.

---

## batch normalization

implementing batch normalization manually was one of the hardest parts.

I had to handle:

* batch mean
* batch variance
* running mean
* running variance

training and inference use different statistics, so saving the running values was required for the deployed model to generate correctly.

---

## backpropagation

instead of using:

```python
loss.backward()

I manually calculated gradients for:

• embeddings
• linear layers
• batch normalization
• tanh activations
• softmax cross entropy

this helped me understand how neural networks actually learn instead of treating them as black boxes.

---

random generation

generation is probabilistic.

even with the same trained model, outputs change because the next character is sampled from the model's probability distribution.

---

training details

MLP

dataset:

pokemon names

training:

• optimizer: SGD
• batch size: 32
• steps: 300,000
• learning rate: 0.1 → 0.01 after 100k steps

parameters:

C
W1
W2
b2
bngain
bnbias

---

WaveNet

dataset:

pokemon names

training:

• optimizer: SGD
• batch size: 32
• steps: 10,000
• learning rate: 0.1 → 0.01 after 8000 steps

parameters:

embeddings
linear layers
batch normalization parameters

---

deployment

the model is deployed using Hugging Face Spaces with Gradio.

the demo loads the trained numpy weights and runs inference without PyTorch or external ML frameworks.

the deployed model uses:

• trained embeddings
• linear layer weights
• batch normalization parameters
• vocabulary mappings

the entire inference pipeline runs using manually implemented numpy layers.

---

usage

install dependencies:

pip install numpy

run:

python mlp.py
python wavenet.py

you will need:

data/

└── pokemon.txt

with one pokémon name per line.

---

what I learned

this project taught me how language models are built from the ground up.

the biggest takeaway was that improving a model is not always about making it bigger. changing the architecture and giving the model better ways to understand context can have a larger impact than simply adding more parameters.

---

license

MIT

heavily inspired by Andrej Karpathy's makemore series — highly recommend if you want to understand neural networks from the inside out :)