neural-net-checklist 1.0.0

A ready-to-use implementation of Andrei Karpathhy's "recipe for training neural networks"

neural_net_checklist: A Codified Recipe for Training Neural Networks

This repository implements a set of diagnostic assertions based on Andrei Karpathy's blog post "A Recipe for Training Neural Networks". Instead of a manual checklist, we provide programmatic checks to diagnose and debug neural networks efficiently.

Why This Exists

Training neural networks can be tricky. This toolkit saves you time by automating common diagnostic steps, allowing you to focus on the interesting parts of your model development.

What's Included

We've implemented the following checks:

1. Verify loss @ init (for balanced classification tasks)

   assert_balanced_classification_cross_entropy_loss_at_init
   

2. Init well (for balanced classification tasks)

   assert_balanced_classification_cross_entropy_loss_at_init
   

3. Non-zero gradients

   assert_non_zero_gradients
   

4. Batch independence

   • Forward pass (memory-efficient, but note: batchnorm breaks this naturally)

     assert_batch_independence_forward
     

   • Backward pass (uses more memory, checks gradients)

     assert_batch_independence_backward
     

5. Overfit one batch

   assert_overfit_one_batch
   

6. Visualize just before the net

   patch_module_raise_inputs
   

Quick Start

To run all assertions:

• For classification tasks (e.g., computer vision):

  assert_all_for_classification_cross_entropy_loss
  

• For causal language models:

  assert_all_for_llm_cross_entropy_loss
  

Installation

pip install neural_net_checklist

Usage Example

import neural_net_checklist.torch_diagnostics as torch_diagnostics

# Assume you have a model and a DataLoader
model = YourModel()
train_loader = YourDataLoader()

# Run all checks
torch_diagnostics.assert_all_for_classification_cross_entropy_loss(model, train_loader)

Contributing

Contributions are welcome! If you have ideas for additional checks or improvements, please open an issue or submit a pull request.

License

This project is licensed under the MIT License - see the LICENSE file for details.

---

Happy debugging! Remember, neural nets can be finicky, but with the right tools, we can tame them. 🧠🔧