slangmod 0.0.4

Small language model.

slangmod

Small language model.

Ever wondered how large language models (LLMs) like ChatGPT, Claude, LLama, Deepseek, etc., actually work, like, really work? I did. And I figured there is only one way to find out: Make one yourself. From scratch.

Of course, I wasn't expecting to beat the big players at their own game, but I wanted to know what you can do on consumer hardware (meaning a state-of-the art gaming PC with a single graphics card supported by PyTorch). So, naturally, it was going to be a small language model. These hardware limitations are reflected in software design choices. Specifically, slangmod does not employ any type of parallelization that would keep multiple GPUs busy at the same time, and all training data are loaded into CPU RAM at once, to be drip-fed to the model on the GPU from there (1 billion tokens take up about 7.5 GB worth of 64-bit integer numbers).

Having said that, slangmod provides everything you need to

• preprocess and clean your text corpus;
• chose and train one of the HuggingFace tokenizers;
• specify a Transformer model including the type of positional encodings and the feedforward block;
• train your model with a choice of optimizers and learning-rate schedulers, employing early-stopping if you like;
• monitor convergence and experiment on hyperparameters;
• explore text-generation algorithms like top-k, top-p or beamsearch;
• and, finally, chat with your model.

To do all these things, slangmod provides a command-line interface (CLI) with fine-grained configuration options on one hand, and the raw building blocks it is made of on the other hand. Leveraging the foundational functionalities provided by the fiercely functional swak package, any other workflow can thus be quickly coded up.

Installation

• Create a new virtual environment running at least python 3.12.
• The easiest way of installing slangmod is from the python package index PyPI, where it is hosted. Simply type

  pip install slangmod
  

  or treat it like any other python package in your dependency management.
• While it is, in principle, possible to run slangmod on the CPU, this is only intended for debugging purposes. To get any results in finite time, you also need a decent graphics card, and you must have a working installation of PyTorch to make good use of it. Because there is no way of knowing which version of CUDA (or ROC) you have installed on your machine and how you installed it, PyTorch is not an explicit dependency of slangmod. You will have to install it yourself, e.g., following these instructions. If you are using pipenv for dependency management, you can also have a look at the Pipfile in the root of the slangmod repository and taylor it to your needs. Personally, I go

  pipenv sync --categories=cpu
  

  for a CPU-only installation of PyTorch and

  pipenv sync --categories=cuda
  

  if I want GPU support.

Documentation

The documentation for both the CLI and the API of slangmod is hosted on GitHub Pages.