TokenCraft
Learn how GPT-style language models work — from tokens to text generation.
TokenCraft is an educational, decoder-only GPT-style language model implemented with PyTorch. It helps learners explore the building blocks behind transformer language models: tokenization, embeddings, causal self-attention, transformer blocks, training, and autoregressive text generation.
Install TokenCraft
pip install tokencraft-gpt
Author: Ranjeet Epili
TokenCraft is an educational project—not a pretrained general-purpose chatbot. It is designed to help you understand how a GPT-style model works and experiment with training on your own text.
Table of Contents
- What is TokenCraft?
- The big picture
- How the transformer works
- Installation
- How to use TokenCraft
- Project structure
- Limitations
- Inspiration and acknowledgements
- Author and license
What is TokenCraft?
A language model learns statistical patterns in sequences. Given a context such as the model learns, it estimates which token could come next. TokenCraft demonstrates this process through a compact GPT-style architecture.
It includes educational implementations of:
- Tokenization and token IDs
- Token and positional embeddings
- Causal self-attention
- Query, Key, and Value projections
- Multi-head attention
- Feed-forward networks and transformer blocks
- Next-token training
- Autoregressive text generation
The big picture
The model converts text into token IDs, maps those IDs into vectors, processes the vectors through transformer blocks, and produces scores for possible next tokens. During generation, a selected token is appended to the context and the model runs again.
How the transformer works
1. Text and tokenization
A tokenizer turns text into a sequence of integer token IDs. Depending on the tokenizer, a token can represent a word, a word fragment, punctuation, or another text unit.
For a small illustrative vocabulary:
| Token | Example ID |
|---|---|
the |
12 |
model |
37 |
learns |
21 |
Token IDs are not numerical measurements of meaning. They are indices used to look up vectors in an embedding table. The IDs above are only an example; your tokenizer's vocabulary will determine the actual IDs.
2. Token and positional embeddings
The embedding table maps every token ID to a learned vector. But a sequence also has order: “the model” and “model the” contain the same words in different positions.
A positional embedding provides information about where a token appears. Token and positional embeddings are combined to form the initial representation:
Input representation = token embedding + positional embedding
This gives the transformer information about both the token and its position in the sequence.
3. Causal self-attention
Self-attention lets each token build a context-aware representation by gathering information from other tokens. A GPT-style model uses causal attention: a position may attend to itself and earlier positions, but not future tokens.
In this diagram, green cells are allowed and gray cells are blocked. The third token can use tokens one, two, and three—but cannot see token four. This prevents the model from “cheating” by looking at the answer while learning to predict the next token.
4. Query, Key, and Value
Each input vector is transformed into three learned representations:
- Query (Q): what the current position is looking for
- Key (K): what each position can be matched against
- Value (V): the information each position contributes
The scaled dot-product attention operation is:
[ \operatorname{Attention}(Q,K,V) =\operatorname{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}+M\right)V ]
Here, (d_k) is the key dimension and (M) is the causal mask. Query-key similarity produces attention scores. The mask blocks future positions, softmax converts the scores into weights, and those weights combine the value vectors.
5. Multi-head attention
One attention head learns one set of contextual relationships. Multi-head attention runs several attention heads in parallel, each with its own learned projections.
The heads' outputs are concatenated and projected into a shared representation. Different heads can learn different patterns—for example, relationships between nearby tokens or dependencies across a longer context.
6. Feed-forward network and transformer block
After attention, a feed-forward network applies learned transformations to each position's representation. A transformer block combines attention and feed-forward processing, along with residual connections and normalization where implemented.
Stacking transformer blocks lets the model refine its contextual representations layer by layer.
7. Next-token training
GPT-style language modeling is trained as a next-token prediction task. The target sequence is shifted by one token relative to the input:
| Context provided to the model | Target token |
|---|---|
the |
model |
the model |
learns |
the model learns |
from |
The model produces logits—scores for tokens in its vocabulary. Cross-entropy loss compares the predictions with the correct next tokens. An optimizer such as AdamW updates the model parameters to reduce this loss.
Training text, model size, and training duration influence what the model learns. A small corpus is useful for demonstrating the process, but does not give a model broad language knowledge.
8. Autoregressive generation
After training, the model can generate text from a prompt. It predicts the next token, appends the selected token to the context, and predicts again.
For example, a prompt might be:
The model
The model may continue with a sequence such as:
The model learns from text
This is an illustrative example, not a guaranteed output. The actual continuation depends on learned weights, prompt, and sampling settings.
Temperature adjusts the sampling distribution:
- Lower temperature generally makes token selection more focused.
- Higher temperature generally makes token selection more varied.
Generation is iterative: each generated token becomes part of the context for the next prediction.
Installation
Install TokenCraft from PyPI:
pip install tokencraft-gpt
The distribution name used by pip is tokencraft-gpt. The Python import name is tokencraft.
How to use TokenCraft
1. Import the package
import tokencraft
from tokencraft import GPT, Tokenizer
print(tokencraft.__version__)
2. Inspect the available API
The model and tokenizer classes are exposed by the package. Use Python's built-in help to inspect the constructor arguments and methods available in your installed version:
help(GPT)
help(Tokenizer)
3. Prepare a text dataset
Create a plain-text file containing the material you want the model to learn from. For example, save this as data.txt:
A language model learns patterns from text.
A transformer uses attention to process sequences.
TokenCraft demonstrates next-token prediction.
For meaningful experiments, use a larger and appropriately prepared dataset. Keep track of the data you train on and ensure you have permission to use it.
4. Train the model
A typical training workflow is:
- Load the training text.
- Build the tokenizer vocabulary and encode the text into token IDs.
- Create input sequences and next-token targets.
- Initialize the GPT model using the model configuration.
- Run the training loop with a next-token loss such as cross-entropy.
- Update the parameters with an optimizer such as AdamW.
- Save the trained model parameters for later use.
TokenCraft's exact constructor arguments and training options are version-specific. Inspect GPT and Tokenizer with help() and refer to the project's training example or scripts for the precise commands.
5. Generate text from a trained model
The generation workflow is:
- Load the trained model parameters.
- Encode a prompt with the same tokenizer used during training.
- Pass the prompt tokens to the model.
- Select a next token from the model's output.
- Append the token and repeat until the desired length or stopping condition is reached.
- Decode the generated token IDs back into text.
The model must be trained first—or loaded from a compatible checkpoint—before its output can reflect learned patterns. The package does not automatically download pretrained weights.
Quick API check
Use this small snippet to confirm that the package installed and its main classes are importable:
import tokencraft
print("TokenCraft version:", tokencraft.__version__)
print("GPT class:", tokencraft.GPT)
print("Tokenizer class:", tokencraft.Tokenizer)
Project structure
The package is organized around the main GPT components:
tokencraft/
├── __init__.py
├── tokenizer.py
└── model/
├── __init__.py
├── gpt.py
├── embedding.py
├── positional_embedding.py
├── input_embedding.py
├── self_attention.py
├── multi_head_attention.py
├── feed_forward.py
└── transformer_block.py
The source repository may also contain training scripts, generation examples, and sample data.
Limitations
TokenCraft is intended for education and experimentation:
- It is not a pretrained production chatbot.
- Generated text depends on the training data and model configuration.
- Small models and small datasets may generate repetitive or incoherent text.
- It is not designed to match the capabilities of large-scale language models.
Inspiration and acknowledgements
TokenCraft's educational approach is inspired by transformer literature and visual explanations such as Jay Alammar's The Illustrated Transformer. This README uses original diagrams and explanations tailored to a decoder-only GPT-style model; it does not reproduce that article's illustrations.
Author and license
Author: Ranjeet Epili
License: MIT
Metadata
Release files for tokencraft-gpt 0.1.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| tokencraft_gpt-0.1.1.tar.gz | 132.5 kB | Without provenance | Details for tokencraft_gpt-0.1.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| tokencraft_gpt-0.1.1-py3-none-any.whl | Python 3 | none | any | Details for tokencraft_gpt-0.1.1-py3-none-any.whl |
Total release size: 144.2 kB
Release files / tokencraft_gpt-0.1.1.tar.gz
| Download URL | tokencraft_gpt-0.1.1.tar.gz |
|---|---|
| Size | 132.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
4548ce497624a272a5e4c2e8fc4583a188a2768e6f60bf1a116a5127ba9eedf2
|
|
BLAKE2b-256 checksum How to use checksums |
67b4fa3b9b93ac342fcc964a271ff94e63751d645a4e92cfda84e15b4e55f63a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.15.1
|
Release files / tokencraft_gpt-0.1.1-py3-none-any.whl
| Download URL | tokencraft_gpt-0.1.1-py3-none-any.whl |
|---|---|
| Size | 11.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
8d4fb6da794681ec9bc162acdebf2d1f027ab64fd2cc98fffe79bbfe65485313
|
|
BLAKE2b-256 checksum How to use checksums |
3e47fd13626fe9dfb92891315d7577f269a744d9d0a36de82b7538c67cb82b2c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.15.1
|
Security for tokencraft-gpt 0.1.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.