Skip to main content

TokenCraft

Learn how GPT-style language models work — from tokens to text generation.

TokenCraft is an educational, decoder-only GPT-style language model implemented with PyTorch. It helps learners explore the building blocks behind transformer language models: tokenization, embeddings, causal self-attention, transformer blocks, training, and autoregressive text generation.

Install TokenCraft

pip install tokencraft-gpt

Author: Ranjeet Epili

TokenCraft is an educational project—not a pretrained general-purpose chatbot. It is designed to help you understand how a GPT-style model works and experiment with training on your own text.


Table of Contents

What is TokenCraft?

A language model learns statistical patterns in sequences. Given a context such as the model learns, it estimates which token could come next. TokenCraft demonstrates this process through a compact GPT-style architecture.

It includes educational implementations of:

  • Tokenization and token IDs
  • Token and positional embeddings
  • Causal self-attention
  • Query, Key, and Value projections
  • Multi-head attention
  • Feed-forward networks and transformer blocks
  • Next-token training
  • Autoregressive text generation

The big picture

TokenCraft decoder-only GPT architecture

The model converts text into token IDs, maps those IDs into vectors, processes the vectors through transformer blocks, and produces scores for possible next tokens. During generation, a selected token is appended to the context and the model runs again.

How the transformer works

1. Text and tokenization

Text to tokens to token IDs

A tokenizer turns text into a sequence of integer token IDs. Depending on the tokenizer, a token can represent a word, a word fragment, punctuation, or another text unit.

For a small illustrative vocabulary:

Token Example ID
the 12
model 37
learns 21

Token IDs are not numerical measurements of meaning. They are indices used to look up vectors in an embedding table. The IDs above are only an example; your tokenizer's vocabulary will determine the actual IDs.

2. Token and positional embeddings

Token plus positional embeddings

The embedding table maps every token ID to a learned vector. But a sequence also has order: “the model” and “model the” contain the same words in different positions.

A positional embedding provides information about where a token appears. Token and positional embeddings are combined to form the initial representation:

Input representation = token embedding + positional embedding

This gives the transformer information about both the token and its position in the sequence.

3. Causal self-attention

Self-attention lets each token build a context-aware representation by gathering information from other tokens. A GPT-style model uses causal attention: a position may attend to itself and earlier positions, but not future tokens.

Causal attention mask

In this diagram, green cells are allowed and gray cells are blocked. The third token can use tokens one, two, and three—but cannot see token four. This prevents the model from “cheating” by looking at the answer while learning to predict the next token.

4. Query, Key, and Value

Each input vector is transformed into three learned representations:

  • Query (Q): what the current position is looking for
  • Key (K): what each position can be matched against
  • Value (V): the information each position contributes

The scaled dot-product attention operation is:

[ \operatorname{Attention}(Q,K,V) =\operatorname{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}+M\right)V ]

Here, (d_k) is the key dimension and (M) is the causal mask. Query-key similarity produces attention scores. The mask blocks future positions, softmax converts the scores into weights, and those weights combine the value vectors.

Query Key Value attention flow

5. Multi-head attention

Multi-head attention

One attention head learns one set of contextual relationships. Multi-head attention runs several attention heads in parallel, each with its own learned projections.

The heads' outputs are concatenated and projected into a shared representation. Different heads can learn different patterns—for example, relationships between nearby tokens or dependencies across a longer context.

6. Feed-forward network and transformer block

Transformer block

After attention, a feed-forward network applies learned transformations to each position's representation. A transformer block combines attention and feed-forward processing, along with residual connections and normalization where implemented.

Stacking transformer blocks lets the model refine its contextual representations layer by layer.

7. Next-token training

Next-token training loop

GPT-style language modeling is trained as a next-token prediction task. The target sequence is shifted by one token relative to the input:

Context provided to the model Target token
the model
the model learns
the model learns from

The model produces logits—scores for tokens in its vocabulary. Cross-entropy loss compares the predictions with the correct next tokens. An optimizer such as AdamW updates the model parameters to reduce this loss.

Training text, model size, and training duration influence what the model learns. A small corpus is useful for demonstrating the process, but does not give a model broad language knowledge.

8. Autoregressive generation

After training, the model can generate text from a prompt. It predicts the next token, appends the selected token to the context, and predicts again.

Autoregressive generation

For example, a prompt might be:

The model

The model may continue with a sequence such as:

The model learns from text

This is an illustrative example, not a guaranteed output. The actual continuation depends on learned weights, prompt, and sampling settings.

Temperature adjusts the sampling distribution:

  • Lower temperature generally makes token selection more focused.
  • Higher temperature generally makes token selection more varied.

Generation is iterative: each generated token becomes part of the context for the next prediction.

Installation

Install TokenCraft from PyPI:

pip install tokencraft-gpt

The distribution name used by pip is tokencraft-gpt. The Python import name is tokencraft.

How to use TokenCraft

1. Import the package

import tokencraft
from tokencraft import GPT, Tokenizer

print(tokencraft.__version__)

2. Inspect the available API

The model and tokenizer classes are exposed by the package. Use Python's built-in help to inspect the constructor arguments and methods available in your installed version:

help(GPT)
help(Tokenizer)

3. Prepare a text dataset

Create a plain-text file containing the material you want the model to learn from. For example, save this as data.txt:

A language model learns patterns from text.
A transformer uses attention to process sequences.
TokenCraft demonstrates next-token prediction.

For meaningful experiments, use a larger and appropriately prepared dataset. Keep track of the data you train on and ensure you have permission to use it.

4. Train the model

A typical training workflow is:

  1. Load the training text.
  2. Build the tokenizer vocabulary and encode the text into token IDs.
  3. Create input sequences and next-token targets.
  4. Initialize the GPT model using the model configuration.
  5. Run the training loop with a next-token loss such as cross-entropy.
  6. Update the parameters with an optimizer such as AdamW.
  7. Save the trained model parameters for later use.

TokenCraft's exact constructor arguments and training options are version-specific. Inspect GPT and Tokenizer with help() and refer to the project's training example or scripts for the precise commands.

5. Generate text from a trained model

The generation workflow is:

  1. Load the trained model parameters.
  2. Encode a prompt with the same tokenizer used during training.
  3. Pass the prompt tokens to the model.
  4. Select a next token from the model's output.
  5. Append the token and repeat until the desired length or stopping condition is reached.
  6. Decode the generated token IDs back into text.

The model must be trained first—or loaded from a compatible checkpoint—before its output can reflect learned patterns. The package does not automatically download pretrained weights.

Quick API check

Use this small snippet to confirm that the package installed and its main classes are importable:

import tokencraft

print("TokenCraft version:", tokencraft.__version__)
print("GPT class:", tokencraft.GPT)
print("Tokenizer class:", tokencraft.Tokenizer)

Project structure

The package is organized around the main GPT components:

tokencraft/
├── __init__.py
├── tokenizer.py
└── model/
    ├── __init__.py
    ├── gpt.py
    ├── embedding.py
    ├── positional_embedding.py
    ├── input_embedding.py
    ├── self_attention.py
    ├── multi_head_attention.py
    ├── feed_forward.py
    └── transformer_block.py

The source repository may also contain training scripts, generation examples, and sample data.

Limitations

TokenCraft is intended for education and experimentation:

  • It is not a pretrained production chatbot.
  • Generated text depends on the training data and model configuration.
  • Small models and small datasets may generate repetitive or incoherent text.
  • It is not designed to match the capabilities of large-scale language models.

Inspiration and acknowledgements

TokenCraft's educational approach is inspired by transformer literature and visual explanations such as Jay Alammar's The Illustrated Transformer. This README uses original diagrams and explanations tailored to a decoder-only GPT-style model; it does not reproduce that article's illustrations.

Author and license

Author: Ranjeet Epili

License: MIT

Metadata

Release files for tokencraft-gpt 0.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for tokencraft-gpt 0.1.1
File Size Uploaded Provenance Actions
tokencraft_gpt-0.1.1.tar.gz 132.5 kB Without provenance Details for tokencraft_gpt-0.1.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for tokencraft-gpt 0.1.1
File Interpreter ABI Platform Actions
tokencraft_gpt-0.1.1-py3-none-any.whl Python 3 none any Details for tokencraft_gpt-0.1.1-py3-none-any.whl

Total release size: 144.2 kB

Release files / tokencraft_gpt-0.1.1.tar.gz

Download URL tokencraft_gpt-0.1.1.tar.gz
Size 132.5 kB
Tags Source
SHA-256 checksum
How to use checksums
4548ce497624a272a5e4c2e8fc4583a188a2768e6f60bf1a116a5127ba9eedf2
BLAKE2b-256 checksum
How to use checksums
67b4fa3b9b93ac342fcc964a271ff94e63751d645a4e92cfda84e15b4e55f63a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.15.1

Release files / tokencraft_gpt-0.1.1-py3-none-any.whl

Download URL tokencraft_gpt-0.1.1-py3-none-any.whl
Size 11.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
8d4fb6da794681ec9bc162acdebf2d1f027ab64fd2cc98fffe79bbfe65485313
BLAKE2b-256 checksum
How to use checksums
3e47fd13626fe9dfb92891315d7577f269a744d9d0a36de82b7538c67cb82b2c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.15.1

Security for tokencraft-gpt 0.1.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.1 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page