Sentinel

C++/CUDA framework for full-train causal LMs (no LoRA): CPU/OpenMP plus a device train path with packed batches, flash attention, FP16 AMP, int8 Adam / Muon, selective activation checkpointing, and a consumer-GPU ~4B offload path (FP16 GPU weights + host masters/grads + host SGD).

v0.1 ships as an installable C++ library (Sentinel::sentinel), a demo harness, examples, and a pip-installable Python package (sentinel-lm).

Note on “zero-dependency”: That claim (e.g. in the GitHub About blurb — zero-dependency C++/CUDA DL framework focused on single consumer GPUs) refers to the engine runtime itself: the C++/CUDA library links only against the CUDA toolkit (cuBLAS / cuBLASLt), OpenMP, and the C++ standard library — no cuDNN, no PyTorch/TensorFlow, no other third-party DL stacks at train/infer time. It does not mean the optional Python bindings are dependency-free: building sentinel-lm needs a Python toolchain plus build-time packages (scikit-build-core, nanobind, CMake, a CUDA-capable compiler). At Python runtime there are still no extra PyPI deps — only the same CUDA / driver / OpenMP requirements as the C++ engine.

Requirements

• OS: Windows or Linux (x86_64)
• C++20 compiler — MSVC (v145+), GCC ≥ 11, or Clang ≥ 14 — plus CMake ≥ 3.24
• CUDA Toolkit (13.x; developed against v13.3) with matching NVIDIA driver
• NVIDIA GPU — GeForce 20 / 30 / 40 / 50 (sm_75 / sm_80+sm_86 / sm_89 / sm_120). CUDA 13 floor is Turing.
• OpenMP (libgomp / LLVM OpenMP / MSVC OpenMP)
• Python ≥ 3.10 (only for the Python package)
• Data for the SERA demo only (not included): SERA-Data/sera_best_subset (HF Arrow) and/or *.jsonl

Python (pip)

Needs CUDA toolkit on the build machine (compiles the same C++/CUDA core) and at runtime (cuBLASLt DLLs / .so). No extra PyPI runtime deps.

On Windows, import sentinel auto-adds $CUDA_PATH\bin\x64 (and bin) via os.add_dll_directory so Python 3.8+ can load cublasLt64_*.dll. Set CUDA_PATH if the toolkit is not under the default Program Files\NVIDIA GPU Computing Toolkit\CUDA layout.

# From repo root (editable / local wheel)
pip install .

# Or editable while hacking bindings (needs build deps already installed):
pip install -e . --no-build-isolation

python examples/python/train_tiny.py
python examples/python/generate.py tiny_demo.snlm "the cat"

import sentinel as S

tok = S.BPETokenizer()
tok.train(["the cat sat on the mat", "cuda packs batches"], vocab_size=256)
data = S.LanguageModelDataset.build(["the cat sat on the mat"], tok, maximum_token_count=48)

model = S.LanguageModel(tok.vocab_size, embedding_dim=64, maximum_position_count=64, learning_rate=3e-3)
if S.cuda_available():
    model.enable_cuda()
    model.set_prefer_flash_attention(True)
    model.enable_cuda_train()

model.train(data, epochs=4, batch_size=4)
model.save_safetensors("toy.safetensors")
print(tok.decode(model.generate(tok.encode("the"), new_token_count=16)))

Override CUDA arches (e.g. faster local build for your GPU only):

pip install . -C cmake.define.SENTINEL_CUDA_ARCHITECTURES=native
# or a subset:
pip install . -C cmake.define.SENTINEL_CUDA_ARCHITECTURES="75;86;89;120"

Publishing wheels (GitHub → PyPI)

Workflow: .github/workflows/publish.yml. Builds sdist + cp310–cp312 wheels for Linux (manylinux_2_34, Ubuntu 22.04 / glibc ≥ 2.34) and Windows (CUDA 13 toolkit on the runner; fat binary arches). Publishes on a GitHub Release, or via workflow_dispatch with publish_to_pypi=true.

One-time setup:

1. On PyPI Trusted Publishers, add publisher for project sentinel-lm: repo + workflow publish.yml + environment pypi.
2. In GitHub → Settings → Environments, create pypi (optional protection rules).
3. Cut a release (or run the workflow manually).

Wheels do not bundle the CUDA runtime — installers still need CUDA 13 + a matching driver, same as the C++ engine.

Build & run (C++)

CMake (recommended)

Builds libsentinel (static by default), the sentinel demo harness, and C++ examples.

Linux

# CUDA on PATH, e.g. export PATH=/usr/local/cuda/bin:$PATH
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j"$(nproc)"

# Tiny end-to-end (no dataset)
./build/bin/sentinel_train_tiny tiny_demo.snlm
./build/bin/sentinel_generate tiny_demo.snlm "the cat"

# Full harness (cwd matters for SERA paths)
cd sentinel && ../build/bin/sentinel

Windows

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j
build\bin\sentinel_train_tiny.exe tiny_demo.snlm
build\bin\sentinel_generate.exe tiny_demo.snlm "the cat"
cd sentinel
..\build\bin\sentinel.exe

Useful options:

Install / consume from another CMake project

cmake --install build --prefix /path/to/prefix

find_package(Sentinel 0.1 REQUIRED)
target_link_libraries(my_app PRIVATE Sentinel::sentinel)
# #include "NeuralNet/Network/LanguageModel.hpp"

MSBuild / Visual Studio (Windows only)

Monolithic sentinel.vcxproj still builds the harness only (no separate lib). Prefer CMake for library + examples.

msbuild sentinel\sentinel.vcxproj /p:Configuration=Release /p:Platform=x64
cd sentinel
..\x64\Release\sentinel.exe

Demo harness flags (main.cpp)

Default SERA demo (when suite flags are off): Arrow corpus, ~8×768. Working directory must be sentinel/ for relative ../SERA-Data/....

Layout

Data

LanguageModelChunkSource source(path, maxChars, maxTokens, chunk, trainRatio, seed, testCap);
auto sample = source.prepareTokenizerSample(2000);
tokenizer.train(sample, vocabSize);
source.setTokenizer(&tokenizer);
source.materialize();   // one corpus+BPE pass → cached token ids
model.train(source, epochs, logEvery, batchSize, gradAccum);

• JSONL (.jsonl) or HF Arrow (.arrow / save_to_disk dir with data-*.arrow; cache-* skipped)
• Path auto-detect via createTextRowReader — no Apache Arrow C++ dependency (custom IPC stream reader)
• Progress logs for sample / BPE / materialize (SmokeLog::progress)

Training

LanguageModel model(vocab, embed, maxPos, Adam(0.001f), blocks, heads);
model.enableCuda();
model.enableCudaTrain();  // pack budget, AMP, int8 Adam, ckpt=Off (peak tok/s)
model.setCudaPreferFlashAttention(true);
model.train(source, epochs, 1, batchSize, gradAccum);

model.setActivationCheckpointMode(ActivationCheckpointMode::Selective); // or Full / Off
model.saveCheckpoint("run.snlm", true);
model.loadCheckpoint("run.snlm");
model.saveSafeTensors("run.safetensors");   // weights only, HF-compatible container
model.loadSafeTensors("run.safetensors");   // or loadCheckpoint("*.safetensors")

Checkpoint files:

• .snlm — native SNLM (weights + optional Adam/Muon)
• .safetensors — zero-dep F32 safetensors export (token_embedding.weight, blocks.{i}.attn.*, ffn.*, final_norm.weight, lm_head.*); metadata carries arch dims / tie_embedding

Measured throughput (RTX 5070 Ti 16 GB)

Numbers below are fresh synthetic packed-train probes (probeCudaPackedTrainTokensPerSecond, seq=256) unless noted. Desktop WDDM / pack budget matter — re-measure after big train-path changes (runMuonThroughputProbe, runSmallSuite).

~60M — 8×768 (speed path)

~97M — 12×768 (consumer VRAM proof)

runScale100M — full multi-epoch train on sera_scale.jsonl (tied embed, no LoRA). Headline probe is Adam with ckpt=off (act scratch retained; graphs on).

Adam vs Muon × checkpoint (same 12×768 shape)

Fresh one-shot models (runMuonThroughputProbe), FP16 AMP + int8 Adam + flash:

Muon adds Newton–Schulz cost on hidden weights; comparing a Muon probe to an Adam README number looks like a 2× “regression” even when Adam is unchanged.

Full SERA epoch numbers still need sera_scale.jsonl + runScale100M=true.

~4B — FP16 GPU + host SGD (fits 16 GB)

runScale4BTrainStep — one packed train step (not multi-epoch):

Target ≥600 tok/s full-step is not yet reached on this offload+Full-ckpt path.

Smaller smoke: CudaLanguageModel::runConsumerVramDemo() (~8k/256/4L).

Verify

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build -j
./build/bin/sentinel_train_tiny
./build/bin/sentinel_generate tiny_demo.snlm "the cat"
# Optional: in main.cpp set runSmallSuite=true, then:
# cd sentinel && ../build/bin/sentinel

Expect tiny examples to run without external data. Full suite: smokes/parities OK, speed 768 ~18–20k tok/s, scale-100M probe ~25k (Adam ckpt=off, WMMA flash).

FlashAttention headDim=64 / tiles 64×64 uses CUDA WMMA Tensor Cores (toolkit only — no cuDNN). Toggle runWmmaFaVerify in main.cpp for a quick parity + probe.