mlsys-lab 0.1.0

Auto-graded exercises in low-level ML systems: real C++, real CUDA-C on a software GPU, deterministic metrics

mlsys-lab

Auto-graded exercises in low-level ML systems. You write real C++, real CUDA-C and Python; a local grader runs it and checks a measured number against declarative gates.

No cloud, no account, no GPU required. Every metric is deterministic, so the same submission scores identically on any machine.

$ mlsys grade gpu-1-padding-to-remove-transpose-bank-conflicts

  max_abs_err        0.0        <= 1e-12   PASS
  smem_wave_ratio    16.5       <= 1.05    FAIL
                     ^ your tile is 32 wide, so every column read hits one bank

Why the numbers are trustworthy

Wall-clock timing is never a gate. Instead the engine models the hardware and counts what actually happened:

you write	it runs on	it measures
solve.cu	a software GPU: 32-lane warps, 128-byte transactions, 32 shared-memory banks, barriers, warp shuffles	coalescing transactions, bank-conflict waves, warp divergence, cycles
solve.cpp	the local clang++ -O2 -std=c++20, plus a modelled set-associative cache	cache misses, lines touched, and the program's own output
solve.py	the interpreter, with an instrumented probe	allocations, copies, numerical error against a computed oracle

The reference answer is always computed by the grader, never hard-coded — so a task cannot be passed by guessing the expected output.

Install

pip install mlsys-lab           # the engine and all 2052 tasks

mlsys list                      # browse the bank
mlsys grade <task-id> --file my_solution.py

The bank ships inside the package, so there is nothing to clone and nothing to download on first run. Your solutions live wherever you put them — --file takes a path, and the bank is never written to.

C++ tasks need a C++20 compiler (clang++ or g++); everything else is Python plus NumPy. CUDA tasks need no NVIDIA hardware — the .cu is executed by the simulator in src/mlsys/sim/.

VS Code

Install mlsys-lab from the Marketplace, then run mlsys-lab: Open Workspace. If the package is not installed yet, the extension offers to do it. Solutions go to ~/mlsys-lab/<task-id>/ (configurable via mlsys.workDir).

Contributing to the bank

git clone https://github.com/mikamika06/mlsys-lab
cd mlsys-lab && pip install -e .

A checkout takes precedence over the installed copy, so the tasks you are editing are the ones that get graded. $MLSYS_TASKS overrides both.

What is in the bank

Three native tracks plus the systems material:

• Deep Python — the data model, descriptors, the GIL, memory layout
• Deep C++ — ownership, moves, ABI, undefined behaviour you can observe
• CPU performance — cache blocking, layout (AoS/SoA), branches, SIMD
• GPU / CUDA — coalescing, bank conflicts, divergence, barriers, shuffles
• Numerics & tensors — stability, shapes and strides, low-precision formats
• LLM internals — attention, RoPE, GQA, the KV cache, sampling
• LLM systems — paged KV, continuous batching, parallelism, speculative decoding
• Applied — quantization, attention/KV, export & compilation, serving, memory & offload, sparsity & distillation

Concepts, explained with a number

docs/concepts/ — one page per concept, each with a measurement produced by the simulator in this repo and the command that regenerates it, then the graded exercise. Start with memory coalescing (128-byte transactions against read stride, 2 → 33) or false sharing (coherence invalidations against padding, 7,999 → 0).

What else exists

docs/LANDSCAPE.md surveys the other resources aimed at each of these areas — 141 of them, every link checked — and marks whether each one actually grades your work or only shows you code. It says where this bank is not your best option, which is most of the point of having the page.

The short version: GPU/CUDA, algorithms-from-scratch and LLM internals are well served already (GPU-Puzzles, LeetGPU, Tensara, deep-ml, CS336, LeetCUDA). Six of the fourteen areas have no auto-graded resource anywhere, and four — applied quantization, compilation & export, memory & offload, and sparsity/pruning/distillation — have nothing to practise against at all: papers, production libraries and documentation, but no exercise that tells you that you got it wrong.

Writing a task

A task is a directory under tasks/. See docs/TASK_FORMAT.md.

tasks/<id>/
  meta.json     id, title, difficulty, gates       ("native": "cpp" | "cuda")
  task.md       ## Context / ## Task / ## Example / ## What the gate checks
  check.py      grade(sol, fx) -> {metric: value}  — computes its own reference
  starter.py    the empty contract the learner starts from
  solution_ref.py

tools/verify_task.sh   <id>    # python task:  reference passes, starter fails
tools/verify_native.sh <id>    # C++ / CUDA task

Both print TASK_OK only if the reference passes the gates and the shipped starter fails them — a task that anything can pass is not a task.

Layout

src/mlsys/        the engine
  sim/            software GPU, CUDA-C front end, cache model, ABI model
  runners/        cpp.py (clang++), cuda.py (software GPU)
  scorers.py      the metrics tasks gate on
tasks/            the bank
editor/           VS Code extension
tools/            verify, queue helpers, browser preview
docs/             curriculum and research

License

MIT