mlsys-lab

Auto-graded exercises in low-level ML systems: quantization, KV cache and attention,
batching, numerics, CPU performance, CPython internals, GPU memory behaviour.
Mostly Python, with real C++ and real CUDA-C. A measured number is the gate, never a clock.

$ mlsys grade gpu-1-padding-to-remove-transpose-bank-conflicts

  Kill transpose bank conflicts with +1 padding

  ✓ max_abs_err                 0   gate <= 1.000e-09
  ✗ smem_wave_ratio          16.5   gate <= 1.05

  ● FAIL  1/2 gates

Your transpose is numerically perfect and 16.5× slower than it needs to be, because the tile is 32 floats wide and every column read lands in one shared-memory bank. Nothing was timed to find that out.

pip install mlsys-lab
mlsys start gpu-1-padding-to-remove-transpose-bank-conflicts   # writes the starter here
mlsys grade gpu-1-padding-to-remove-transpose-bank-conflicts   # measure it

No cloud, no account, no GPU. Every metric is deterministic, so the same submission scores identically on any machine.

Fourteen areas, from the CPython data model to paged attention. A survey of 135 other resources found that six of those fourteen have no automatically graded material anywhere — the survey is in RESOURCES.md, and it says where other people teach something better than this does.

Projects

A task is one function. A project is a repo: a ticket that names a symptom rather than the defect, several files to edit, and milestones graded one at a time so progress is visible before the whole thing is finished.

The last milestone of every project is the same shape: you write a regression test, we inject a fault into your own code, and your test has to catch it.

mlsys.sim.server makes this work without hardware — a deterministic model of continuous batching over a paged KV cache (reference-counted blocks, chunked prefill, prefix caching, preemption by recompute or swap) on an integer clock, so a scheduler you write scores the same on any machine.

Why the numbers are trustworthy

Wall-clock timing is never a gate. Instead the engine models the hardware and counts what actually happened:

The reference answer is always computed by the grader, never hard-coded — so a task cannot be passed by guessing the expected output.

Install

PyPI · VS Code Marketplace · Open VSX

pip install mlsys-lab                             # the engine and the whole task bank

mlsys list                                        # browse the bank
mlsys start gpu-ex-cuda-coalesced-scale           # writes ./<task-id>/solve.cu to edit
mlsys run   gpu-ex-cuda-coalesced-scale           # execute it, see what it does
mlsys grade gpu-ex-cuda-coalesced-scale           # measure it

mlsys project list                                # multi-file projects
mlsys project start p-continuous-batching-scheduler
mlsys project grade p-continuous-batching-scheduler --milestone 1

  ✗ max_abs_err          9.632   gate <= 1.000e-09
  ✓ transactions             0   gate <= 20

  ● FAIL  1/2 gates

The bank ships inside the package, so there is nothing to clone and nothing to download on first run. mlsys start copies the task's starter into a directory of yours; mlsys run and mlsys grade then find it. All three work the same for Python, C++ and CUDA tasks — the file extension follows the task. The bank itself is never written to, so your work and the tasks never mix.

run and grade answer different questions. Grading applies the gates and is the only thing that marks a task solved; running just executes the file — the script, or clang++ against the task's own driver, or the kernel on the software GPU — and shows the print, the traceback and the compiler diagnostic that a verdict deliberately hides.

C++ tasks need a C++20 compiler (clang++ or g++); everything else is Python plus NumPy. CUDA tasks need no NVIDIA hardware — the .cu is executed by the simulator in src/mlsys/sim/.

Sixteen tasks have an oracle that needs more than NumPy — scipy, ml-dtypes, mpmath, or torch. Rather than make every learner install torch, each declares what it needs and mlsys grade tells you:

pip install "mlsys-lab[extras]"     # scipy, ml-dtypes, mpmath — 15 of the 16
pip install "mlsys-lab[torch]"      # the remaining one

Five tasks use ARM NEON intrinsics and one loads Apple's Accelerate framework, so they only build on the platform they target. The verifier skips those by name rather than pretending they passed.

VS Code

Install mlsys-lab from the Marketplace, then run mlsys-lab: Open Workspace. If the package is not installed yet, the extension offers to do it. Solutions go to ~/mlsys-lab/<task-id>/ (configurable via mlsys.workDir).

Contributing to the bank

git clone https://github.com/mikamika06/mlsys-lab
cd mlsys-lab && pip install -e .

A checkout takes precedence over the installed copy, so the tasks you are editing are the ones that get graded. $MLSYS_TASKS overrides both.

What is in the bank

Three native tracks plus the systems material:

• Deep Python — the data model, descriptors, the GIL, memory layout
• Deep C++ — ownership, moves, ABI, undefined behaviour you can observe
• CPU performance — cache blocking, layout (AoS/SoA), branches, SIMD
• GPU / CUDA — coalescing, bank conflicts, divergence, barriers, shuffles
• Numerics & tensors — stability, shapes and strides, low-precision formats
• LLM internals — attention, RoPE, GQA, the KV cache, sampling
• LLM systems — paged KV, continuous batching, parallelism, speculative decoding
• Applied — quantization, attention/KV, export & compilation, serving, memory & offload, sparsity & distillation

Concepts, explained with a number

concepts/ — 37 pages, one per concept, each with a measurement produced by this repo and the command that regenerates it, then the graded exercise. Two checks run in CI: one for structure and links, one that executes each page's own snippet and fails if it prints anything the page does not say.

Start with memory coalescing (128-byte transactions against read stride, 2 → 33), false sharing (coherence invalidations against padding, 7,999 → 0), or gguf vs safetensors (bytes per weight actual rather than the nominal 4 bits).

Where else to practise this

RESOURCES.md — 135 other places to practise the same material, in one list, grouped by whether they check your work. Of the 135, 19 give you an automatic verdict and the rest do not; each entry says what it is, what it costs, and whether it is still alive.

LANDSCAPE.md is the same 135 organised by this bank's own areas, with a verdict on each: it surveys the other resources aimed at each of these areas — 141 of them, every link checked — and marks whether each one actually grades your work or only shows you code. It says where this bank is not your best option, which is most of the point of having the page.

The short version: GPU/CUDA, algorithms-from-scratch and LLM internals are well served already (GPU-Puzzles, LeetGPU, Tensara, deep-ml, CS336, LeetCUDA). Six of the fourteen areas have no auto-graded resource anywhere, and four — applied quantization, compilation & export, memory & offload, and sparsity/pruning/distillation — have nothing to practise against at all: papers, production libraries and documentation, but no exercise that tells you that you got it wrong.

Writing a task

A task is a directory under tasks/. See TASK_FORMAT.md.

tasks/<id>/
  meta.json     id, title, difficulty, gates       ("native": "cpp" | "cuda")
  task.md       ## Context / ## Task / ## Example / ## What the gate checks
  check.py      grade(sol, fx) -> {metric: value}  — computes its own reference
  starter.py    the empty contract the learner starts from
  solution_ref.py

tools/verify_task.sh   <id>    # python task:  reference passes, starter fails
tools/verify_native.sh <id>    # C++ / CUDA task

Both print TASK_OK only if the reference passes the gates and the shipped starter fails them — a task that anything can pass is not a task.

Layout

src/mlsys/        the engine
  sim/            software GPU, CUDA-C front end, cache model, ABI model
  runners/        cpp.py (clang++), cuda.py (software GPU)
  scorers.py      the metrics tasks gate on
  task_list2.json the curriculum: 14 areas, 291 sub-tracks
tasks/            the bank
concepts/         one page per concept, each with a measured number
editor/           VS Code extension
tools/            verify, queue helpers, browser preview

License

MIT