emmy-ml 0.3.21

Compile → Benchmark → Deploy any LLM on any GPU. Optimized compiler, LLM benchmarking, and deployment stack. Optimize inference via kernel fusion, autotuning, and advanced scheduling. See the blog post: Outperforming vLLM (cuBLAS and FlashAttention) on Gemma4-12B.

Install

pip install emmy-ml          # the CLI, with the recommended recipes bundled
emmy --version

The compiler needs its own extra (pip install "emmy-ml[compile]" — torch, transformers, cppyy). The wheel carries the runtime extension that launches its kernels (emmy.emmy_runtime, built from crates/emmy-runtime-py) for Linux x86_64; on another platform the sdist builds it when a Rust toolchain is present and installs pure without one, which keeps every command that needs no GPU working. To hack on emmy itself, clone instead:

git clone https://github.com/cloudrift-ai/emmy.git
cd emmy && make setup

Kernels compile with the CUDA toolkit's nvcc and launch through the Rust runtime (make setup builds it into the venv, so a Rust toolchain is a prerequisite). On a pre-Turing GPU (V100 sm_70, P100 sm_60) the toolkit must be a CUDA 12 release: CUDA 13 dropped every architecture below sm_75.

The torch wheel is a separate pre-Turing trap: the default +cu130 build carries no sm_70 kernels at all, so the reference side of every accuracy check and every --bench-backends eager,tcompile comparison dies with no kernel image is available for execution on the device. The 2.13.0+cu126 build includes Volta kernels:

pip install --force-reinstall "torch==2.13.0+cu126" --index-url https://download.pytorch.org/whl/cu126

Ask for the +cu126 local version explicitly — a bare torch==2.13.0 matches the already-installed +cu130 wheel and pip reports the requirement satisfied without changing anything.

Commands that only drive remote hardware (deploy, bench, vm, teardown, …) never touch the local toolchain and keep working on such a host.

Compile

A hackable PyTorch → Graph IR → CUDA compiler. Trace any nn.Module, fuse it into one kernel, run it, and inspect the emitted CUDA. See the blog post: A Principled ML Compiler Stack in 5,000 Lines of Python.

# Compile a single operation
emmy compile -c "nn.RMSNorm(2048)(torch.randn(1,32,2048))"
# Benchmark kernel on a local GPU
emmy run --bench --profile -c "torch.nn.Softmax(dim=-1)(torch.randn(1, 28, 2048, 2048))"
# Tune kernels with their weights fetched from memory; cold evidence stays separate from hot evidence
emmy run --bench --cold-cache --tune 20 -c "nn.Linear(4096,4096,bias=False)(torch.randn(1,4096))"
# Trace a dynamic model layer into an unmeasured working golden
emmy trace Qwen/Qwen3-0.6B --layer 0 --dynamic seq_len@x:1 -o _tune/qwen3/working.json
# Bench every realization and record the measurements as deploy evidence (add --realization NAME to select one)
emmy run --golden _tune/qwen3/working.json --bench --strict --json _tune/qwen3/results
# Record the kernel set the greedy pick took for one realization as measured rows a strict compile picks again
emmy run --golden _tune/qwen3/working.json --realization mlp.layer0 --bench --record-greedy
# Capture one symbolic serving inventory with every release realization, then validate it on the pinned GPU
emmy trace /models/gemma --serving-twins --serving-config docker/vllm-emmy-serve/models/gemma-4-12b-it.env \
  -o _tune/gemma/working.json
emmy eval golden --golden recipes/gemma-4-12B-it/golden/rtx5090_sm120.json \
  --serving-config docker/vllm-emmy-serve/models/gemma-4-12b-it.env

Fast math is enabled by default. EMMY_FAST_MATH=0 disables NVCC fast math and the compiler's precision-trading optimizations; individual precision pins override that umbrella. For a separate-rounding accuracy diagnostic, add --nvcc-flags=--fmad=false. That custom flag disables multiply-add contraction without changing the other policies.

Layer-norm-style reduction (two reductions, broadcast subtract, elementwise chain) fused into single kernel:

emmy compile -c "
class LN(torch.nn.Module):
    def forward(self, x):
        m = x.mean(-1, keepdim=True)
        v = ((x - m) ** 2).mean(-1, keepdim=True)
        return (x - m) * torch.rsqrt(v + 1e-6)
LN()(torch.randn(64, 2048))"

Any nn.Module can also run on emmy through torch.compile, under torch.no_grad(): torch.compile(model, backend="emmy", dynamic=False, options={...}). emmy/dynamo.py lists the options.

Principled compilation stack with six IR stages, each printable on demand via --ir <stage>:

1. Torch IR — captures the FX graph as a 1:1 mirror of PyTorch's op set (rmsnorm, linear, softmax, ...)
2. Tensor IR — decomposes Torch ops into generic elementwise, reduction, indexing, and value-conversion primitives
3. Loop IR — lifts each primitive to a LoopOp and fuses
4. Tile IR — schedules kernels onto GPU
5. Kernel IR — materializes the schedule into framework-agnostic hardware primitives
6. CUDA — optimized CUDA code ready for nvcc

Readable Schedule: emmy compile -c "nn.RMSNorm(2048)(torch.randn(1,32,2048))" --ir tile

kernel k_rms_norm_reduce  inputs: rms_norm_mean_count, rms_norm_eps, x, p_weight  outputs: rms_norm
    in0 = load rms_norm_mean_count[0]
    in1 = load rms_norm_eps[0]
    Tile(axes=(a0:256=THREAD, a1:32=BLOCK)):
        x_smem = Stage(x, origin=(0, a1, 0), slab=(a2:2048@2)) async
        p_weight_smem = Stage(p_weight, origin=(0), slab=(a3:2048@0)) async
        StridedLoop(a2 = a0; < 2048; += 256):  # reduce
            in2 = load x_smem[a2]
            v0 = multiply(in2, in2)
            acc0 <- add(acc0, v0)
        v1 = divide(acc0, in0)
        v2 = add(v1, in1)
        v3 = rsqrt(v2)
        StridedLoop(a3 = a0; < 2048; += 256):  # free
            in3 = load x_smem[a3]
            in4 = load p_weight_smem[a3]
            v4 = multiply(in3, v3)
            v5 = multiply(v4, in4)
            rms_norm[0, a1, a3] = v5

Optimized CUDA kernel: emmy compile -c "nn.RMSNorm(2048)(torch.randn(1,32,2048))" --ir cuda

extern "C" __global__
__launch_bounds__(256) void k_rms_norm_reduce(const float* x, const float* p_weight, float* rms_norm) {
    float in0 = 2048.0f;
    float in1 = 1e-06f;
    {
        int a1 = blockIdx.x;
        int a0 = threadIdx.x;
        float acc0 = 0.0f;
        __syncthreads();
        __shared__ float x_smem[2048];
        for (int x_smem_flat = a0; x_smem_flat < 2048; x_smem_flat += 256) {
            {
                unsigned int _smem_addr = __cvta_generic_to_shared(&x_smem[x_smem_flat]);
                asm volatile("cp.async.ca.shared.global [%0], [%1], 4;\n"
                             :: "r"(_smem_addr), "l"(&x[a1 * 2048 + x_smem_flat])
                             : "memory");
            }
        }
        asm volatile("cp.async.commit_group;\n" ::: "memory");
        asm volatile("cp.async.wait_group 0;\n" ::: "memory");
        __syncthreads();
        __shared__ float p_weight_smem[2048];
        for (int p_weight_smem_flat = a0; p_weight_smem_flat < 2048; p_weight_smem_flat += 256) {
            {
                unsigned int _smem_addr = __cvta_generic_to_shared(&p_weight_smem[p_weight_smem_flat]);
                asm volatile("cp.async.ca.shared.global [%0], [%1], 4;\n"
                             :: "r"(_smem_addr), "l"(&p_weight[p_weight_smem_flat])
                             : "memory");
            }
        }
        asm volatile("cp.async.commit_group;\n" ::: "memory");
        asm volatile("cp.async.wait_group 0;\n" ::: "memory");
        __syncthreads();
        for (int a2 = a0; a2 < 2048; a2 += 256) {
            float in2 = x_smem[a2];
            float v0 = in2 * in2;
            acc0 += v0;
        }
        __shared__ float acc0_smem[256];
        acc0_smem[a0] = acc0;
        __syncthreads();
        for (int s = 128; s > 0; s >>= 1) {
            if (a0 < s) {
                acc0_smem[a0] = acc0_smem[a0] + acc0_smem[a0 + s];
            }
            __syncthreads();
        }
        __syncthreads();
        float acc0_b = acc0_smem[0];
        float v1 = acc0_b / in0;
        float v2 = v1 + in1;
        float v3 = rsqrtf(v2);
        for (int a3 = a0; a3 < 2048; a3 += 256) {
            float in3 = x_smem[a3];
            float in4 = p_weight_smem[a3];
            float v4 = in3 * v3;
            float v5 = v4 * in4;
            rms_norm[a1 * 2048 + a3] = v5;
        }
    }
}

Fit the priors

Two priors ship in the repo, both small CatBoost rankers stored as one JSON file each, fit GPU-free on the repository goldens and consulted by a compile only where nothing was measured: the schedule prior (emmy/compiler/pipeline/search/prior/weights/schedule.json) ranks a kernel's schedule rows, and the placement prior (weights/placement.json beside it) ranks the arms of a kernel-set fork — keep the kernel whole, cut one of its seams, or split its reduction across CTAs.

# 1. Load every repository golden — the hardware goldens and each maintained recipe's — into a DB of its own
emmy db import --db _data/dataset.db --fresh --repository
# 2. Export one dataset per space: every golden pool enumerated and featurized, or every kernel-set fork's arms
emmy db export --db _data/dataset.db _data/schedule --space schedule
emmy db export --db _data/dataset.db _data/placement --space placement
# 3. Fit each prior from its dataset, rewriting the checked-in weights file
emmy fit _data/schedule emmy/compiler/pipeline/search/prior/weights/schedule.json
emmy fit _data/placement emmy/compiler/pipeline/search/prior/weights/placement.json
# 4. Where each golden decision ranks under the shipped weights, and what the greedy would pick with no measurement
emmy eval prior _data/schedule
emmy eval prior _data/placement

emmy fit also writes a metrics file under _tune/fits/<timestamp>/; two fits are compared by diffing their metrics files.

To compare a candidate with the shipped schedule prior without replacing it:

emmy fit _data/schedule _data/schedule-candidate.json --folds 0
emmy eval prior _data/schedule \
  --offline-file emmy/compiler/pipeline/search/prior/weights/schedule.json \
  --compare-to _data/schedule-candidate.json --json _data/schedule-comparison.json

The JSON report includes both rank summaries and the comparison decision.

Nightly refresh owns routine refits of the schedule and placement priors from the repository goldens. A PR that changes a hardware golden refits in the PR; one that adds a recipe golden leaves the weights to nightly while the reproduction gate below stays green. Nightly refresh refits each prior independently and compares the shipped and fitted weights on the same dataset and commits a candidate directly to main only when a GPU, tier and pool-size group's median golden rank falls by at least 5% and no group's median rises or loses coverage. The nightly summary in #emmy-robots carries each result, and how the weights left on main pick: how many pools the prior re-decides as their golden did, and the regret of the picks a golden row measured (eval prior --json). Golden rank measures where a verified row landed, not the latency of a wrong pick; the reproduction gate below still runs before a candidate is committed. The same run lists every golden and realization corpus row behind torch.compile and the corpus's expected-failure cases (emmy golden list).

The reproduction gate. tests/compiler/pipeline/search/prior/test_reproduction.py holds the shipped priors to every repository golden, with no measurement in scope, at one tolerance over each corpus and space: a kernel-set fork's recorded arm is the prior's pick, and a recorded schedule row sits within the better half of a draw of its pool as the prior orders it — a baseline that tightens as the schedule prior improves (the median golden sits at 4 percent). Every repository golden runs in make test, its pools in slices of 16 so the work spreads over the workers: one node is one slice of one golden in one space, and holds the tolerance over that slice; the schedule half draws as many rows per pool as a greedy compile in the suite does (EMMY_POOL_DRAW, 512 there). A red node names the rows the prior cannot reproduce, and the gate stays green on every PR. Hardware goldens are held strictly: a PR that changes one refits the priors so they reproduce it. A recipe golden the shipped priors do not reproduce is refit for, or its recipe is tagged prior-pending: the gate skips that golden, which stays evidence and training data, until a refit reproduces it and drops the tag. Never lower the tolerance.

Benchmark

emmy bench experiments/gemma-4-12B/*                                    # All Gemma experiments
emmy bench experiments/gemma-4-12B/gsm8k_mtp_rtx5090                    # A single experiment
emmy bench experiments/gemma-4-12B/* --filter "deploy.gpu=*5090*"       # Subset
emmy bench experiments/gemma-4-12B/* --gpu-concurrency 4                # Parallel VMs per GPU
emmy bench experiments/gemma-4-12B/* --local                            # On this machine
emmy bench experiments/gemma-4-12B/* --ssh user@host1 --ssh user@host2  # Pre-allocated hosts

Each real run creates a timestamped raw-results directory and writes one system-only YAML experiment record per matrix row. Use $run-experiment to run or customize an experiment, replace the platform's LFS-backed results_<gpu-short>x<gpu-count>.tar.gz, verify its included records, update the shared RESULTS.md interpretation, and commit the durable platform snapshot. Records remain inside the archive rather than as top-level experiment files. Other platform snapshots remain unchanged. The runner and experiment code never interpret measurements; the skill reviews the raw evidence. The timestamped directory is ignored and may be deleted after its archive has been extracted or byte-checked against the raw files.

Deploy

# Remote server via SSH
emmy deploy ssh --recipe recipes/gemma-4-12B-it --ssh user@host

# Remote server via SSH, several models on its GPUs (a plan file instead of --recipe)
emmy deploy ssh --plan plan.json --ssh user@host

# Local Docker Compose
emmy deploy local --recipe recipes/gemma-4-12B-it

# Cloud (auto-provisions a VM)
emmy deploy cloud --recipe recipes/gemma-4-12B-it --gpu "NVIDIA H200 141GB" --gpu-count 8

# Cloud, several models on one VM (a plan file instead of --recipe)
emmy deploy cloud --plan plan.json --result-json out.json --lease lease.json --owner relay/deployment-42

--recipe also takes a bare recipe name (--recipe gemma-4-12B-it). An editable install resolves it from the live checkout; a wheel install resolves it from the packaged catalog. Emmy copies the recipe into the current directory first because deploy writes its compose file next to it and bench its timestamped run directory. A path that exists always wins, so an edited working copy is never overwritten.

A plan names the VM to rent (gpu, gpu_count) and an ordered list of models, each with a recipe, its gpu_memory_utilization and the gpu_device_ids it is pinned to. Model i runs as its own container on host port 8000 + i; models sharing a GPU start one after another. --result-json writes one endpoint per model on success, and --lease persists the instance id the moment the VM is rented. The deploy command reference has the details.

Publish a serving image

The serving recipe pins the canonical immutable image reference. Validate the local image, its provenance labels, and the registry collision before requesting publication approval; only then log in and perform the push:

emmy publish recipes/DeepSeek-V4-Flash-0731 --dry-run
emmy publish recipes/DeepSeek-V4-Flash-0731 --source-image local-baked-image --yes

Published references use cloudriftai/<runtime-family>-<model-slug>:<runtime-version>-<source-sha>; see the prebuilt-serving-image architecture for the release gates and labels.

Serve (compiled embeddings via vLLM)

# vLLM's OpenAI shell (/v1/embeddings, tokenizer, scheduler, pooler) over emmy-compiled kernels
emmy serve Qwen/Qwen3-Embedding-0.6B

curl localhost:8000/v1/embeddings -H 'Content-Type: application/json' \
  -d '{"model":"Qwen/Qwen3-Embedding-0.6B","input":"Hello"}'

# One-shot benchmark (vllm bench serve against the started server), and the raw-vLLM baseline
emmy serve Qwen/Qwen3-Embedding-0.6B --bench --random-input-len 32
emmy serve Qwen/Qwen3-Embedding-0.6B --bench --random-input-len 32 --stock

For the generative vLLM runner, pass --runner generate --dtype bfloat16 to keep a dense model's compiled trunk in BF16. The generative default remains FP16.

Experimental native generation

A dense Qwen3 or Qwen3.5 text model, FP16 or FP8 (the FP8 trunk stays coded, weight-only), can be prepared as a standalone artifact and run through the Rust cached-generation loop. This single-request path and experimental native HTTP adapter support greedy or seeded temperature/top-p sampling and optional CUDA graphs. Output logits, residuals, and attention/rotary intermediates use FP32. Prefill uses fixed-width chunks; vLLM remains the serving default. See the native generation contract for preparation, commands, limitations, and qualification.

make native-dist  # install the archive's matching binaries on PATH
emmy serve Qwen/Qwen3-0.6B --runner generate --native --revision REVISION

Native serving exposes text and chat completions with streaming, stop strings, usage, and one active request.

Recipe

# Automatically inspect live source recipes for an editable install, or the
# runnable recipes bundled in an installed wheel.
emmy recipe list --json

# Count one lifecycle group in automation.
emmy recipe query --filter 'tags contains "maintained"' --json

# Select the hottest available onboarding shell that has not failed. Referencing deployment.* expands each recipe into
# deployment rows; CloudRift availability is resolved only because this query uses it.
emmy recipe query \
  --filter 'lifecycle == "onboarding"' \
  --filter 'tags not contains "onboarding-failed"' \
  --filter 'deployment.availability.cloudrift == true' \
  --sort 'heat desc nulls-last' \
  --sort 'model_id asc' \
  --limit 1 --json

# Select a hot runnable recipe that has no Emmy serving variant yet.
emmy recipe query \
  --filter 'lifecycle in ["maintained", "best-effort"]' \
  --filter 'runnable == true' \
  --filter 'heat >= 70' \
  --filter 'emmy_serving == false' \
  --sort 'heat desc' \
  --json

# Check one exact external candidate, including a model without a recipe yet.
emmy recipe query --candidate org/model-name "NVIDIA H200 141GB" 1 \
  --filter 'deployment.availability.cloudrift == true' --json

# Create an untested onboarding shell with one to three proposed deployments.
emmy recipe create org/model-name --rationale "Why this model should be onboarded." \
  --deployment "NVIDIA H200 141GB" 1 --deployment "NVIDIA B200" 1

recipe list --json is a versioned machine interface (schema version 2). It returns an object with schema_version and recipes; each recipe carries its directory name, model ID, task, lifecycle-aware runnable state, heat score, and matrix-expanded deployments, each with its GPU, GPU count, GPU memory fraction and effective context length. Two entries that differ only by fraction are both listed: that is how a recipe declares it may share its GPU. Consumers must reject unknown schema versions. Fields may be added to a schema version, but existing fields are not removed or redefined. Emmy always detects its installation: an editable checkout uses its live top-level recipes/, while a regular wheel uses its packaged runnable recipe bundle. recipe query --json returns a separate versioned rows interface for generic predicates and stable sort keys. Its optional --candidate MODEL GPU COUNT source hydrates lifecycle metadata from an existing recipe or represents a new model as onboarding work. CloudRift-derived fields require CLOUDRIFT_API_KEY; team-access checks also require CLOUDRIFT_TEAM_ID.

tags:
  - maintained

model:
  huggingface: "org/model-name"
  rationale: "Why this model belongs at its current lifecycle level."
  heat: 75

engine:
  llm:
    tensor_parallel_size: 8
    gpu_memory_utilization: 0.9
    context_length: 16384
    max_concurrent_requests: 512
    vllm:
      image: "vllm/vllm-openai:v0.23.0"
      extra_args: "--kv-cache-dtype fp8"

benchmark:
  max_concurrency: 128
  num_prompts: 256
  random_input_len: 8000
  random_output_len: 8000

# Cross-product: 3 GPUs × 2 concurrency configs = 6 variants
matrices:
  cross:
    deploy.gpu_count: 1
    deploy.gpu:
      - "NVIDIA GeForce RTX 5090"
      - "NVIDIA H100 80GB"
      - "NVIDIA H200 141GB"
    zip:
      engine.llm.max_concurrent_requests: [128, 512]
      benchmark.max_concurrency: [128, 512]

For one pinned LoRA adapter on vLLM, set engine.llm.vllm.lora_adapter with its name, Hugging Face repository, 40-character revision, and rank. Emmy downloads it beside the base checkpoint and enables it in vLLM. Set model.request_name to the base model or adapter name when a smoke test or benchmark should target a particular one; with an adapter and no override, the request targets the adapter. The benchmark still uses the base tokenizer. This requires a regular vLLM image with a writable model cache. The V100 LoRA experiment is a tested example. The experimental emmy serve --runner generate path also accepts a selectable rank-8 LoRA for plain FP16 Llama layers when its serving golden covers the configured decode and prefill widths. Its V100 behavior and speed are recorded in the same experiment's results.

Discovery keeps ten tested recipes tagged maintained, plus any a person tagged lifecycle-locked, which discovery leaves alone, and records a current 0-100 heat score and rationale under every recipe's model block. Useful lower-priority recipes stay runnable as best-effort; technically superseded or unusable models become obsolete. Every promising new model becomes an onboarding plus untested shell with up to three proposed deployment matrix entries. Disabled recipes are not deployable or bundled. Each recipe may keep a DISCOVERY.md beside its recipe.yaml and RESULTS.md to explain its discovery decision. Discovery creates a missing note and changes an existing one only for a factual correction or substantial new evidence; small daily changes leave the note, heat, rationale, and maintained choices in place.

Canonical model goldens live beside their recipe at recipes/<model>/golden/<gpu-slug>_<compute-cap>.json, with one file per exact GPU. A model with complete compiler evidence but no serving recipe receives an onboarding/untested recipe shell before its golden is committed. Hardware goldens, including qualified programs derived from models, live under emmy/compiler/pipeline/search/golden/records/, one file per exact GPU, and train the priors alongside recipe goldens. Add missing cases to that card's file, reusing kernels already stored there. A golden holds the tune DB's tables for one card — the kernels, the kernel-set decisions taken on them and the measured rows — beside the traced programs they came from. It stores inputs only: a compile imports its rows under each kernel's identity, computed from the stored Loop IR. make test holds both kinds to the fresh lowering of their programs, with no GPU needed, so a compiler change that re-keys a recorded kernel fails the suite.

Generic workload (run any tool on the VM, pull back result files):

command:
  stage: ["scripts"]
  run: |
    nvidia-smi --query-gpu=name,memory.used --format=csv > $task_dir/result.csv
  result_files: ["result.csv"]
  timeout: 60

matrices:
  deploy.gpu: "NVIDIA GeForce RTX 5090"
  deploy.gpu_count: 1

Virtual Machine Management

# GPU-based allocation with an interrupt-safe ownership lease
emmy vm create gpu --gpu "NVIDIA H200 141GB" --gpu-count 1 --exact-gpu-count \
  --lease /tmp/emmy-vm.json --owner local-run --json
emmy vm delete lease /tmp/emmy-vm.json --owner local-run

# GCP
emmy vm create gcp --instance my-vm --zone us-central1-a --machine-type a2-highgpu-1g
emmy vm delete gcp --instance my-vm --zone us-central1-a

# CloudRift
emmy vm create cloudrift --instance-type rtx4090.1 --ssh-key ~/.ssh/id_ed25519.pub
emmy vm delete cloudrift --instance-id <id>

Development

make test      # run the whole pytest suite — takes many minutes, run it once when finishing a PR
make lint      # check code and test-duration formatting
make format    # auto-fix code and sort test durations
make wheel     # build the sdist and this host's wheel into dist/
make pypi-dist # dry-run the exact PyPI sdist + wheel build into dist/

Release

Bump version in pyproject.toml on main, then run the Publish to PyPI workflow — it takes the version from there, and refuses to run if that version is already tagged. Direct commits on main are accepted. It builds and smoke-tests the distribution, uploads to PyPI via trusted publishing, and only then creates the tag and GitHub release, so a failed upload leaves nothing behind. Publishing a GitHub release by hand works too; the tag must agree with pyproject.toml. Lint and the full test suite run in pull-request checks, independently of publication.

Pull requests run make pypi-dist in a bare Python 3.13 job. The same target installs the minimal release-build dependencies, stages the distribution tree, and builds both artifacts the publishing workflow uploads: the sdist and one wheel for the runner's platform, retagged by auditwheel to the manylinux baseline its symbols allow, since the wheel carries the runtime extension and a compiled extension makes a wheel platform-specific.

scripts/prepare_dist.py stages the tree for a distribution build: --recipes copies runnable recipe YAML plus all recipe-local model goldens into the package (make wheel runs this), and --readme rewrites this file's repo-relative links to absolute GitHub URLs, which the workflow runs because PyPI renders the README detached from the repo.

Related Projects

These repositories provide the platform, serving runtimes, and deployment services that Emmy integrates with. The CloudRift repositories are typically checked out beside Emmy, so use ../<repository-name> for local code inspection when that sibling exists, and verify the remote source when current compatibility matters. Some are internal and require CloudRift organization access.

Project Structure

• AGENTS.md — shared coding-agent instructions; CLAUDE.md imports the same guidance for Claude Code
• .agents/skills/ — canonical repository skills; .claude/skills/ exposes them through compatibility symlinks
• prompts/ — reusable agent prompts shared by repository skills and non-interactive workflows
• .opencode/ — API-agent provider, permissions, and workflow profiles
• .github/ — Pull-request checks, releases, cloud experiments, and model discovery/onboarding workflows (see ARCHITECTURE.md)
• emmy/ — Python package
  • emmy.py — CLI entrypoint
  • logging_setup.py — CLI logging configuration
  • hardware.py — GPU specs and instance type mapping
  • detect.py — GPU detection via PCI sysfs (local and remote)
  • redact.py — Secret redaction for logs and dumps
  • commands/ — CLI layer (thin argparse handlers, see ARCHITECTURE.md)
    • deploy/ — deploy local, deploy ssh, deploy cloud commands
    • bench/ — bench command
    • vm/ — vm create/delete/audit commands (GCP, CloudRift, owned leases)
    • teardown.py — teardown command
    • pull.py — pull command (download HF model)
    • trace.py — trace command (PyTorch → Graph IR)
    • compile.py — compile command (decomposition → optimization → fusion → kernel/CUDA lowering)
    • run.py — run command (compile + execute on CUDA backend, optional benchmarks)
    • inspect_graph.py — inspect command (graph summary)
  • compiler/ — PyTorch → Graph IR → CUDA compiler (see ARCHITECTURE.md)
    • graph.py — Graph, Node, Tensor, Hints container
    • pretty.py — graph dumps as Rust-shaped pseudocode (see IR-PSEUDOCODE-TORCH.md)
    • ir/ — per-dialect op definitions (torch / tensor / loop / kernel / cuda) (see ARCHITECTURE.md)
      • schedule/ — schedule interfaces, reusable choices, and classic scheduling (see ARCHITECTURE.md)
    • trace/ — PyTorch/HuggingFace → Graph IR capture (see ARCHITECTURE.md)
    • pipeline/ — rewrite engine + passes + dump hooks (see ARCHITECTURE.md)
    • backend/ — numpy / loop / CUDA execution (see ARCHITECTURE.md)
      • cuda/ — CUDA backend internals (see ARCHITECTURE.md)
  • recipe/ — Recipe loading, dataclass types, engine flag mapping (see ARCHITECTURE.md)
  • serving/ — vLLM out-of-tree embedding plugin (see ARCHITECTURE.md)
  • deploy/ — Compose generation, deploy orchestration
  • provisioning/ — Cloud provisioning, SSH transport, VM lifecycle
  • benchmark/ — Benchmark tracking, config, task enumeration, execution (see ARCHITECTURE.md)
  • planner/ — Groups benchmark tasks into execution groups for VM allocation
• recipes/ — The recommended serving configuration and per-GPU model goldens, one directory per model — what emmy deploy runs (see ARCHITECTURE.md; benchmark grids belong in experiments/)
• docker/ — Custom image builds (vllm-emmy — vLLM + the emmy plugin; vllm-emmy-serve — prebuilt per-model images: warmed cubins + baked model snapshot; 1cat-vllm-sm70 — source-pinned 1Cat-vLLM runtimes and request-time GPU caches for Volta)
• experiments/ — Benchmark parameter sweeps, self-contained recipe + committed results — what emmy bench runs (see ARCHITECTURE.md)
• kernels/ — Standalone CUDA kernel sources
• crates/emmy-runtime/ — The Rust runtime: the executor behind every launch, the standalone pack loader and cached generation (design, hosts, and qualification in ARCHITECTURE.md); crates/emmy-runtime-py/ is its in-process host, the emmy.emmy_runtime extension setuptools-rust builds into the package
• crates/emmy-server/ — Native text and HTTP adapter (see ARCHITECTURE.md)
• docs/ — Docusaurus user-docs site (getting started, benchmarking, custom configurations, deployment)
• tests/ — pytest tests (see ARCHITECTURE.md)
  • compiler/passes/ — compiler pass tests (see ARCHITECTURE.md)
  • compiler/realization/ — the realization corpus: checked-in reproducers of pinned schedules, replayed as data (see ARCHITECTURE.md)
  • perf/ — GPU perf comparison vs PyTorch (see ARCHITECTURE.md)
• scripts/ — Analysis and visualization scripts
• utils/ — Standalone utility scripts
• config.yaml — Benchmark configuration
• Makefile — Build automation
• pyproject.toml — Package metadata and tool config

Contributing

1. Fork and branch from main (e.g. feature/my-change)
2. Follow STYLE.md and per-directory ARCHITECTURE.md files
3. Add tests in tests/ (see tests/ARCHITECTURE.md)
4. While developing, run only the tests that cover your change, and open a draft PR early
5. Finish the PR once, at the end: audit the diff, update the docs, then make test && make lint (use make format to auto-fix), use .github/PULL_REQUEST_TEMPLATE.md as the guide for the PR body without editing it, and mark the PR ready for review

License

Licensed under the Apache License 2.0.