Compile → Benchmark → Deploy. Emmy compiles PyTorch models into CUDA kernels, benchmarks LLM inference, and deploys models on local or cloud GPUs with vLLM and SGLang.
Compile · Tune · Deploy · Benchmark · Serve · Recipes
See Emmy in action: Outperforming vLLM on Gemma4-12B.
Install
Requires Python 3.12 or newer.
pip install emmy-ml
emmy --version
The package includes ready-to-use recipes. Remote deployment and benchmarking do not require a local GPU or CUDA
installation. For gated Hugging Face models, accept the model's license and set HF_TOKEN.
Compile
Compile a PyTorch operation, compare its speed with PyTorch, or inspect the generated CUDA:
pip install "emmy-ml[compile]"
# Compile an operation
emmy compile -c "nn.RMSNorm(2048)(torch.randn(1,32,2048))"
# Benchmark against eager PyTorch and torch.compile
emmy run --bench -c "nn.Softmax(dim=-1)(torch.randn(1,28,2048,2048))"
# Inspect the generated CUDA
emmy compile -c "nn.RMSNorm(2048)(torch.randn(1,32,2048))" --ir cuda
Running compiled kernels requires an NVIDIA GPU and the CUDA toolkit (nvcc); source builds also need Rust.
For GPUs older than Turing, use CUDA 12 and a PyTorch build that supports your GPU. Fast math is enabled by default;
set EMMY_FAST_MATH=0 to disable it.
You can also compile a custom module:
emmy compile -c "
class LN(nn.Module):
def forward(self, x):
m = x.mean(-1, keepdim=True)
v = ((x - m) ** 2).mean(-1, keepdim=True)
return (x - m) * torch.rsqrt(v + 1e-6)
LN()(torch.randn(64, 2048))"
For Python integration, use torch.compile(model, backend="emmy", dynamic=False) under torch.no_grad().
Read the compiler guide or the
compiler walkthrough for more.
Lowered kernel example
Readable Schedule: emmy compile -c "nn.RMSNorm(2048)(torch.randn(1,32,2048))" --ir tile
kernel k_rms_norm_reduce inputs: rms_norm_mean_count, rms_norm_eps, x, p_weight outputs: rms_norm
in0 = load rms_norm_mean_count[0]
in1 = load rms_norm_eps[0]
Tile(axes=(a0:256=THREAD, a1:32=BLOCK)):
x_smem = Stage(x, origin=(0, a1, 0), slab=(a2:2048@2)) async
p_weight_smem = Stage(p_weight, origin=(0), slab=(a3:2048@0)) async
StridedLoop(a2 = a0; < 2048; += 256): # reduce
in2 = load x_smem[a2]
v0 = multiply(in2, in2)
acc0 <- add(acc0, v0)
v1 = divide(acc0, in0)
v2 = add(v1, in1)
v3 = rsqrt(v2)
StridedLoop(a3 = a0; < 2048; += 256): # free
in3 = load x_smem[a3]
in4 = load p_weight_smem[a3]
v4 = multiply(in3, v3)
v5 = multiply(v4, in4)
rms_norm[0, a1, a3] = v5
Optimized CUDA kernel: emmy compile -c "nn.RMSNorm(2048)(torch.randn(1,32,2048))" --ir cuda
extern "C" __global__
__launch_bounds__(256) void k_rms_norm_reduce(const float* x, const float* p_weight, float* rms_norm) {
float in0 = 2048.0f;
float in1 = 1e-06f;
{
int a1 = blockIdx.x;
int a0 = threadIdx.x;
float acc0 = 0.0f;
__syncthreads();
__shared__ float x_smem[2048];
for (int x_smem_flat = a0; x_smem_flat < 2048; x_smem_flat += 256) {
{
unsigned int _smem_addr = __cvta_generic_to_shared(&x_smem[x_smem_flat]);
asm volatile("cp.async.ca.shared.global [%0], [%1], 4;\n"
:: "r"(_smem_addr), "l"(&x[a1 * 2048 + x_smem_flat])
: "memory");
}
}
asm volatile("cp.async.commit_group;\n" ::: "memory");
asm volatile("cp.async.wait_group 0;\n" ::: "memory");
__syncthreads();
__shared__ float p_weight_smem[2048];
for (int p_weight_smem_flat = a0; p_weight_smem_flat < 2048; p_weight_smem_flat += 256) {
{
unsigned int _smem_addr = __cvta_generic_to_shared(&p_weight_smem[p_weight_smem_flat]);
asm volatile("cp.async.ca.shared.global [%0], [%1], 4;\n"
:: "r"(_smem_addr), "l"(&p_weight[p_weight_smem_flat])
: "memory");
}
}
asm volatile("cp.async.commit_group;\n" ::: "memory");
asm volatile("cp.async.wait_group 0;\n" ::: "memory");
__syncthreads();
for (int a2 = a0; a2 < 2048; a2 += 256) {
float in2 = x_smem[a2];
float v0 = in2 * in2;
acc0 += v0;
}
__shared__ float acc0_smem[256];
acc0_smem[a0] = acc0;
__syncthreads();
for (int s = 128; s > 0; s >>= 1) {
if (a0 < s) {
acc0_smem[a0] = acc0_smem[a0] + acc0_smem[a0 + s];
}
__syncthreads();
}
__syncthreads();
float acc0_b = acc0_smem[0];
float v1 = acc0_b / in0;
float v2 = v1 + in1;
float v3 = rsqrtf(v2);
for (int a3 = a0; a3 < 2048; a3 += 256) {
float in3 = x_smem[a3];
float in4 = p_weight_smem[a3];
float v4 = in3 * v3;
float v5 = v4 * in4;
rms_norm[a1 * 2048 + a3] = v5;
}
}
}
Tune
Try 20 schedules for a single kernel on your GPU, with cold-cache measurements:
emmy run --bench --cold-cache --tune 20 -c "nn.Linear(4096,4096,bias=False)(torch.randn(1,4096))"
Successful measurements are saved so later compiles can reuse them when the kernel and measurement settings match.
Fit the priors
The schedule prior ranks kernel schedules; the placement prior chooses how to split work into kernels. Both guide compilation when no compatible measurement is available. You can refit them from the repository's golden files in a source checkout, without a GPU:
# Load the repository goldens and the measurement freeze into a separate database
emmy db import --db _data/dataset.db --fresh --repository
# Export a dataset for each prior
emmy db export --db _data/dataset.db _data/schedule --space schedule
emmy db export --db _data/dataset.db _data/placement --space placement
# Refit the shipped weights
emmy fit _data/schedule emmy/compiler/pipeline/search/prior/weights/schedule.json
emmy fit _data/placement emmy/compiler/pipeline/search/prior/weights/placement.json
# Evaluate both priors
emmy eval prior _data/schedule
emmy eval prior _data/placement
To experiment without replacing the shipped weights, fit a separate candidate and compare it on the same dataset:
emmy fit _data/schedule _data/schedule-candidate.json --folds 0
emmy eval prior _data/schedule \
--offline-file emmy/compiler/pipeline/search/prior/weights/schedule.json \
--compare-to _data/schedule-candidate.json --json _data/schedule-comparison.json
Adjust --iterations, --depth, or --learning-rate to try different fits. --folds 0 skips cross-validation;
omit it to use the default five folds. Each fit writes metrics under _tune/fits/, and the comparison command writes
a JSON report. See the CLI reference for all tuning and fitting options.
Keep what a tune measured
emmy run --bench --tune N writes every clean row it measures into the tune DB of the machine it ran on, and
freezes the same rows into one file beside the card's under emmy/compiler/pipeline/search/freezes/, the
measurement freeze the repository keeps, one golden file per card in git LFS: --repository reads it beside the
goldens, and the nightly refresh reports the prior's regret over its pools. Commit the tune's file with your change
(run git lfs install once on the machine that commits); the nightly refresh folds the files into each card's file
on main, then compiles every row of the freeze under its own schedule and drops the rows the compiler no longer
realizes (emmy golden realize --drop). A tune on a rented card writes the file into the copy there: copy it back
next to the card's file, as you do the working golden. EMMY_FREEZE_DIR names another directory or, set empty, none.
A tune DB from a machine with no checkout is merged by hand (~/.cache/emmy/autotune.db, or the file EMMY_TUNE_DB
names, copied over):
emmy db import --db _data/freeze.db --fresh emmy/compiler/pipeline/search/freezes autotune.db
emmy db freeze --db _data/freeze.db --out emmy/compiler/pipeline/search/freezes
A schedule measured on both sides takes the tune DB's row, faster or slower, a kernel the compiler no longer lowers drops out, and merging the same tune DB again changes nothing. Commit the rewritten files.
Deploy
Choose a recipe that matches your GPU. This Gemma recipe targets one RTX 5090:
emmy recipe list
# Local Docker Compose
emmy deploy local --recipe gemma-4-12B-it
# An existing server
emmy deploy ssh --recipe gemma-4-12B-it --ssh user@host
# Rent a cloud VM and deploy
emmy deploy cloud --recipe gemma-4-12B-it
A recipe name creates an editable copy in the current directory. You can also pass a recipe directory with
--recipe path/to/recipe. Deployment leaves the server running and prints its endpoint.
Local deployment needs Docker Compose and NVIDIA container support. Cloud deployment needs provider credentials:
CLOUDRIFT_API_KEY for CloudRift, or an authenticated gcloud CLI for GCP.
See the deployment reference for teardown, provider options, and multi-model plans.
Benchmark
From a source checkout, run an experiment on cloud GPUs, your machine, or existing servers:
# Rent GPUs, run one experiment, and tear down
emmy bench experiments/gemma-4-12B/gsm8k_mtp_rtx5090
# Select RTX 5090 variants across Gemma experiments
emmy bench experiments/gemma-4-12B/* --filter "deploy.gpu=*5090*"
# Use this machine or an existing server
emmy bench experiments/gemma-4-12B/gsm8k_mtp_rtx5090 --local
emmy bench experiments/gemma-4-12B/gsm8k_mtp_rtx5090 --ssh user@host
Results and logs are saved in a timestamped directory beside the experiment recipe. Cloud VMs are removed after the run; existing hosts are kept. Local benchmarks use SSH to localhost, so local SSH access must be configured. Browse experiments and results or read the benchmark reference.
Serve
Serve an embedding model through vLLM with Emmy-compiled kernels:
pip install "emmy-ml[compile,serving]"
emmy serve Qwen/Qwen3-Embedding-0.6B
Then send a request from another terminal:
curl localhost:8000/v1/embeddings -H 'Content-Type: application/json' \
-d '{"model":"Qwen/Qwen3-Embedding-0.6B","input":"Hello"}'
For a one-shot benchmark and a stock-vLLM comparison:
emmy serve Qwen/Qwen3-Embedding-0.6B --bench --random-input-len 32
emmy serve Qwen/Qwen3-Embedding-0.6B --bench --random-input-len 32 --stock
See the serving guide for text generation and the native generation guide for the experimental Rust server.
Recipes
A recipe sets the model, serving engine, GPU, and optional benchmark workload. Save this as my-benchmark/recipe.yaml
and run emmy bench my-benchmark:
model:
huggingface: "Qwen/Qwen3-0.6B"
engine:
llm:
tensor_parallel_size: 1
context_length: 4096
vllm:
image: "vllm/vllm-openai:v0.23.0"
benchmark:
num_prompts: 256
random_input_len: 1024
random_output_len: 128
# Cross-product: 2 GPUs × 2 concurrency settings = 4 variants
matrices:
cross:
deploy.gpu_count: 1
deploy.gpu: ["NVIDIA GeForce RTX 5090", "NVIDIA H100 80GB"]
zip:
engine.llm.max_concurrent_requests: [32, 128]
benchmark.max_concurrency: [32, 128]
Browse serving recipes for existing configurations. The recipe reference covers fields, parameter sweeps, LoRA adapters, and custom commands.
Virtual Machine Management
# Rent a GPU server and save its lease for cleanup
emmy vm create gpu --gpu "NVIDIA GeForce RTX 5090" --gpu-count 1 --exact-gpu-count \
--lease /tmp/emmy-vm.json --owner local-run --json
# Delete that server
emmy vm delete lease /tmp/emmy-vm.json --owner local-run
See the VM reference for provider-specific commands and availability queries.
Documentation
- CLI reference — command options, tuning, refitting priors, and image publication
- Compiler tutorials — tracing, optimization, and measuring kernels
- Glossary — compiler and serving terminology
Related Projects
These repositories provide the platform, serving runtimes, and deployment services that Emmy integrates with. The
CloudRift repositories are typically checked out beside Emmy, so use ../<repository-name> for local code inspection
when that sibling exists, and verify the remote source when current compatibility matters. Some are internal and
require CloudRift organization access.
| Project | Relationship to Emmy |
|---|---|
| CloudRift | GPU cloud platform and API/protocol implementation behind Emmy's CloudRift provisioning and deployments. |
| 1Cat-vLLM | Volta/SM70 vLLM fork and source for docker/1cat-vllm-sm70; consult it when stock vLLM lacks target-GPU support. |
| Rift Provisioning | Ansible automation for CloudRift provider nodes and the CI and agent runner hosts used by Emmy workflows. |
| Rift Relay | OpenAI-compatible inference proxy and model-offering service that consumes Emmy recipes and invokes Emmy deployments. |
| vLLM | Primary serving engine upstream and authoritative model-support, release, and image source. |
| SGLang | Alternative serving engine upstream and model-support source. |
Project Structure
- AGENTS.md — shared coding-agent instructions; CLAUDE.md imports the same guidance for Claude Code
- .agents/skills/ — canonical repository skills; .claude/skills/ exposes them through compatibility symlinks
- prompts/ — reusable agent prompts shared by repository skills and non-interactive workflows
- .opencode/ — API-agent provider, permissions, and workflow profiles
- .github/ — Pull-request checks, releases, cloud experiments, and model discovery/onboarding workflows (see ARCHITECTURE.md)
- emmy/ — Python package
- emmy.py — CLI entrypoint
- logging_setup.py — CLI logging configuration
- hardware.py — GPU specs and instance type mapping
- detect.py — GPU detection via PCI sysfs (local and remote)
- redact.py — Secret redaction for logs and dumps
- commands/ — CLI layer (thin argparse handlers, see
ARCHITECTURE.md)
- deploy/ —
deploy local,deploy ssh,deploy cloudcommands - bench/ —
benchcommand - vm/ —
vm create/delete/auditcommands (GCP, CloudRift, owned leases) - teardown.py —
teardowncommand - pull.py —
pullcommand (download HF model) - trace.py —
tracecommand (PyTorch → Graph IR) - compile.py —
compilecommand (decomposition → optimization → fusion → kernel/CUDA lowering) - run.py —
runcommand (compile + execute on CUDA backend, optional benchmarks) - inspect_graph.py —
inspectcommand (graph summary)
- deploy/ —
- compiler/ — PyTorch → Graph IR → CUDA compiler (see
ARCHITECTURE.md)
- graph.py —
Graph,Node,Tensor,Hintscontainer - pretty.py — graph dumps as Rust-shaped pseudocode (see IR-PSEUDOCODE-TORCH.md)
- ir/ — per-dialect op definitions (torch / tensor / loop / kernel / cuda) (see
ARCHITECTURE.md)
- schedule/ — schedule interfaces, reusable choices, and classic scheduling (see ARCHITECTURE.md)
- trace/ — PyTorch/HuggingFace → Graph IR capture (see ARCHITECTURE.md)
- pipeline/ — rewrite engine + passes + dump hooks (see ARCHITECTURE.md)
- backend/ — numpy / loop / CUDA execution (see
ARCHITECTURE.md)
- cuda/ — CUDA backend internals (see ARCHITECTURE.md)
- graph.py —
- recipe/ — Recipe loading, dataclass types, engine flag mapping (see ARCHITECTURE.md)
- serving/ — vLLM out-of-tree embedding plugin (see ARCHITECTURE.md)
- deploy/ — Compose generation, deploy orchestration
- provisioning/ — Cloud provisioning, SSH transport, VM lifecycle
- benchmark/ — Benchmark tracking, config, task enumeration, execution (see ARCHITECTURE.md)
- planner/ — Groups benchmark tasks into execution groups for VM allocation
- recipes/ — The recommended serving configuration and per-GPU model goldens, one directory per model —
what
emmy deployruns (see ARCHITECTURE.md; benchmark grids belong inexperiments/) - docker/ — Custom image builds (vllm-emmy — vLLM + the emmy plugin; vllm-emmy-serve — prebuilt per-model images: warmed cubins + baked model snapshot; emmy-native — a recipe's native lane: the native server and its prepared pack; 1cat-vllm-sm70 — source-pinned 1Cat-vLLM runtimes and request-time GPU caches for Volta)
- experiments/ — Benchmark parameter sweeps, self-contained recipe + committed results —
what
emmy benchruns (see ARCHITECTURE.md) - kernels/ — Standalone CUDA kernel sources
- crates/emmy-runtime/ — The Rust runtime: the executor behind every launch, the standalone
pack loader and cached generation (design, hosts, and qualification in
ARCHITECTURE.md); crates/emmy-runtime-py/ is
its in-process host, the
emmy.emmy_runtimeextension setuptools-rust builds into the package - crates/emmy-ir/ — The compiler's statement and expression vocabulary in Rust: the statement types,
bodies, their text, value numbering and normalization (see ARCHITECTURE.md);
crates/emmy-ir-py/ is the
emmy.emmy_irextension their/stmtandir/expressionmodules export from - crates/emmy-server/ — Native HTTP adapter: text and image processing (see ARCHITECTURE.md)
- docs/ — Docusaurus user-docs site (getting started, benchmarking, custom configurations, deployment)
- tests/ — pytest tests (see ARCHITECTURE.md)
- compiler/passes/ — compiler pass tests (see ARCHITECTURE.md)
- compiler/realization/ — the realization corpus: checked-in reproducers of pinned schedules, replayed as data (see ARCHITECTURE.md)
- perf/ — GPU perf comparison vs PyTorch (see ARCHITECTURE.md)
- scripts/ — Analysis and visualization scripts
- utils/ — Standalone utility scripts
- config.yaml — Benchmark configuration
- Makefile — Build automation
- pyproject.toml — Package metadata and tool config
Contributing
Source setup requires Python 3.12+ and a Rust toolchain:
git clone https://github.com/cloudrift-ai/emmy.git
cd emmy
make setup
Follow STYLE.md and the contribution instructions for development, testing, and PRs. Release procedures live in the workflow documentation.
License
Licensed under the Apache License 2.0.
Metadata
Release files for emmy-ml 0.3.25
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| emmy_ml-0.3.25.tar.gz | 7.6 MB | With provenance | Details for emmy_ml-0.3.25.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| emmy_ml-0.3.25-cp312-abi3-manylinux_2_34_x86_64.whl | CPython 3.12 | abi3 | Linux glibc 2.34+ x86-64 | Details for emmy_ml-0.3.25-cp312-abi3-manylinux_2_34_x86_64.whl |
Total release size: 17.9 MB
Release files / emmy_ml-0.3.25.tar.gz
| Download URL | emmy_ml-0.3.25.tar.gz |
|---|---|
| Size | 7.6 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
b2ab6d4a84bde7b89266d5983a12871f8bf42e88b2dfe0e9a938a78da1061202
|
|
BLAKE2b-256 checksum How to use checksums |
177e813fb01c5dfe6b1dc891a66d36f05acca9fe515093ccdfaeeed3a95ec3f3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.
Transparency logRelease files / emmy_ml-0.3.25-cp312-abi3-manylinux_2_34_x86_64.whl
| Download URL | emmy_ml-0.3.25-cp312-abi3-manylinux_2_34_x86_64.whl |
|---|---|
| Size | 10.3 MB |
| Tags | CPython 3.12 Linux glibc 2.34+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
d353e3d1546038157ecda625ffdb304e639ecaa3adf5aa16f4640f4b9fcffb4f
|
|
BLAKE2b-256 checksum How to use checksums |
bccd8d399345e77ad4270c0646fbadf571bfb6651d91e2eb423d45be3e2fe733
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.
Transparency logSecurity for emmy-ml 0.3.25
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for emmy_ml-0.3.25.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.