Skip to main content

PyPI Tests Discord

Compile → Benchmark → Deploy. Emmy compiles PyTorch models into CUDA kernels, benchmarks LLM inference, and deploys models on local or cloud GPUs with vLLM and SGLang.

Compile · Tune · Deploy · Benchmark · Serve · Recipes

See Emmy in action: Outperforming vLLM on Gemma4-12B.

Install

Requires Python 3.12 or newer.

pip install emmy-ml
emmy --version

The package includes ready-to-use recipes. Remote deployment and benchmarking do not require a local GPU or CUDA installation. For gated Hugging Face models, accept the model's license and set HF_TOKEN.

Compile

Compile a PyTorch operation, compare its speed with PyTorch, or inspect the generated CUDA:

pip install "emmy-ml[compile]"

# Compile an operation
emmy compile -c "nn.RMSNorm(2048)(torch.randn(1,32,2048))"

# Benchmark against eager PyTorch and torch.compile
emmy run --bench -c "nn.Softmax(dim=-1)(torch.randn(1,28,2048,2048))"

# Inspect the generated CUDA
emmy compile -c "nn.RMSNorm(2048)(torch.randn(1,32,2048))" --ir cuda

Running compiled kernels requires an NVIDIA GPU and the CUDA toolkit (nvcc); source builds also need Rust. For GPUs older than Turing, use CUDA 12 and a PyTorch build that supports your GPU. Fast math is enabled by default; set EMMY_FAST_MATH=0 to disable it.

You can also compile a custom module:

emmy compile -c "
class LN(nn.Module):
    def forward(self, x):
        m = x.mean(-1, keepdim=True)
        v = ((x - m) ** 2).mean(-1, keepdim=True)
        return (x - m) * torch.rsqrt(v + 1e-6)
LN()(torch.randn(64, 2048))"

For Python integration, use torch.compile(model, backend="emmy", dynamic=False) under torch.no_grad(). Read the compiler guide or the compiler walkthrough for more.

Lowered kernel example

Readable Schedule: emmy compile -c "nn.RMSNorm(2048)(torch.randn(1,32,2048))" --ir tile

kernel k_rms_norm_reduce  inputs: rms_norm_mean_count, rms_norm_eps, x, p_weight  outputs: rms_norm
    in0 = load rms_norm_mean_count[0]
    in1 = load rms_norm_eps[0]
    Tile(axes=(a0:256=THREAD, a1:32=BLOCK)):
        x_smem = Stage(x, origin=(0, a1, 0), slab=(a2:2048@2)) async
        p_weight_smem = Stage(p_weight, origin=(0), slab=(a3:2048@0)) async
        StridedLoop(a2 = a0; < 2048; += 256):  # reduce
            in2 = load x_smem[a2]
            v0 = multiply(in2, in2)
            acc0 <- add(acc0, v0)
        v1 = divide(acc0, in0)
        v2 = add(v1, in1)
        v3 = rsqrt(v2)
        StridedLoop(a3 = a0; < 2048; += 256):  # free
            in3 = load x_smem[a3]
            in4 = load p_weight_smem[a3]
            v4 = multiply(in3, v3)
            v5 = multiply(v4, in4)
            rms_norm[0, a1, a3] = v5

Optimized CUDA kernel: emmy compile -c "nn.RMSNorm(2048)(torch.randn(1,32,2048))" --ir cuda

extern "C" __global__
__launch_bounds__(256) void k_rms_norm_reduce(const float* x, const float* p_weight, float* rms_norm) {
    float in0 = 2048.0f;
    float in1 = 1e-06f;
    {
        int a1 = blockIdx.x;
        int a0 = threadIdx.x;
        float acc0 = 0.0f;
        __syncthreads();
        __shared__ float x_smem[2048];
        for (int x_smem_flat = a0; x_smem_flat < 2048; x_smem_flat += 256) {
            {
                unsigned int _smem_addr = __cvta_generic_to_shared(&x_smem[x_smem_flat]);
                asm volatile("cp.async.ca.shared.global [%0], [%1], 4;\n"
                             :: "r"(_smem_addr), "l"(&x[a1 * 2048 + x_smem_flat])
                             : "memory");
            }
        }
        asm volatile("cp.async.commit_group;\n" ::: "memory");
        asm volatile("cp.async.wait_group 0;\n" ::: "memory");
        __syncthreads();
        __shared__ float p_weight_smem[2048];
        for (int p_weight_smem_flat = a0; p_weight_smem_flat < 2048; p_weight_smem_flat += 256) {
            {
                unsigned int _smem_addr = __cvta_generic_to_shared(&p_weight_smem[p_weight_smem_flat]);
                asm volatile("cp.async.ca.shared.global [%0], [%1], 4;\n"
                             :: "r"(_smem_addr), "l"(&p_weight[p_weight_smem_flat])
                             : "memory");
            }
        }
        asm volatile("cp.async.commit_group;\n" ::: "memory");
        asm volatile("cp.async.wait_group 0;\n" ::: "memory");
        __syncthreads();
        for (int a2 = a0; a2 < 2048; a2 += 256) {
            float in2 = x_smem[a2];
            float v0 = in2 * in2;
            acc0 += v0;
        }
        __shared__ float acc0_smem[256];
        acc0_smem[a0] = acc0;
        __syncthreads();
        for (int s = 128; s > 0; s >>= 1) {
            if (a0 < s) {
                acc0_smem[a0] = acc0_smem[a0] + acc0_smem[a0 + s];
            }
            __syncthreads();
        }
        __syncthreads();
        float acc0_b = acc0_smem[0];
        float v1 = acc0_b / in0;
        float v2 = v1 + in1;
        float v3 = rsqrtf(v2);
        for (int a3 = a0; a3 < 2048; a3 += 256) {
            float in3 = x_smem[a3];
            float in4 = p_weight_smem[a3];
            float v4 = in3 * v3;
            float v5 = v4 * in4;
            rms_norm[a1 * 2048 + a3] = v5;
        }
    }
}

Tune

Try 20 schedules for a single kernel on your GPU, with cold-cache measurements:

emmy run --bench --cold-cache --tune 20 -c "nn.Linear(4096,4096,bias=False)(torch.randn(1,4096))"

Successful measurements are saved so later compiles can reuse them when the kernel and measurement settings match.

Fit the priors

The schedule prior ranks kernel schedules; the placement prior chooses how to split work into kernels. Both guide compilation when no compatible measurement is available. You can refit them from the repository's golden files in a source checkout, without a GPU:

# Load the repository goldens and the measurement freeze into a separate database
emmy db import --db _data/dataset.db --fresh --repository

# Export a dataset for each prior
emmy db export --db _data/dataset.db _data/schedule --space schedule
emmy db export --db _data/dataset.db _data/placement --space placement

# Refit the shipped weights
emmy fit _data/schedule emmy/compiler/pipeline/search/prior/weights/schedule.json
emmy fit _data/placement emmy/compiler/pipeline/search/prior/weights/placement.json

# Evaluate both priors
emmy eval prior _data/schedule
emmy eval prior _data/placement

To experiment without replacing the shipped weights, fit a separate candidate and compare it on the same dataset:

emmy fit _data/schedule _data/schedule-candidate.json --folds 0
emmy eval prior _data/schedule \
  --offline-file emmy/compiler/pipeline/search/prior/weights/schedule.json \
  --compare-to _data/schedule-candidate.json --json _data/schedule-comparison.json

Adjust --iterations, --depth, or --learning-rate to try different fits. --folds 0 skips cross-validation; omit it to use the default five folds. Each fit writes metrics under _tune/fits/, and the comparison command writes a JSON report. See the CLI reference for all tuning and fitting options.

Keep what a tune measured

emmy run --bench --tune N writes every clean row it measures into the tune DB of the machine it ran on, and freezes the same rows into one file beside the card's under emmy/compiler/pipeline/search/freezes/, the measurement freeze the repository keeps, one golden file per card in git LFS: --repository reads it beside the goldens, and the nightly refresh reports the prior's regret over its pools. Commit the tune's file with your change (run git lfs install once on the machine that commits); the nightly refresh folds the files into each card's file on main, then compiles every row of the freeze under its own schedule and drops the rows the compiler no longer realizes (emmy golden realize --drop). A tune on a rented card writes the file into the copy there: copy it back next to the card's file, as you do the working golden. EMMY_FREEZE_DIR names another directory or, set empty, none.

A tune DB from a machine with no checkout is merged by hand (~/.cache/emmy/autotune.db, or the file EMMY_TUNE_DB names, copied over):

emmy db import --db _data/freeze.db --fresh emmy/compiler/pipeline/search/freezes autotune.db
emmy db freeze --db _data/freeze.db --out emmy/compiler/pipeline/search/freezes

A schedule measured on both sides takes the tune DB's row, faster or slower, a kernel the compiler no longer lowers drops out, and merging the same tune DB again changes nothing. Commit the rewritten files.

Deploy

Choose a recipe that matches your GPU. This Gemma recipe targets one RTX 5090:

emmy recipe list

# Local Docker Compose
emmy deploy local --recipe gemma-4-12B-it

# An existing server
emmy deploy ssh --recipe gemma-4-12B-it --ssh user@host

# Rent a cloud VM and deploy
emmy deploy cloud --recipe gemma-4-12B-it

A recipe name creates an editable copy in the current directory. You can also pass a recipe directory with --recipe path/to/recipe. Deployment leaves the server running and prints its endpoint.

Local deployment needs Docker Compose and NVIDIA container support. Cloud deployment needs provider credentials: CLOUDRIFT_API_KEY for CloudRift, or an authenticated gcloud CLI for GCP. See the deployment reference for teardown, provider options, and multi-model plans.

Benchmark

From a source checkout, run an experiment on cloud GPUs, your machine, or existing servers:

# Rent GPUs, run one experiment, and tear down
emmy bench experiments/gemma-4-12B/gsm8k_mtp_rtx5090

# Select RTX 5090 variants across Gemma experiments
emmy bench experiments/gemma-4-12B/* --filter "deploy.gpu=*5090*"

# Use this machine or an existing server
emmy bench experiments/gemma-4-12B/gsm8k_mtp_rtx5090 --local
emmy bench experiments/gemma-4-12B/gsm8k_mtp_rtx5090 --ssh user@host

Results and logs are saved in a timestamped directory beside the experiment recipe. Cloud VMs are removed after the run; existing hosts are kept. Local benchmarks use SSH to localhost, so local SSH access must be configured. Browse experiments and results or read the benchmark reference.

Serve

Serve an embedding model through vLLM with Emmy-compiled kernels:

pip install "emmy-ml[compile,serving]"
emmy serve Qwen/Qwen3-Embedding-0.6B

Then send a request from another terminal:

curl localhost:8000/v1/embeddings -H 'Content-Type: application/json' \
  -d '{"model":"Qwen/Qwen3-Embedding-0.6B","input":"Hello"}'

For a one-shot benchmark and a stock-vLLM comparison:

emmy serve Qwen/Qwen3-Embedding-0.6B --bench --random-input-len 32
emmy serve Qwen/Qwen3-Embedding-0.6B --bench --random-input-len 32 --stock

See the serving guide for text generation and the native generation guide for the experimental Rust server.

Recipes

A recipe sets the model, serving engine, GPU, and optional benchmark workload. Save this as my-benchmark/recipe.yaml and run emmy bench my-benchmark:

model:
  huggingface: "Qwen/Qwen3-0.6B"

engine:
  llm:
    tensor_parallel_size: 1
    context_length: 4096
    vllm:
      image: "vllm/vllm-openai:v0.23.0"

benchmark:
  num_prompts: 256
  random_input_len: 1024
  random_output_len: 128

# Cross-product: 2 GPUs × 2 concurrency settings = 4 variants
matrices:
  cross:
    deploy.gpu_count: 1
    deploy.gpu: ["NVIDIA GeForce RTX 5090", "NVIDIA H100 80GB"]
    zip:
      engine.llm.max_concurrent_requests: [32, 128]
      benchmark.max_concurrency: [32, 128]

Browse serving recipes for existing configurations. The recipe reference covers fields, parameter sweeps, LoRA adapters, and custom commands.

Virtual Machine Management

# Rent a GPU server and save its lease for cleanup
emmy vm create gpu --gpu "NVIDIA GeForce RTX 5090" --gpu-count 1 --exact-gpu-count \
  --lease /tmp/emmy-vm.json --owner local-run --json

# Delete that server
emmy vm delete lease /tmp/emmy-vm.json --owner local-run

See the VM reference for provider-specific commands and availability queries.

Documentation

  • CLI reference — command options, tuning, refitting priors, and image publication
  • Compiler tutorials — tracing, optimization, and measuring kernels
  • Glossary — compiler and serving terminology

These repositories provide the platform, serving runtimes, and deployment services that Emmy integrates with. The CloudRift repositories are typically checked out beside Emmy, so use ../<repository-name> for local code inspection when that sibling exists, and verify the remote source when current compatibility matters. Some are internal and require CloudRift organization access.

Project Relationship to Emmy
CloudRift GPU cloud platform and API/protocol implementation behind Emmy's CloudRift provisioning and deployments.
1Cat-vLLM Volta/SM70 vLLM fork and source for docker/1cat-vllm-sm70; consult it when stock vLLM lacks target-GPU support.
Rift Provisioning Ansible automation for CloudRift provider nodes and the CI and agent runner hosts used by Emmy workflows.
Rift Relay OpenAI-compatible inference proxy and model-offering service that consumes Emmy recipes and invokes Emmy deployments.
vLLM Primary serving engine upstream and authoritative model-support, release, and image source.
SGLang Alternative serving engine upstream and model-support source.

Project Structure

Contributing

Source setup requires Python 3.12+ and a Rust toolchain:

git clone https://github.com/cloudrift-ai/emmy.git
cd emmy
make setup

Follow STYLE.md and the contribution instructions for development, testing, and PRs. Release procedures live in the workflow documentation.

License

Licensed under the Apache License 2.0.

Metadata

Release files for emmy-ml 0.3.25

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for emmy-ml 0.3.25
File Size Uploaded Provenance Actions
emmy_ml-0.3.25.tar.gz 7.6 MB With provenance Details for emmy_ml-0.3.25.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for emmy-ml 0.3.25
File Interpreter ABI Platform Actions
emmy_ml-0.3.25-cp312-abi3-manylinux_2_34_x86_64.whl CPython 3.12 abi3 Linux glibc 2.34+ x86-64 Details for emmy_ml-0.3.25-cp312-abi3-manylinux_2_34_x86_64.whl

Total release size: 17.9 MB

Release files / emmy_ml-0.3.25.tar.gz

Download URL emmy_ml-0.3.25.tar.gz
Size 7.6 MB
Tags Source
SHA-256 checksum
How to use checksums
b2ab6d4a84bde7b89266d5983a12871f8bf42e88b2dfe0e9a938a78da1061202
BLAKE2b-256 checksum
How to use checksums
177e813fb01c5dfe6b1dc891a66d36f05acca9fe515093ccdfaeeed3a95ec3f3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.

Transparency log

Release files / emmy_ml-0.3.25-cp312-abi3-manylinux_2_34_x86_64.whl

Download URL emmy_ml-0.3.25-cp312-abi3-manylinux_2_34_x86_64.whl
Size 10.3 MB
Tags CPython 3.12 Linux glibc 2.34+ x86-64 abi3
SHA-256 checksum
How to use checksums
d353e3d1546038157ecda625ffdb304e639ecaa3adf5aa16f4640f4b9fcffb4f
BLAKE2b-256 checksum
How to use checksums
bccd8d399345e77ad4270c0646fbadf571bfb6651d91e2eb423d45be3e2fe733
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.

Transparency log

Security for emmy-ml 0.3.25

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for emmy_ml-0.3.25.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page