g2n 0.5.6

Optimize and run PyTorch models: an open-core compiler (fusion, buffer planning, persistent compile cache) plus a license-gated serving platform that runs your models behind an inference server.

g2n

Optimize and run PyTorch models. g2n is the open-core compiler at the center of the g2n platform: pointwise fusion, buffer-reuse planning, and a persistent cross-run compile cache so repeat builds skip recompilation. A license unlocks the enhanced planner, the persistent cache, and a full serving layer that runs your models behind an HTTP inference server.

pip install g2n

import torch
import g2n

model = MyModule().eval()
compiled = g2n.compile(model)                 # optimize
# or register as a torch.compile backend:
compiled = torch.compile(model, backend="g2n")

Two halves, one license

	Community (free)	Pro	Enterprise
Optimize — fusion, JIT pointwise codegen, CPU fallback	✓	✓	✓
Enhanced buffer planner + persistent compile cache		✓	✓
Run — model registry + inference server (g2n.serve())		✓	✓
Dynamic batching, multi-accelerator routing, model-zoo			✓

Activate a license to light up the paid tiers (the same code path — gated features turn on, otherwise it falls back to the open-core path):

g2n activate G2N-XXXX-XXXX-XXXX

Run your models (Pro+)

The enterprise client (pip install g2n-enterprise) adds the serving platform:

import g2n_enterprise as g2n
g2n.register_model("resnet", "torchscript:/models/resnet50.pt",
                   precision="auto", cuda_graph=True, max_batch=16)
g2n.serve(port=8900)        # POST /v1/models/resnet/predict
res = g2n.benchmark("resnet", sample, rounds=200)   # eager vs optimized, measured on your box

Serving applies real inference techniques — inference_mode, fp16/bf16/int8, CUDA-graph capture/replay (which removes the launch overhead that makes "compiled tie eager" on small GPUs), and a VRAM residency manager so a small card serves more models than fit. Speedups are hardware-dependent: benchmark on your own GPU rather than trusting a quoted number.

Custom kernels (Pro / Enterprise)

With a licensed tier, the g2n backend runs a real custom compile pass: it fuses LayerNorm (and a trailing GELU) into a Triton kernel via a torch.library custom op, then hands the rest of the graph to TorchInductor. See ARCHITECTURE.md. Correctness is covered by tests/test_layernorm.py.

The fusion is inference-only. The fused kernel is forward-only, so the pass skips any differentiable (training) graph and lets stock lowering handle it — training compiles correctly, just unfused. Inference under torch.no_grad() / torch.inference_mode() (which the serving runtime always uses) gets the fused kernel. Benchmark on your own GPU before quoting a speedup.

Docs: https://g2n.dev/docs · Pricing: https://g2n.dev/pricing