ISAT

Inference Stack Auto-Tuner

A production-grade inference engine for ONNX models.
91 CLI commands. Any GPU. Any framework. One tool.

pip install isat-tuner

# Convert + auto-tune in one command
isat onnx facebook/opt-1.3b
isat tune model.onnx

# Serve with continuous batching
isat serve-llm model.onnx --port 8000

# 2-4x LLM speedup
isat speculate model.onnx --draft draft.onnx

Key Numbers

Feature Overview

ISAT is organized into three tiers of capability:

Tier 0 — Trillion-Dollar Infrastructure (NEW in v0.12.0)

Features deployed at Google, OpenAI, Meta, Microsoft, and Anthropic.

Tier 1 — LLM Inference Engine

These features put ISAT on par with dedicated LLM serving frameworks.

Tier 2 — Model Engineering

Tools for adapting, parallelizing, and optimizing models before deployment.

Tier 3 — Production Operations

Everything needed to deploy, monitor, secure, and validate models in production.

Validated Models

Every release is tested end-to-end against real HuggingFace models:

Installation

pip install isat-tuner

# All optional features
pip install "isat-tuner[all]"

# Model conversion
pip install "isat-tuner[convert]"       # All backends
pip install "isat-tuner[convert-hf]"    # HuggingFace (optimum)
pip install "isat-tuner[convert-pt]"    # PyTorch
pip install "isat-tuner[convert-tf]"    # TensorFlow

# Feature-specific
pip install "isat-tuner[stream]"        # Streaming inference (transformers)
pip install "isat-tuner[encrypt]"       # Model encryption (cryptography)

# Platform-specific
pip install "isat-tuner[rocm]"          # AMD ROCm
pip install "isat-tuner[cuda]"          # NVIDIA CUDA
pip install "isat-tuner[server]"        # FastAPI server
pip install "isat-tuner[bayesian]"      # Bayesian optimization

# From GitHub (latest)
pip install git+https://github.com/SID-Devu/isat-tuner.git

# Development
git clone https://github.com/SID-Devu/isat-tuner.git
cd isat && pip install -e ".[dev,all]"

Quick Start

Convert any model to ONNX

isat onnx google/vit-base-patch16-224          # HuggingFace Vision Transformer
isat onnx facebook/opt-1.3b                    # HuggingFace LLM (1.3B params)
isat onnx model.pt --input-shape 1,3,224,224   # Local PyTorch
isat onnx saved_model/                         # TensorFlow SavedModel
isat onnx model.tflite                         # TFLite

Auto-tune for your hardware

isat tune model.onnx                           # Auto-detect + optimize
isat tune model.onnx --profile cloud           # Cloud deployment profile
isat tune model.onnx --bayesian --max-configs 20  # Bayesian search

Serve with continuous batching

isat serve-llm model.onnx --tokenizer gpt2 --port 8000

# OpenAI-compatible API
curl http://localhost:8000/v1/completions \
  -d '{"prompt": "Hello", "max_tokens": 50, "stream": true}'

Speculative decoding (2-4x speedup)

isat speculate target.onnx --draft draft.onnx --benchmark
isat speculate target.onnx --mode self         # Self-speculation (no draft needed)

Grammar-constrained generation

isat constrain model.onnx --schema '{"type":"object","properties":{"name":{"type":"string"},"age":{"type":"integer"}}}'
isat constrain model.onnx --regex '[0-9]{3}-[0-9]{2}-[0-9]{4}'
isat constrain model.onnx --grammar grammar.gbnf

LoRA adapter management

isat lora base.onnx --adapter lora_weights.npz --action fuse -o fused.onnx
isat lora base.onnx --action merge --merge-method ties --merge-models a.onnx b.onnx c.onnx

Quantize and profile precision

isat quantize model.onnx --method int4 --block-size 128
isat amp-profile model.onnx --action optimize --max-mse 0.001 -o mixed.onnx

Architecture surgery

isat a2a model.onnx --action analyze
isat a2a model.onnx --action prune-heads --ratio 0.5 --method magnitude -o pruned.onnx
isat a2a model.onnx --action shrink-depth --ratio 0.5 -o smaller.onnx

Knowledge distillation

isat distill-train teacher.onnx --epochs 20 --temperature 4.0 -o student.onnx

Deploy to cloud

isat cloud-deploy model.onnx --output-dir deploy/   # Docker + K8s + SageMaker + Azure + GCP

Monitor in production

isat monitor-live --model model.onnx                 # TUI dashboard + anomaly detection
isat test model.onnx --junit                         # Automated testing with CI output

# Comprehensive benchmarking
isat benchmark-suite model.onnx --batch-sizes 1,4,16,64
isat llm-bench model.onnx --seq-lengths 32,64,128,256

# Model security
isat encrypt model.onnx -o encrypted.onnx --method encrypt --password "secret"
isat encrypt model.onnx -o fingerprinted.onnx --method fingerprint --owner "ACME Corp"
isat safety --input-text "check this text for PII and toxicity"

# Explainability
isat explain model.onnx --method perturbation

# Sharding and merging
isat shard large_model.onnx --num-shards 4 --strategy balanced
isat merge encoder.onnx decoder.onnx -o pipeline.onnx --mode chain

# CI/CD gate
isat tune model.onnx --gate-latency 50 --gate-throughput 100
echo $?  # 0 = pass, 1 = fail

# CUDA/HIP graph capture
isat graph-compile model.onnx --action benchmark

# Tensor parallelism
isat tensor-parallel model.onnx --num-gpus 4 --action split

Using as a Library

from isat.converter.engine import convert
from isat.auto_detect.detector import detect_hardware
from isat.auto_detect.recommender import generate_recommendations

# Convert any model to ONNX
result = convert("google/vit-base-patch16-224", output_dir="./output")

# Auto-detect hardware and get recommendations
hw = detect_hardware()
report = generate_recommendations(hw, result.onnx_path)

# v0.11.0 — LLM Engine
from isat.speculative.engine import SpeculativeDecoder, SelfSpeculativeDecoder, MedusaDecoder
from isat.llm_server.server import LLMServer, create_app, serve_llm
from isat.constrained.grammar import ConstrainedGenerator, constrained_generate

# v0.11.0 — Model Engineering
from isat.lora.adapter import LoRARuntime, MultiLoRARouter
from isat.lora.merger import WeightMerger
from isat.parallel.tensor_parallel import TensorParallelizer, TensorParallelRunner
from isat.graph_compile.capture import GraphCapture, GraphRegionAnalyzer
from isat.amp.profiler import PrecisionProfiler
from isat.amp.optimizer import MixedPrecisionOptimizer
from isat.distill_train.trainer import DistillationTrainer
from isat.arch_convert.converter import ArchitectureConverter

# v0.11.0 — Production
from isat.live_monitor.daemon import InferenceMonitor
from isat.live_monitor.dashboard import MonitorDashboard

# v0.10.0
from isat.quantize.quantizer import ModelQuantizer, quantize_model
from isat.stream.generator import StreamingGenerator
from isat.shard.splitter import ModelSharder
from isat.merge.merger import ModelMerger
from isat.explain.explainer import ModelExplainer
from isat.benchmark_suite.suite import BenchmarkSuite
from isat.encrypt.protector import ModelProtector
from isat.safety.guardrails import SafetyGuard
from isat.cloud_deploy.deployer import CloudDeployer
from isat.model_test.tester import ModelTester

# Core
from isat.fingerprint import fingerprint_hardware, fingerprint_model
from isat.search import SearchEngine
from isat.pruning.pruner import ModelPruner
from isat.fusion.analyzer import FusionAnalyzer

How the Auto-Tuner Works

ISAT explores a 7-dimension search space that would take hours to test manually:

A single wrong choice can leave 40%+ performance on the table. ISAT tests combinations automatically and reports the best configuration.

Deployment Profiles

Architecture

isat/                               30,000+ lines of Python
├── cli.py                          91 CLI commands
│
├── kv_compress/                    KV cache compression (KIVI/H2O)
├── disaggregate/                   Disaggregated prefill-decode
├── moe_runtime/                    MoE expert parallelism + routing
├── model_router/                   Cascade routing + cost-aware selection
├── multimodal/                     Multi-modal pipeline orchestrator
├── rag_engine/                     RAG: HNSW + BM25 + RRF + reranking
├── long_context/                   100K+ context: RoPE scaling, ring attention
├── inference_compiler/             Kernel fusion + memory planning
├── slo_scheduler/                  SLO-aware scheduling + fair queuing
├── prompt_cache/                   Radix tree prompt cache
├── watermark/                      AI text watermarking (Kirchenbauer)
├── token_economics/                Cost attribution + budget enforcement
├── session_manager/                Multi-turn KV persistence
├── shadow_deploy/                  Shadow deployment + auto-promotion
├── edge_split/                     Edge-cloud hybrid inference
│
├── speculative/                    Speculative decoding engine
├── llm_server/                     Continuous batching + PagedAttention
├── constrained/                    Grammar FSM (NFA/DFA/PDA)
├── lora/                           LoRA runtime + TIES/DARE/SLERP
├── parallel/                       Tensor parallelism + all-reduce
├── graph_compile/                  CUDA/HIP graph capture
├── amp/                            Mixed-precision profiling + optimization
├── distill_train/                  Knowledge distillation training
├── arch_convert/                   Architecture surgery
├── live_monitor/                   Anomaly detection + TUI dashboard
│
├── converter/                      Universal model-to-ONNX conversion
├── quantize/                       INT4/INT8/FP16/SmoothQuant
├── stream/                         Token-by-token LLM generation
├── shard/                          Model sharding + pipeline execution
├── merge/                          Model merging + composition
├── explain/                        Explainability tools
├── benchmark_suite/                Latency/throughput/memory benchmarks
├── encrypt/                        AES-256 encryption + watermarking
├── safety/                         PII/toxicity/jailbreak guardrails
├── cloud_deploy/                   Docker/K8s/SageMaker/Azure/GCP
├── model_test/                     Automated testing + JUnit output
│
├── auto_detect/                    Hardware detection + recommendations
├── fingerprint/                    Hardware + model fingerprinting
├── search/                         7-dimension search + Bayesian opt
├── benchmark/                      Runner, stats, thermal, multi-GPU
├── analysis/                       Outliers, significance, Pareto
├── server/                         FastAPI REST API
├── ... (20+ more modules)          Pruning, fusion, canary, tracing, etc.
└── utils/                          sysfs, rocm, onnx utilities

REST API

isat serve --port 8000

CI/CD Integration

# Gate deployments on performance thresholds
isat tune model.onnx --gate-latency 50 --gate-throughput 100
echo $?  # 0 = pass, 1 = fail

# Automated model testing with JUnit XML
isat test model.onnx --junit --output-dir test-results/

Docker

docker-compose up -d

# Or standalone
docker build -t isat .
docker run --device /dev/kfd --device /dev/dri --group-add video \
  -v ./models:/models isat tune /models/model.onnx

Generated Artifacts

Version History

Requirements

• Python >= 3.9
• Runtime: onnxruntime (CPU), onnxruntime-rocm (AMD), or onnxruntime-gpu (NVIDIA)
• Core: onnx, numpy
• Optional: transformers, torch, tensorflow, fastapi, uvicorn, cryptography, scipy, onnxsim

Contributing

Contributions are welcome. Please open an issue first to discuss what you'd like to change.

git clone https://github.com/SID-Devu/isat-tuner.git
cd isat && pip install -e ".[dev,all]"

Citation

@software{isat_tuner,
  author = {Sudheer Ibrahim Daniel Devu},
  title  = {ISAT: Inference Stack Auto-Tuner},
  year   = {2026},
  version = {0.12.0},
  url    = {https://github.com/SID-Devu/isat-tuner},
  note   = {91-command production inference engine for ONNX models}
}

Apache 2.0 — Copyright 2026 Sudheer Ibrahim Daniel Devu