baremetalrt 0.21.19

BareMetalRT — CLI for the Windows-native local AI agent runtime

BareMetalRT

The Windows-native local AI agent runtime for NVIDIA RTX PCs.

Run reliable tool-calling agents — with voice, vision, RAG, memory, and a coding agent — entirely on your own GPU. No cloud, no per-token bill, no data leaving the machine. BareMetalRT uses NVIDIA's own TensorRT-LLM CUDA kernels (the engine behind cloud inference APIs) natively on Windows, and can pool multiple consumer GPUs — even mismatched cards across machines — into a single tensor-parallel runtime over standard networking.

Download Installer | Live Demo | Documentation | PyPI | Technical Paper

How It Works

1. Install the BareMetalRT daemon on each Windows machine with an NVIDIA GPU
2. Connect your GPU to your account at baremetalrt.ai
3. Run inference — the system automatically shards models across your available GPUs

┌──────────────────────────────────┐
│  baremetalrt.ai                  │
│  Auth, routing, OpenAI-compat API│
└──────────────┬───────────────────┘
               │ WebSocket
    ┌──────────┼──────────┐
    │          │          │
┌───▼───┐ ┌───▼───┐ ┌───▼───┐
│Node A │ │Node B │ │Node C │
│3090   │ │4060   │ │3060   │
│24GB   │ │8GB    │ │12GB   │
│Daemon │ │Daemon │ │Daemon │
└───┬───┘ └───┬───┘ └───┬───┘
    └── TCP AllReduce ──┘

Why BareMetalRT

	Cloud (OpenAI, etc.)	Local (Ollama, LM Studio)	Distributed (Petals, Exo)	BareMetalRT
Kernels	Optimized (proprietary)	Generic CUDA	Generic CUDA	TensorRT-LLM (1,500+ optimized .cu)
Multi-GPU	NVLink (datacenter only)	Single GPU only	Pipeline parallelism	Tensor parallelism over TCP
Heterogeneous GPUs	No	N/A	No	Yes — different VRAM, different SMs
Windows native	N/A	Yes	Partial	Yes
Cost	Per-token pricing	Free (your hardware)	Free (your hardware)	Free (your hardware)
Privacy	Data leaves your machine	Fully local	Weights distributed	Weights distributed, execution local

The key difference: every other consumer GPU project uses pipeline parallelism, which leaves GPUs idle 50% of the time. BareMetalRT is the first to achieve tensor parallelism across heterogeneous consumer GPUs — both GPUs compute on every layer, every token.

Benchmarks

Single-GPU throughput — the everyday path

Most work runs on one card. Measured on an RTX 4070 SUPER (12 GB), int4 (W4A16-AWQ), streaming decode:

Model	Throughput
Llama 3.2 3B	130 tok/s
DeepSeek-R1 Distill 7B / Llama 8B	75 tok/s
Qwen 2.5 Coder 7B	75 tok/s
Llama 3.1 8B	74 tok/s
Qwen 2.5 7B	73 tok/s

Every one of these streams far faster than a person reads, on a single consumer GPU.

Heterogeneous tensor parallelism — the hard result

BareMetalRT can also split one model across mismatched GPUs — even across machines — over standard networking. Tested with Mistral 7B (14 GB FP16) across an RTX 4070 SUPER (12 GB) and an RTX 4060 Laptop (8 GB), a model too large for either card alone:

Configuration	Throughput	Latency	Notes
TP=2 — Mistral 7B	12.5 tok/s	80 ms/tok	KV cache + overlapped AllReduce
TP=2 — Ethernet (1 ms ping)	3.6 tok/s	276 ms/tok	TinyLlama 1.1B
TP=2 — WiFi (316 ms ping)	3.6 tok/s	277 ms/tok	TinyLlama 1.1B

Key finding: a 300× improvement in network speed (WiFi → Ethernet) yielded zero throughput change. GPU-synchronization overhead — not network latency — is the dominant bottleneck. The correctness result — identical output from mismatched GPUs over a commodity network — is what matters.

System Requirements

• Windows 10/11 (64-bit)
• NVIDIA GPU (RTX 2000+; 8 GB VRAM or more recommended)
• An up-to-date NVIDIA driver

The CUDA runtime and TensorRT-LLM are bundled with the installer — no separate CUDA Toolkit or TensorRT installation is required.

Quick Start

1. Install

Download the code-signed MSI from GitHub Releases and run it. It bundles the CUDA runtime and TensorRT-LLM — no separate installs — and registers the daemon to start on boot. Everything is served locally at http://localhost:8080.

2. Load a model and chat

Open the app at baremetalrt.ai/app — it auto-detects the daemon on your machine — or drive it from the terminal with the bmrt CLI:

pip install baremetalrt
bmrt status                  # GPU + daemon state
bmrt models                  # catalog + download status
bmrt pull qwen3-4b-int4      # download a catalog model
bmrt run qwen3-4b-int4       # load it and chat in the terminal

The CLI is a thin wrapper over the local daemon's REST API, defaulting to http://localhost:8080 (override with --url).

3. Use the OpenAI-compatible API

On the box, the API is keyless:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3-4b-int4", "messages": [{"role": "user", "content": "Hello!"}]}'

To reach your GPU remotely through the cloud relay, sign in at baremetalrt.ai/app, mint a bmrt_ key in Account Settings, and point any OpenAI- or Anthropic-compatible client (Python openai SDK, Continue, Cursor, curl) at https://baremetalrt.ai/v1:

curl https://baremetalrt.ai/v1/chat/completions \
  -H "Authorization: Bearer bmrt_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3-4b-int4", "messages": [{"role": "user", "content": "Hello!"}]}'

Technical Details

• FP32 precision correctness — custom CUDA kernel performs AllReduce on-GPU in FP32, achieving the theoretical floor of IEEE 754 arithmetic. 2,500x more accurate than FP16. Identical to NCCL on NVLink.
• Asymmetric-tolerant transport — GPUs with different VRAM and compute capabilities participate in the same AllReduce without barrier stalls. The slower GPU sets the pace; the faster GPU waits on a non-blocking receive.
• TensorRT plugin integration — custom IPluginV2DynamicExt plugins intercept every AllReduce/AllGather call at execution time, replacing NCCL with our TCP transport without modifying TRT-LLM's model definitions.
• Overlapped AllReduce — TCP recv runs in a background thread during GPU sync wait. When sync_wait > recv_time, the network transfer adds zero time to the critical path.
• Double-buffered pinned memory — four page-locked host buffers alternate between consecutive AllReduce calls, preventing data races between in-flight transfers.
• TensorRT-LLM on Windows — full native port of NVIDIA's inference engine (Conan profiles, FMHA kernels, nanobind bindings, MSVC/CUDA interop). No WSL, no Docker.

See Architecture for the full system design, or read the technical paper.

What's in This Repo

This is the public product repo — the server, web UI, installer, and documentation.

baremetalrt/
├── server/        # FastAPI server (auth, chat relay, node management)
├── web/           # Product web app (chat UI, account, downloads)
├── site/          # Landing page and demo
├── installer/     # Windows installer (WiX/Burn MSI, code-signed)
├── cli/           # bmrt CLI (pip install baremetalrt)
├── docs/          # Documentation
│   ├── ARCHITECTURE.md
│   ├── API.md
│   ├── QUICKSTART.md
│   └── MISSION.md
└── paper/         # Technical paper (arXiv-ready LaTeX)

The inference engine, transport layer, and daemon are in a separate private repository.

API

BareMetalRT exposes an OpenAI-compatible API. See API docs or the API reference.

Current Status

v0.21.19 — Changelog

• Single-GPU and cross-machine TP=2 multi-GPU inference on Windows — native, no WSL, no Docker
• Native serving with clean VRAM management, real batching/concurrency, and worker self-heal
• 24 ready-to-run models in the shipping catalog — Qwen 3, Llama 3.x, Mistral, DeepSeek-R1 distills, Phi, Gemma, plus a Qwen2-VL vision model — int4 (W4A16-AWQ) and FP16, from 0.6B to 8B on a single consumer card
• Reliable tool-calling / agent loop, an MCP host, skills, workflows, and routines
• Built-in voice (STT + TTS; runs on 8 GB cards via CPU-mode Whisper)
• Vision (attach images), local RAG with cited answers, and cross-session agent memory
• Local coding agent with a VS Code-class editor and whole-repo map
• Presence — a photoreal voice avatar (preview)
• Egress Mode — seal a node fully air-gapped, with offline Ed25519 licensing
• Web chat UI with streaming; OpenAI- and Anthropic-compatible APIs
• Code-signed MSI installer with automatic GPU claiming
• User accounts: email + password, Google/GitHub OAuth, 2FA, enterprise SSO (OIDC/SAML) + SCIM
• Chat history encrypted on-device — never stored on our servers

Roadmap

• TP=4+ — scale beyond two GPUs using ring AllReduce (transport implemented, untested beyond TP=2)
• Mixture-of-Experts — replace dense AllReduce with sparse expert routing. 2 of 8 experts per token = 75% of the mesh available for concurrent serving. See Architecture § Future.
• Distributed KV cache — page KV entries to peer GPU VRAM across the mesh, enabling 128K+ context on consumer hardware
• Asymmetric weight splitting — proportional column assignment based on per-GPU VRAM

Security

Found a vulnerability? See SECURITY.md for our responsible disclosure policy.

License

BareMetalRT is proprietary software. See LICENSE for terms.