open-jet 0.1.12

Offline local-LLM terminal app for Jetson and edge Linux: chat with on-device models, run agent tools, and manage context safely.

OpenJet

open-jet is an offline-first agent runtime for Jetson-class and other edge Linux systems with tight memory budgets.

It is built for cases where the hard part is not just running a local model, but keeping the agent useful under constrained RAM, limited context windows, interrupted sessions, and hardware-specific failure modes.

Quickstart

pip install open-jet

OpenJet does not bundle an inference runtime. By default it uses llama-server from llama.cpp, though setup can also configure SGLang or TensorRT-LLM instead.

For the default llama.cpp path, you need:

1. the open-jet Python package
2. a working llama-server binary
3. a local model file such as a .gguf

Then run:

open-jet --setup
open-jet

Installation

1. Install the Python package

pip install open-jet

2. Install llama-server from llama.cpp

llama.cpp setup is required for the default OpenJet runtime. If you choose SGLang or TensorRT-LLM in open-jet --setup, follow the runtime-specific model/runtime setup for those backends instead.

Jetson:

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
mkdir build && cd build

cmake .. \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=87 \
  -DGGML_CUDA_FA_ALL_QUANTS=ON

cmake --build . --target llama-server -j$(nproc)

Linux x86 + NVIDIA:

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
mkdir build && cd build

cmake .. -DGGML_CUDA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON
cmake --build . --target llama-server -j$(nproc)

CPU-only:

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
mkdir build && cd build

cmake ..
cmake --build . --target llama-server -j$(nproc)

OpenJet looks for llama-server on PATH first, then at ~/llama.cpp/build/bin/llama-server.

3. Have a local model ready

You need a model reference for the selected runtime:

• llama.cpp: typically a local .gguf file, or an Ollama-backed model selected during setup
• SGLang: a local model directory or supported HF model id
• TensorRT-LLM: a local model directory or supported HF model id

4. Run setup

open-jet --setup

The setup flow guides you through:

1. hardware detection/profile
2. model source selection
3. model path or download choice
4. context window size
5. GPU offload configuration

After setup:

open-jet

What it provides

• bounded-context local chat with your on-device model
• automatic context condensing under pressure
• session resume and harness state recovery
• OpenTelemetry instrumentation with collector export
• hardware-aware runtime setup for Jetson and edge Linux
• controlled tool use and slash commands
• Python SDK access to the same backend

Why OpenJet exists

Most local LLM tools stop at “chat with a model on your box.” That breaks down on edge hardware.

OpenJet is built around:

• limited prompt memory on-device
• interrupted work and session recovery
• deterministic tool and approval boundaries
• real traces for evaluation
• reliable operator workflows on constrained local models

Docs

• Quickstart
• Installation
• Runtime: llama.cpp
• Runtime: SGLang
• Runtime: TensorRT-LLM
• Usage: CLI
• Usage: Slash commands
• Usage: Workflow harness
• Usage: Session state and logging
• Telemetry
• Python SDK
• Deployment: Jetson
• Deployment: Linux x86 + NVIDIA
• Deployment: CPU-only
• Configuration
• Licensing

License

AGPL-3.0-only, with commercial licensing available under separate terms.