Foretoken command-line tool

English | 简体中文

The Foretoken command-line tool installs the shared Kubernetes platform, deploys model services from Kustomize configurations, reports serving readiness, resolves frontend URLs, and runs benchmarks through one foretoken entry point.

Before you start

You need Python 3.11 or later, an active Kubernetes context, kubectl, and Helm. GPU nodes must already have their vendor driver and Kubernetes device plugin.

Install the command-line tool

Install the published command-line tool with pip:

pip install foretoken

# From a source checkout:
# pip install -e .

Or create and activate a virtual environment with uv:

uv venv
source .venv/bin/activate
uv pip install foretoken

Run foretoken --version to check the installed CLI version.

Create a local cluster

On a Linux GPU host with Docker, NVIDIA Container Toolkit, and k3d installed:

# Name the local cluster and use GPU index 0 from nvidia-smi.
# To use two GPUs, pass --gpus 0,1.
foretoken cluster create k3d --name foretoken-dev --gpus 0

For a local kind development cluster:

foretoken cluster create kind --name foretoken-dev

Remove a cluster created by the CLI with:

foretoken cluster delete k3d --name foretoken-dev

Install the Kubernetes platform

foretoken install installs the Foretoken CRDs and controller in the active Kubernetes context. Platform resources use the foretoken-platform namespace. The command also configures monitoring and, in Gateway mode, the Gateway resources. Deploy model services separately with foretoken deploy.

Default installation

The default uses release images and local access through a LoadBalancer Service:

foretoken install

Installation selects the NVIDIA or MetaX runtime and automatically reuses or installs LeaderWorkerSet and the shared RDMA device plugin. Explicit runtime settings in --values take precedence; in a mixed-GPU cluster, select a resource with runtime.vllm.gpu.resourceName or restrict the nodes with runtime.vllm.gpu.nodeSelector.

Log collection and persistence are enabled by default. See Observability for configuration, log queries, dashboards, and alerts.

Gateway mode

Gateway mode creates a dedicated GatewayClass and Gateway, installing Envoy Gateway if no compatible controller is available:

foretoken install --frontend-mode gateway

With another Gateway Controller, reuse a Gateway managed by that controller:

foretoken install \
  --frontend-mode gateway \
  --gateway-name inference-gateway \
  --gateway-namespace gateway-system

Add --gateway-section-name LISTENER only when more than one listener matches.

Current source

Build and install from the repository root. The cluster needs a default StorageClass for compiler caches; see the source deployment guide for storage overrides.

foretoken install -e .

This builds the platform in dedicated Pods and binds the checkout to the target cluster.

After editing it, use foretoken deploy to redeploy source changes. Use --engine-source PATH to also bind a vLLM engine checkout.

A standard active kind or k3d context loads the built images directly into its nodes. Other Kubernetes contexts need a registry reachable by the Build Pods and nodes. For an internal registry without authentication:

foretoken install -e . --registry registry.example.com:5000/foretoken

If the registry requires authentication, follow the source deployment guide before installation.

Model distribution

To share public model downloads between nodes through Dragonfly, save this in deploy/platform-values.yaml:

modelDistribution:
  dragonfly:
    enabled: true

For a published platform installation, apply the values with:

foretoken install --values deploy/platform-values.yaml

For a source installation, run foretoken install -e . --values deploy/platform-values.yaml from the repository root, retaining the original registry and engine-source options.

Installation prepares Dragonfly or reuses an existing installation. Models that require authentication and custom model endpoints download directly from their provider. To select a particular Dragonfly Helm release, set existingRelease: {name: dragonfly, namespace: dragonfly-system} under modelDistribution.dragonfly.

On NVIDIA clusters with RDMA, ModelExpress can load weights from running replicas. Add this alongside dragonfly to enable it:

modelDistribution:
  modelexpress:
    enabled: true

Automatic weight transfer uses remote models with a persistent cache, data parallelism of one, and fixed expert placement. An explicit load-format remains unchanged. Each GPU worker selects a nearby available RDMA interface; replicas without a compatible source load the prepared files.

Reapply the installation command after changing either setting. Set enabled: false to disable it. foretoken uninstall removes managed Dragonfly resources once no workloads use them; reused installations are retained.

Installation options

Use --values only to override platform image, runtime, or hardware settings. Without an override, installation compares supported public sources for default platform images and OCI charts. Use --oci-registry to select a registry explicitly; image references supplied through values remain unchanged. Source selection runs on the CLI host, so the selected registry must also be reachable from the cluster nodes.

Model services are reached through an IP address outside the cluster. k3d, k3s, and cloud clusters assign one automatically. Clusters built with kubeadm, RKE2, or kubespray have no address assignment by default, so installation there ends with LoadBalancer support Not verified. Give Foretoken a range of unused addresses in the nodes' subnet, confirmed with the cluster administrator, and it assigns them to services:

loadBalancer:
  managedAddresses:
    - 192.168.1.240-192.168.1.250

Deploy and operate model services

Run from the repository checkout prepared in the Quick Start. Deploy one frontend and all models rendered by a Kustomize root.

See the multi-model example for resources and model storage for directory or PVC configuration. Use examples/quickstart for a single model.

foretoken deploy examples/multi-model-quickstart --timeout 20m

The command applies the configuration, shows service status, and streams Pod and container logs with source prefixes while waiting. It exits when every service reports Ready and its selected alerts are configured. Without --timeout, it waits up to ten minutes. Configure service alerts in the Kustomize deployment; see service observability.

Inspect the same deployment without applying it:

foretoken status examples/multi-model-quickstart

Inspect every Foretoken service in a namespace. With --watch, follow service state changes and Pod/container logs until Ctrl+C:

foretoken status -n foretoken-multi-model-demo
foretoken status -n foretoken-multi-model-demo --watch

Resolve the public frontend URL after deployment:

FORETOKEN_FRONTEND_URL="$(foretoken endpoint examples/multi-model-quickstart)"

For an HTTP Gateway, resolve its request Host separately:

FORETOKEN_REQUEST_HOST="$(foretoken endpoint examples/multi-model-quickstart --host)"

--host returns the host and optional port for direct access, or the configured routing hostname for an HTTP Gateway. foretoken endpoint waits for the LoadBalancer or Gateway address; use foretoken deploy to wait for the services to become ready.

Measure serving performance

Use foretoken perf to measure response latency and request or token throughput. Pass a Kustomize directory, or --url with --model for an existing endpoint. Choose a workload in Performance examples.

Evaluate and compare models

Use foretoken eval to score model answers with lm-evaluation-harness or EvalScope. It accepts the same service selection options; task and scoring parameters use the selected framework's syntax. See Quality evaluation. Add --reference to compare a candidate's probabilities against a reference.

Export figures

Use --output local,wandb,plot with a benchmark, or foretoken plot RESULT_DIR to redraw a saved run or sweep without running inference. Export options and comparisons are in parameter sweeps.

Find execution bottlenecks

Add --profile to foretoken deploy or foretoken perf to record CPU/GPU execution, then browse captures with foretoken profile view. Setup and commands are in Profiling.

Clean up

Delete the deployed services before uninstalling the platform:

foretoken delete examples/multi-model-quickstart
foretoken uninstall

CRDs and reused cluster components are retained. Managed LeaderWorkerSet and MetalLB controllers are also retained while workloads still depend on them.