constella-gpu 0.1.0

Lightweight realtime NVIDIA GPU and Ascend NPU cluster monitoring service.

Constella

Lightweight GPU Cluster Monitoring & Workload History

Monitor today. Review tomorrow.

English | 简体中文

Like stars in a constellation, Constella gathers independent GPU nodes into one observable cluster.

Constella is a lightweight GPU monitoring platform for labs, AI teams, and personal GPU servers.

Unlike terminal tools that only show the current state, Constella automatically records GPU workload history, making it easy to review completed training and inference jobs. It supports standalone servers and small GPU clusters without requiring a heavyweight Prometheus/Grafana stack.

Screenshots

Cluster Overview	GPU & Process Detail

Workload Curves

Features

Workload History

• Automatically record GPU curves for completed workloads.
• Review training and inference jobs from the last 7 days.
• Prefer high-resolution memory cache for recent short jobs, with SQLite rollups for persisted history.

GPU Monitoring

• Monitor a standalone server or a small GPU cluster from one Web UI.
• Track GPU utilization, memory, power, temperature, clocks, processes, users, PIDs, and command fingerprints.
• Use NVML first and fall back to nvidia-smi when needed.

Multi-User Analytics

• See user GPU usage rankings, job duration rankings, node trends, and range-aware heatmaps.
• Detect low-utilization reservations and off-hour activity.
• Keep realtime monitoring available even when historical analytics are disabled.

Lightweight Deployment

• No root privileges, system service, Prometheus, or Grafana required.
• One manager process receives data from local and remote GPU agents.
• Remote GPU nodes only need Python, NVIDIA drivers, and SSH access.

Why Constella?

Capability	nvitop	Prometheus/Grafana	Constella
Realtime GPU view	Yes	Yes	Yes
Workload history	No	Requires setup	Yes
Small cluster view	Limited	Yes	Yes
Lightweight setup	Yes	No	Yes
Web UI	No	Yes	Yes
User/job analytics	No	Custom dashboards	Built in

Constella sits between terminal monitoring and a full observability stack: more historical and shareable than nvitop, but much lighter to deploy than Prometheus/Grafana for a small lab.

Quick Start

Install the PyPI distribution and start the managed local stack:

pip install constella-gpu
constella service start

On an Ascend host, use constella service start --device ascend.

From a source checkout, start the manager and local accelerator agent with:

cd Constella
./scripts/service/setup.sh
./scripts/service/start.sh

For an Ascend host, select the hardware backend explicitly:

./scripts/service/start.sh --device ascend

Open:

http://127.0.0.1:8765/overview

If the service runs on a remote server, forward the port from your local machine:

ssh -N -L 8765:127.0.0.1:8765 <user>@<server>

Enable SQLite history when workload history and analytics are needed:

DB_PATH=run/constella.db ./scripts/service/start.sh

Start the high-resolution sidecar when short-job curve cache should run outside the manager process:

DB_PATH=run/constella.db HIGHRES_SIDECAR=1 ./scripts/service/start.sh

The sidecar listens on 127.0.0.1:8766 by default and subscribes to the manager stream at ws://127.0.0.1:8765/api/highres/stream. Simple deployments can skip the sidecar; the manager still exposes the built-in /api/highres/* endpoints.

Cluster Mode

Prepare the remote node manifest:

cp docs/nodes.example.yaml nodes.yaml

Edit manager_url, manager_hostname, and each node's device (nvidia or ascend), then configure passwordless SSH from the manager host to each node.

flowchart LR
  M["Manager<br/>FastAPI + Web UI"] -->|"SSH setup/control"| A["gpu-node-a<br/>agent"]
  M -->|"SSH setup/control"| B["gpu-node-b<br/>agent"]
  M -->|"SSH setup/control"| C["gpu-node-c<br/>agent"]
  A -->|"WebSocket samples"| M
  B -->|"WebSocket samples"| M
  C -->|"WebSocket samples"| M

Start remote GPU agents:

./scripts/cluster/start.sh

• scripts/service/start.sh creates run/agent-token on first local-agent startup, and scripts/cluster/start.sh uses that token for remote agents.
• If the manager host should not monitor local GPUs, start with LOCAL_AGENT=0.
• Remote nodes do not need uv; the manager syncs a minimal agent runtime.

Ascend NPU support

Constella uses independent, explicitly selected hardware chains:

• nvidia: NVML, then nvidia-smi fallback.
• ascend: DCMI (libdcmi.so), then npu-smi fallback.

The DCMI backend exposes AICore and HBM utilization, memory, temperature, power, PCI identity, driver/DCMI versions, and running-process memory. Multi-die cards remain visible as one device card per die. The API includes card_id, die_id, card_count, and accelerator_count; rated power and live power are counted once per physical card, while duplicate PIDs across dies are counted as one active process.

Architecture

flowchart LR
  LA["Local agent<br/>selected device chain"] -->|"WS /api/agents/ws"| M["Manager<br/>FastAPI ingest"]
  RA["Remote agents<br/>NVML→nvidia-smi or DCMI→npu-smi"] -->|"WS /api/agents/ws"| M
  M --> S["ClusterState<br/>latest snapshots + 120-point history"]
  S --> API["HTTP /api/cluster/snapshot"]
  S --> WS["WebSocket /ws/cluster"]
  S -.optional.-> DB["SQLite<br/>rollups + sessions"]
  DB -.optional.-> AN["Analytics + job curves"]
  S -.optional.-> HR["Highres cache / sidecar"]
  API --> UI["Vite TypeScript UI"]
  WS --> UI
  AN --> UI
  HR --> UI

The manager does not sample GPUs directly. Local and remote nodes both report current sample points through the same agent WebSocket path. SQLite, analytics, and high-resolution job curves are optional side paths and do not block realtime snapshots. See Design for the full data flow.

Docs

• Design: architecture, data path, low-overhead strategy, and data contracts.
• Operations: startup, access, cluster agent management, status, and verification commands.
• SQLite History: persistence, rollups, maintenance, and job curves.
• Cloudflare Tunnel: domain access without opening an inbound server port.
• Node manifest example: nodes.yaml template for remote agents.
• PyPI CLI: installed service, probe, agent, and cluster commands.
• Packaging: build and safely smoke-test wheel and source distributions.
• Scripts: service, cluster, tunnel, maintenance, and dev script entry points.

Project Layout

src/constella/          Python backend, agents, cluster manager, NVML/DCMI samplers, API/WebSocket
frontend/               Vite + TypeScript frontend
scripts/                categorized service, cluster, tunnel, maintenance, and dev scripts
docs/                   design and operations notes
tests/                  unit tests

Development

uv sync
uv run pytest

cd frontend
npm install
npm run build

Frontend dev server:

cd frontend
npm run dev

For production, build frontend/dist; FastAPI serves the static frontend directly.

API

• GET /api/health
• GET /api/cluster/snapshot
• GET /api/settings
• PATCH /api/settings
• WS /ws/cluster
• WS /api/agents/ws
• GET /api/history/gpu
• GET /api/history/tasks
• GET /api/users
• GET /api/analytics/overview
• GET /api/analytics/node/{node_id}
• GET /api/highres/status
• GET /api/highres/jobs
• GET /api/highres/jobs/{job_key}
• GET /api/highres/jobs/{job_key}/gpu
• GET /api/docs

When SQLite is not enabled, history, analytics, and job curve search APIs return enabled:false; realtime cluster monitoring continues through /api/cluster/snapshot and /ws/cluster.

License

MIT