KubeGuard CLI — Intelligent Kubernetes Workload Observatory & Risk Engine

KubeGuard AI is an AI-powered Kubernetes workload health monitoring, predictive risk calculation, and automated disaster recovery platform.

The kubeguard-cli package (kubeguard) is the official operator command-line tool for installing, inspecting, diagnosing, managing model lifecycles, and orchestrating backup & disaster recovery operations across Kubernetes clusters and multi-cluster fleets.

---

What is KubeGuard AI?

Traditional Kubernetes monitoring tools are strictly reactive—firing alerts only after a pod crashes (CrashLoopBackOff), exhausts memory (OOMKilled), or saturates node CPU.

KubeGuard AI bridges the gap between observability and reliability engineering:

• Predictive Risk Scoring (0–100): Continuously observes memory trend slopes, CPU degradation patterns, and restart velocity to forecast workload failures before they impact availability.
• Incident Correlation Engine: Groups fragmented Prometheus alert storms, metric anomalies, and ML signals into unified, contextualized incident timelines.
• Closed-Loop Auto-Healing: Executes policy-bounded remediation actions (RESTART_POD, ROLLOUT_RESTART, POD_EVICT) with strict rate limiting, cooldown protection, and Pod Disruption Budget (PDB) enforcement.
• Enterprise ML Model Governance: Provides candidate model validation, shadow canary isolation, cryptographic checksum verification, and instant rollback.
• Automated Backup & Disaster Recovery (Phase 35): Delivers transactional database backup creation, online WAL snapshotting, SHA-256 package verification, retention pruning, and off-site synchronization.

---

Architecture Overview

KubeGuard operates as a non-invasive, security-hardened service inside your Kubernetes cluster. All CLI interactions connect through authenticated endpoints or transient kubectl tunnels—requiring zero public ingress exposure.

 ┌─────────────────────────────────────────────────────────────────────────────┐
 │                         KUBERNETES CLUSTER / FLEET                          │
 │                                                                             │
 │  ┌───────────────────────┐          ┌────────────────────────────────────┐  │
 │  │ Prometheus Monitoring │─────────▶│    KubeGuard Prediction Service    │  │
 │  │  - CPU & Memory Slopes│          │  - Ingestion & Feature Engineering │  │
 │  │  - Alertmanager Feeds │          │  - AI/ML Risk Engine (0-100 Score) │  │
 │  └───────────────────────┘          │  - Incident Correlation Engine     │  │
 │                                     │  - Model Registry & Canary Engine  │  │
 │  ┌───────────────────────┐          │  - Multi-Action Auto-Healing       │  │
 │  │  Cluster Workloads    │◀─────────│  - Disaster Recovery Engine        │  │
 │  │  - Pods & Deployments │ (Actions)└────────────────────────────────────┘  │
 │  │  - Namespaces & PDBs  │                            │                     │
 │  └───────────────────────┘                            ▼                     │
 │                                             ┌──────────────────┐            │
 │                                             │ Persistent Store │            │
 │                                             │ - SQLite / PG    │            │
 │                                             │ - Model Artifacts│            │
 │                                             └──────────────────┘            │
 └───────────────────────────────────────────────────────▲─────────────────────┘
                                                         │
                                               Transient Tunnel
                                            (or In-Cluster Network)
                                                         │
                                           ┌───────────────────────────┐
                                           │       KubeGuard CLI       │
                                           │   (pip: kubeguard-cli)    │
                                           │                           │
                                           │  - kubeguard status       │
                                           │  - kubeguard pods         │
                                           │  - kubeguard incidents    │
                                           │  - kubeguard model        │
                                           │  - kubeguard backup       │
                                           │  - kubeguard fleet        │
                                           └───────────────────────────┘

Core Architecture Pillars

1. Prediction & Ingestion Pipeline: Scans cluster metrics at regular intervals, evaluates trend slopes over time windows, and calculates multidimensional risk scores.
2. Persistence & Disaster Recovery: Stores telemetry observations, incident records, model metadata, and governance audits in an ACID-compliant database with online WAL snapshot capabilities.
3. Model Governance & Registry: Tracks model versions, ensures reproducibility, validates SHA-256 artifacts, and supports atomic rollbacks.
4. Transient Tunneling Mechanism: The CLI communicates directly with cluster services via encrypted, transient kubectl port-forward sessions, ensuring no external endpoints are exposed.

---

Requirements

Requirement	Supported Versions	Purpose
Python	3.9, 3.10, 3.11, 3.12, 3.13	CLI runtime
kubectl	1.24+	Cluster communication & secure port-forwarding
Helm	3.8+	Kubernetes deployment management (optional)
Prometheus	Any (v2+)	Cluster telemetry source (kube-prometheus-stack)

---

Installation

Install kubeguard-cli from PyPI using pip:

pip install kubeguard-cli

Verify the installation:

kubeguard --help
kubeguard version

---

Global Options

The following flags can be passed to all kubeguard commands:

Option	Environment Variable	Default	Description
--context TEXT	KUBEGUARD_CONTEXT	Current context	Target Kubernetes cluster context
--namespace TEXT, -n	KUBEGUARD_NAMESPACE	kubeguard	Namespace where KubeGuard is deployed
--token TEXT	KUBEGUARD_API_TOKEN	None	API authentication token (if enabled)

Example:

kubeguard --context kind-kubeguard --namespace kubeguard status

---

Command Reference & Usage Examples

1. Installation & Cluster Status

# Display overall health, pod count, active model, and deployment state
kubeguard status

# Output status in machine-readable JSON format
kubeguard status --json

# Install KubeGuard via Helm into the connected cluster
kubeguard install --namespaces "default,production" --interval 30

2. Diagnostics & Service Health

# Run comprehensive pre-flight and runtime diagnostics
kubeguard diagnostics
kubeguard diagnostics --json

# Check endpoint connectivity and worker process responsiveness
kubeguard health

3. Workload Risk Scoring & Telemetry

# Inspect all monitored pods and operational risk scores (0–100)
kubeguard pods

# Filter workloads by namespace or risk threshold
kubeguard pods --namespace production --risk high
kubeguard pods --json

4. Active Alerts & Correlated Incidents

# View active Prometheus and Alertmanager alerts affecting monitored pods
kubeguard alerts

# List correlated multi-signal incidents
kubeguard incidents list

# Inspect detailed incident context, signals, and root cause timeline
kubeguard incidents detail --incident-id INC-20261010-001

5. ML Model Registry & Rollback

# View active model version, SHA-256 checksum, and training metadata
kubeguard model status

# List version history and candidate validation records
kubeguard model list

# Instantly trigger an emergency rollback to a known-stable model version
kubeguard model rollback --target-version 55

6. Auto-Healing & Remediation Controls

# View auto-healing engine configuration, dry-run state, and safety bounds
kubeguard healing status

# Review audit trail of executed remediation attempts and cool-down timers
kubeguard healing history

7. Database Backup & Disaster Recovery (Phase 35)

# 1. Create a verified, atomic backup package (database + model artifacts)
kubeguard backup create --output-dir /data/backups --db-path /data/kubeguard.db

# Create backup with retention policy and off-site sync
kubeguard backup create \
  --output-dir /data/backups \
  --retention-count 7 \
  --external-sync-command "aws s3 cp {archive_path} s3://my-backup-bucket/kubeguard/"

# 2. Verify SHA-256 package integrity and manifest checksums
kubeguard backup verify --archive /data/backups/kubeguard-backup-1791798652.zip
# Or using positional argument:
kubeguard backup verify /data/backups/kubeguard-backup-1791798652.zip

# 3. Restore database and model state (requires explicit confirmation flag)
kubeguard backup restore \
  --archive /data/backups/kubeguard-backup-1791798652.zip \
  --db-path /data/kubeguard.db \
  --force

8. Multi-Cluster Fleet Governance

# List all registered edge clusters in the fleet
kubeguard fleet list

# Inspect fleet heartbeats, sync states, and aggregated incidents
kubeguard fleet status

9. Clean Uninstallation

# Remove KubeGuard Helm release and associated runtime resources
kubeguard uninstall --yes

---

Security & Safety Guardrails

• Zero Ingress Exposure: Uses standard, temporary kubectl port-forwarding rather than requiring public LoadBalancers or NodePorts.
• Disaster Recovery Isolation: Restore operations validate package signatures and reconstruct staging databases before touching destination targets.
• Credential Protection: Shell arguments, tokens, and authorization headers are sanitized and redacted from all command logs.
• Destructive Action Confirmation: High-impact commands such as restore and uninstall strictly enforce --force or explicit operator confirmation flags.

---

License

This project is licensed under the MIT License — Copyright (c) KubeGuard AI Team.