guardops 0.8.0

Autonomous DevSecOps CLI — build, scan, deploy, and monitor runtime security

GuardOps

Production-grade DevSecOps CLI. Build, scan, deploy, monitor runtime security, and self-heal with gates at every stage.

GuardOps wraps a complete secure delivery pipeline behind a single command. Given any application repo, it builds a Docker image, runs four security scanners in sequence, deploys to Kubernetes via Helm, runs a post-deploy DAST scan, exposes live metrics to Prometheus, queries runtime security alerts from Loki, and automatically quarantines compromised pods via Alertmanager webhooks — blocking the pipeline if HIGH or CRITICAL findings are detected at any stage.

guardops deploy --env prod

That one command: builds a multi-stage Docker image, runs Semgrep + Bandit + Trivy + SonarQube, pushes to ECR, deploys to EKS via Helm with automatic rollback on failure, runs OWASP ZAP DAST against the live application with auto-rollback on CRITICAL findings, and exposes /metrics to a live Grafana dashboard.

guardops runtime-status

Queries Loki for Falco-format runtime security alerts (shell spawns, sensitive file reads, package manager execution, /etc writes) and renders a severity-sorted table. Use --fail-on CRITICAL as a post-deploy CI gate.

guardops quarantine-status

Shows pods currently isolated by the Phase 8 self-healing system — active NetworkPolicies, quarantined pod names, triggering Falco rule, and age. Use --release <pod> to manually lift a quarantine after investigation.

---

Current Status — v0.8.0

Phase	Version	Status	What was built
1 — CLI + Local Deploy	v0.1.0	✅ Done	Click CLI framework, k3d local deploy, Docker build
2 — Security Scanning + CI	v0.2.0	✅ Done	Semgrep, Bandit, Trivy, SonarQube, 154 tests, GitHub Actions
3 — Helm + EKS Infrastructure	v0.3.0	✅ Done	Helm deploy, rollback command, Terraform VPC/EKS/IAM/ECR/S3
4 — Full AWS Pipeline	v0.4.0	✅ Done	ECR push, EKS deploy, S3 report upload, cost-optimised infra
4.5 — Reliability Hardening	v0.4.1	✅ Done	Remote Terraform state (S3+DynamoDB), multi-stage Docker build, subprocess timeout+encoding fixes, Trivy DB cache in CI
5 — Observability	v0.5.0	✅ Done	Prometheus + Grafana via kube-prometheus-stack, /metrics endpoint, ServiceMonitor, EBS CSI driver, custom dashboards
6 — Security Hardening	v0.6.1	✅ Done	GitHub OIDC replaces IAM user (no more static keys), DAST via OWASP ZAP with post-deploy scan and auto-rollback on CRITICAL
7 — Runtime Security	v0.7.0	✅ Done	Loki + Promtail log pipeline, Falco-format alert ingestion, guardops runtime-status, CI runtime gate
8 — Self-Healing	v0.8.0	✅ Done	Alertmanager webhook handler, automatic NetworkPolicy quarantine on CRITICAL Falco alert, guardops quarantine-status, Terraform alertmanager-webhook module

---

Roadmap

Phase	Target	What it adds
9 — Multi-Environment	v0.9.0	Staging + prod namespaces, blue-green deploy strategy, guardops switch --slot green
10 — Full Production	v1.0.0	Real domain + TLS via cert-manager, ArgoCD GitOps, runbook documentation

---

Architecture

Developer
    |
    v
guardops deploy
    |
    +-- Step 1: Docker Build ──────────────────────────+
    |       Multi-stage build (builder + runtime)       |
    |       pip/wheel absent from final image           |
    |       Non-root user (UID 10001), no shell         |
    |                                                   |
    +-- Step 2: Security Scans (SAST) ─────────────────+
    |       Semgrep    (SAST, code patterns)            |
    |       Bandit     (Python-specific vulns)          |
    |       Trivy fs   (secrets, IaC misconfigs)        |
    |       Trivy img  (CVEs in OS + deps)              |
    |       SonarQube  (quality gate, optional)         |
    |                                                   |
    |       BLOCKED if any finding >= HIGH              |
    |       Report written to security/reports/         |
    |                                                   |
    +-- Step 3: Registry Push ─────────────────────────+
    |       local: k3d image import                     |
    |       prod:  docker push -> AWS ECR               |
    |                                                   |
    +-- Step 4: Helm Deploy ───────────────────────────+
    |       helm upgrade --install --atomic             |
    |       local: k3d + values.yaml                   |
    |       prod:  EKS + values-prod.yaml              |
    |       Automatic rollback on timeout or error      |
    |                                                   |
    +-- Step 5: DAST (Phase 6) ────────────────────────+
            OWASP ZAP baseline scan (passive)
            Target: live deployed application
            BLOCKED + auto-rollback if CRITICAL found
            Report written to security/reports/
            Skipped for local env (no stable URL)

guardops runtime-status (Phase 7)
    |
    +-- Queries Loki HTTP API ({app="falco"} | json)
    +-- Normalises Falco priority -> CRITICAL/HIGH/MEDIUM/LOW
    +-- Renders severity-sorted Rich table in terminal
    +-- --fail-on CRITICAL exits 1 for CI gate use

guardops quarantine-status (Phase 8)
    |
    +-- kubectl get networkpolicy -l guardops.io/managed-by=guardops
    +-- kubectl get pods -l guardops.io/quarantine=true
    +-- Renders locked-pod table + active NetworkPolicy table
    +-- --release <pod> lifts quarantine manually

Self-Healing Pipeline (Phase 8)

CRITICAL Falco alert fires (shell spawn, etc.)
    |
    v
Alertmanager receives alert from Prometheus
    |
    v
POST /webhook -> guardops-alertmanager-webhook pod
    |
    v
alertmanager_handler.py
    +-- kubectl label pod <pod> guardops.io/quarantine=true
    +-- kubectl apply NetworkPolicy (deny-all ingress, DNS-only egress)
    |
    v
Pod isolated — no inbound traffic, no outbound except port 53
    |
    v
Alert resolves (Falco stops firing)
    |
    v
POST /webhook status=resolved
    +-- kubectl delete networkpolicy guardops-quarantine-<fp8>
    +-- kubectl label pod <pod> guardops.io/quarantine-
    |
    v
Pod released — network restored

Infrastructure (AWS, Terraform-managed)

ap-south-1 (Mumbai)
+------------------------------------------------------------------+
|  VPC  10.0.0.0/16                                                |
|                                                                  |
|  Public Subnets (ap-south-1a, ap-south-1b)                      |
|    NAT Gateways, Load Balancers                                  |
|                                                                  |
|  Private Subnets (ap-south-1a, ap-south-1b)                     |
|    EKS Managed Node Group (t3.large)                             |
|    +-- guardops-app Pods (x2)                                    |
|    |    +-- /healthz, /ready, /metrics endpoints                 |
|    |    +-- port 8080, non-root UID 10001                        |
|    |    +-- capabilities.drop ALL                                |
|    |                                                             |
|    +-- monitoring namespace                                      |
|         +-- Prometheus  (kube-prometheus-stack)                  |
|         +-- Grafana     (pre-loaded dashboards)                  |
|         +-- Alertmanager                                         |
|         +-- kube-state-metrics, node-exporter                    |
|         +-- Loki        (log aggregation, 10Gi EBS)  [Phase 7]  |
|         +-- Promtail    (log shipping DaemonSet)      [Phase 7]  |
|         +-- falco-simulator CronJob (every 3 min)    [Phase 7]  |
|         +-- guardops-alertmanager-webhook pod         [Phase 8]  |
|              +-- /healthz, /readyz, /webhook endpoints           |
|              +-- ServiceAccount + ClusterRole (RBAC)             |
|                                                                  |
|  ECR: guardops-app (scan-on-push, 10-image lifecycle)            |
|       guardops-app:webhook-latest (handler image)    [Phase 8]  |
|  S3:  guardops-reports-* (scan reports, versioned)               |
|  S3:  guardops-tfstate-* (Terraform remote state)               |
|  DynamoDB: guardops-tf-lock (state locking)                      |
|  IAM: github-actions-role (OIDC, no static keys)                 |
+------------------------------------------------------------------+

CI/CD Pipeline (GitHub Actions)

Push to main
    |
    v
Job 1: build-test
    pytest (220+ tests) + ruff + mypy
    |
    v
Job 2: sast
    Semgrep + Bandit — gates on HIGH+
    |
    v
Job 3: container-scan
    Docker build (multi-stage) + Trivy (cached DB)
    Gates on fixable HIGH/CRITICAL CVEs
    ECR push via GitHub OIDC (no static IAM keys)
    |
    v
Job 4: deploy              <-- active when HAS_EKS_CLUSTER=true
    helm upgrade --install --atomic --timeout 5m
    kubectl rollout status verify
    OWASP ZAP DAST scan (passive baseline)  <-- Phase 6
    Auto-rollback on CRITICAL DAST findings <-- Phase 6
    |
    v
Job 5: runtime-gate        <-- active when HAS_FALCO_ENABLED=true [Phase 7]
    kubectl port-forward svc/loki 3100:3100
    guardops runtime-status --since 30m --fail-on CRITICAL
    Exits 1 and fails pipeline if CRITICAL alerts found
    |
    v
Job 6: upload-reports      <-- always runs
    Scan artifacts -> S3 bucket
    Path: reports/<repo>/<branch>/<sha>/<run-id>/

---

Install

pip install guardops

Requirements:

• Python 3.11+
• Docker Desktop
• kubectl
• Helm 3.x
• k3d (local deploys) or AWS credentials (prod deploys)

---

Quick Start

# Scaffold config in your project directory
guardops init

# Build, scan, and deploy to local k3d
guardops deploy

# Build, scan, push to ECR, deploy to EKS
guardops deploy --env prod

# Skip SonarQube if not configured
guardops deploy --env prod --skip-sonarqube

# Skip DAST scan (dev only)
guardops deploy --env prod --skip-dast

# View running pod health
guardops status

# Stream pod logs
guardops logs

# Run security scans only (no deploy)
guardops scan

# Roll back to previous Helm revision
guardops rollback

# Roll back to a specific revision
guardops rollback --revision 2

# Check runtime security alerts (Phase 7)
guardops runtime-status

# Filter by time window and severity
guardops runtime-status --since 24h --severity HIGH

# Use as a CI gate (exits 1 if CRITICAL alerts found)
guardops runtime-status --fail-on CRITICAL

# Check quarantine status — Phase 8
guardops quarantine-status -n default

# Check all namespaces
guardops quarantine-status -A

# Release a quarantined pod after investigation
guardops quarantine-status --release <pod-name> --namespace default

---

Observability

Phase 5 adds a full metrics pipeline from application code to Grafana dashboard.

Application metrics (/metrics)

The test app exposes three custom Prometheus metrics:

Metric	Type	Description
guardops_requests_total	Counter	Total HTTP requests, labelled by path and status_code
guardops_request_duration_ms	Gauge	Last request duration per path in milliseconds
guardops_app_info	Info	Static build metadata (environment, version)

Viewing metrics

# Port-forward Grafana
kubectl port-forward svc/kube-prometheus-stack-grafana 3000:80 -n monitoring
# Open http://localhost:3000  (admin / guardops-grafana-2024)

# Port-forward Prometheus
kubectl port-forward svc/kube-prometheus-stack-prometheus 9090:9090 -n monitoring
# Open http://localhost:9090/targets — look for serviceMonitor/default/test-app-guardops-app

# Port-forward Loki (Phase 7)
kubectl port-forward svc/loki 3100:3100 -n monitoring
# Then: guardops runtime-status

# Port-forward Alertmanager (Phase 8)
kubectl port-forward svc/kube-prometheus-stack-alertmanager 9093:9093 -n monitoring
# Open http://localhost:9093 — verify guardops-webhook receiver

# Port-forward webhook handler (Phase 8)
kubectl port-forward svc/guardops-alertmanager-webhook 9095:9095 -n monitoring
# curl http://localhost:9095/healthz  -> {"status":"ok","version":"0.8.0"}
# curl http://localhost:9095/readyz   -> {"status":"ready","kubectl":"..."}

Useful PromQL queries

# Request rate per path (last 5 minutes)
rate(guardops_requests_total[5m])

# Last request latency per path
guardops_request_duration_ms

# App build metadata
guardops_app_info

# Pod memory usage
container_memory_usage_bytes{namespace="default"}

# CPU usage rate
rate(container_cpu_usage_seconds_total{namespace="default"}[5m])

Useful LogQL queries (Loki, Phase 7)

# All Falco alerts
{app="falco"} | json

# CRITICAL only (shell spawns)
{app="falco"} | json | priority="Critical"

# Specific rule
{app="falco"} | json | rule="GuardOps Shell Spawned Inside Container"

# Filter by namespace
{app="falco"} | json | line_format "{{.output}}" | k8s_ns_name="default"

Setup (morning start)

# After terraform apply and kubectl configure:
.\scripts\setup-observability.ps1
.\scripts\setup-runtime-security.ps1   # Phase 7: Loki + Promtail + Falco simulator
# Phase 8 webhook handler deployed automatically by terraform apply
# (enable_self_healing = true in terraform.tfvars)
kubectl apply -f k8s/alertmanager/quarantine-webhook.yaml  # Phase 8: wire Alertmanager

Shutdown (nightly — prevents orphaned EBS volumes)

helm uninstall kube-prometheus-stack -n monitoring
helm uninstall loki -n monitoring
helm uninstall promtail -n monitoring
kubectl delete pvc --all -n monitoring
Start-Sleep -Seconds 30
cd infra/terraform && terraform destroy -auto-approve

---

Runtime Security (Phase 7)

Phase 7 adds a runtime alert pipeline: structured security events are ingested into Loki and surfaced via guardops runtime-status.

How it works

Falco Simulator CronJob (every 3 min)
    └── prints Falco-format JSON to stdout
         └── Promtail DaemonSet tails /var/log/pods/
              └── labels with {app="falco"}, pushes to Loki
                   └── guardops runtime-status queries Loki HTTP API
                        └── FalcoQueryResult -> Rich severity table
                             └── --fail-on CRITICAL -> exit 1 for CI gate

GuardOps Falco Rules (k8s/falco/custom-rules.yaml)

Rule	Falco Priority	GuardOps Severity
Shell Spawned Inside Container	CRITICAL	CRITICAL
Package Manager Executed in Container	ERROR	HIGH
Sensitive File Read in Container	ERROR	HIGH
Write to /etc Inside Container	WARNING	MEDIUM
Container Running as Root	WARNING	MEDIUM

Falco Simulator

The Falco eBPF kernel sensor requires kernel-level perf buffer allocation (mmap) that is unavailable in this environment. A Kubernetes CronJob (k8s/falco/falco-simulator-cronjob.yaml) fires every 3 minutes and emits identical Falco-format JSON to stdout. Promtail ships these logs to Loki with the {app="falco"} label — the entire downstream pipeline (Loki ingestion, runtime-status queries, Grafana Explore, CI gate) is functionally identical to real Falco output. Production Falco deployment is fully documented in infra/terraform/modules/falco/ and k8s/falco/custom-rules.yaml.

# Deploy the simulator
kubectl apply -f k8s/falco/falco-simulator-configmap.yaml
kubectl apply -f k8s/falco/falco-simulator-cronjob.yaml

# Trigger an alert immediately (without waiting for 3-min cron)
kubectl create job falco-test-1 --from=cronjob/falco-simulator -n monitoring

# Wait for Promtail flush (~90s), then query
guardops runtime-status --since 15m

guardops runtime-status output

GuardOps Runtime Status — last 15m
  Querying Loki at http://localhost:3100 ...
  Falco alerts (15m) — CRITICAL:1  HIGH:5  MEDIUM:1  LOW:0  (query: 0.1s)

SEV        RULE                                        POD              NAMESPACE  TIME (UTC)
CRITICAL   GuardOps Shell Spawned Inside Container     test-app-...     default    09:27:00
HIGH       GuardOps Sensitive File Read in Container   test-app-...     default    09:27:10
HIGH       GuardOps Package Manager Executed in Con..  test-app-...     default    09:24:01
MEDIUM     GuardOps Write to /etc Inside Container     test-app-...     default    09:15:00

  Top alert (GuardOps Shell Spawned Inside Container):
  Shell spawned inside container (user=root shell=sh proc.cmdline=sh -c id ...)

  Grafana: Explore -> Loki datasource -> {app="falco"} | json

---

Self-Healing (Phase 8)

Phase 8 adds automated incident response: when a CRITICAL Falco alert fires, a FastAPI webhook handler automatically isolates the offending pod using a Kubernetes NetworkPolicy and labels it for visibility. When the alert resolves, the quarantine is automatically lifted.

How it works

CRITICAL Falco alert -> Prometheus -> Alertmanager
    -> POST /webhook -> guardops-alertmanager-webhook pod (FastAPI)
         -> kubectl label pod guardops.io/quarantine=true
         -> kubectl apply NetworkPolicy (deny-all ingress, DNS-only egress)
    Alert resolves -> POST /webhook status=resolved
         -> kubectl delete networkpolicy
         -> kubectl label pod guardops.io/quarantine-

NetworkPolicy applied on quarantine

The handler applies a policy named guardops-quarantine-<fingerprint[:8]> that:

• Ingress: denies all inbound traffic (no service can reach the pod)
• Egress: allows only DNS (UDP/TCP port 53) — keeps logging agents working while blocking all exfiltration vectors (HTTP, HTTPS, raw sockets)

The pod-level quarantine label (guardops.io/quarantine=true) ensures only the offending pod is isolated — other replicas of the same deployment continue serving traffic normally.

Webhook handler

The handler runs as a Kubernetes Deployment in the monitoring namespace, deployed by Terraform (modules/alertmanager-webhook). It exposes:

Endpoint	Method	Description
/healthz	GET	Liveness probe — returns {"status":"ok","version":"0.8.0"}
/readyz	GET	Readiness probe — verifies kubectl is reachable; returns 503 if not
/webhook	POST	Alertmanager webhook receiver

RBAC: the handler's ServiceAccount is bound to a ClusterRole with the minimum permissions needed — pods/patch, networkpolicies CRUD, nodes/patch.

guardops quarantine-status output

GuardOps · Quarantine Status
Phase 8 — Self-Healing  |  Active quarantine policies and isolated pods

Checking namespace(s): default

── Quarantined Pods  (1) ───────────────────────────────────────────────────

╭──────────────────────────────────┬───────────┬──────────┬──────────────────────┬─────╮
│ Pod Name                         │ Namespace │  Phase   │ Node                 │ Age │
├──────────────────────────────────┼───────────┼──────────┼──────────────────────┼─────┤
│ 🔒 guardops-app-d9f557c78-2hxcw  │ default   │ Running  │ ip-10-0-11-46...     │  2m │
╰──────────────────────────────────┴───────────┴──────────┴──────────────────────┴─────╯

── Active Quarantine NetworkPolicies  (1) ──────────────────────────────────

╭────────────────────────────────────────┬───────────┬───────────────────────────────────┬──────────┬─────╮
│ Policy Name                            │ Namespace │ Falco Rule                        │ FP       │ Age │
├────────────────────────────────────────┼───────────┼───────────────────────────────────┼──────────┼─────┤
│ guardops-quarantine-testfp12           │ default   │ Shell Spawned Inside Container    │ testfp12 │  2m │
╰────────────────────────────────────────┴───────────┴───────────────────────────────────┴──────────┴─────╯

  To manually release a pod after investigation:
   guardops quarantine-status --release <pod-name> --namespace <ns>

Terraform module

infra/terraform/modules/alertmanager-webhook/
    main.tf       ServiceAccount, ClusterRole, ClusterRoleBinding, Deployment, Service
    variables.tf  project_name, environment, webhook_image (required); webhook_port, replicas (optional)
    outputs.tf    service_url, healthz_url, service_name, deployment_name, namespace

Enable in terraform.tfvars:

enable_self_healing = true
webhook_image       = "123456789012.dkr.ecr.ap-south-1.amazonaws.com/guardops-app:webhook-latest"

Building the webhook image

# Build Dockerfile.webhook (installs fastapi, uvicorn, pyyaml, kubectl into the app image)
cd D:\EXTRA\GuardOps
$ECR = "123456789012.dkr.ecr.ap-south-1.amazonaws.com/guardops-app"
aws ecr get-login-password --region ap-south-1 | docker login --username AWS --password-stdin "123456789012.dkr.ecr.ap-south-1.amazonaws.com"
docker build -t "${ECR}:webhook-latest" -f Dockerfile.webhook .
docker push "${ECR}:webhook-latest"

Testing quarantine manually

# Port-forward the handler
kubectl port-forward svc/guardops-alertmanager-webhook 9095:9095 -n monitoring

# Fire a test quarantine webhook
$pod = kubectl get pods -n default --no-headers -o custom-columns=":metadata.name" | Select-Object -First 1
Invoke-WebRequest -Uri "http://localhost:9095/webhook" -Method POST `
    -ContentType "application/json" -UseBasicParsing -Body (ConvertTo-Json -Depth 10 @{
        receiver="guardops-webhook"; status="firing"
        alerts=@(@{status="firing"; fingerprint="testfp123"
            labels=@{alertname="GuardOpsFalcoCritical"; severity="critical"
                     guardops_action="quarantine"; pod=$pod; namespace="default"
                     rule="Shell Spawned Inside Container"}
            startsAt=(Get-Date).ToUniversalTime().ToString("yyyy-MM-ddTHH:mm:ssZ")
            endsAt="0001-01-01T00:00:00Z"})
        version="4"; groupKey="test"; truncatedAlerts=0
        groupLabels=@{}; commonLabels=@{}; commonAnnotations=@{}
        externalURL="http://alertmanager:9093"
    })

# Verify
guardops quarantine-status -n default
kubectl get networkpolicy -n default -l guardops.io/managed-by=guardops
kubectl get pods -n default -l guardops.io/quarantine=true

---

Security Pipeline

Four pre-deploy tools run in sequence, followed by one post-deploy DAST scan, one post-deploy runtime check, and continuous self-healing in production. All findings are normalised to a unified severity scale before gating.

Tool	Phase	Type	What it catches	Severity mapping
Semgrep	Pre-deploy	SAST	Code patterns, secrets, OWASP Top 10	ERROR=HIGH, WARNING=MEDIUM, INFO=LOW
Bandit	Pre-deploy	SAST	Python-specific vulnerabilities	Adjusted by confidence level
Trivy (fs)	Pre-deploy	Secret/IaC	Hardcoded secrets, misconfigs	Direct
Trivy (image)	Pre-deploy	SCA	CVEs in OS packages and Python deps	UNKNOWN mapped to LOW
SonarQube	Pre-deploy	Quality gate	Security hotspots, code smells	BLOCKER=CRITICAL, CRITICAL=HIGH, MAJOR=MEDIUM
OWASP ZAP	Post-deploy	DAST	Runtime HTTP vulns, missing headers, exposed endpoints	High=CRITICAL, Medium=HIGH, Low=MEDIUM, Info=LOW
Falco (via Loki)	Post-deploy	Runtime	Shell spawns, file reads, package managers, root processes	Maps Falco priority to unified scale
Alertmanager webhook	Continuous	Self-healing	Automatic pod quarantine on CRITICAL Falco alert	CRITICAL triggers quarantine

Bandit confidence adjustment:

Severity	Confidence	Unified result
HIGH	HIGH	CRITICAL
HIGH	LOW	MEDIUM
MEDIUM	HIGH	HIGH
LOW	HIGH	MEDIUM

ZAP severity is bumped one tier vs SAST because a live runtime finding has a shorter exploit distance than a code pattern finding.

Falco priority mapping:

Falco Priority	GuardOps Severity
EMERGENCY / ALERT / CRITICAL	CRITICAL
ERROR	HIGH
WARNING	MEDIUM
NOTICE / INFORMATIONAL / INFO / DEBUG	LOW

Reports are written to security/reports/latest.html and latest.json after every scan. In CI, reports are uploaded to S3 automatically. ZAP reports are uploaded as the zap-dast-report artifact in every deploy run.

---

GitHub OIDC (Phase 6)

Static IAM user credentials (AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY) are fully replaced by GitHub OIDC. GitHub generates a short-lived JWT per workflow run; the CI runner exchanges it for temporary STS credentials valid for 1 hour. No long-lived secrets are stored anywhere.

Required GitHub secrets (Phase 6+):

Secret	Value
AWS_ROLE_ARN	terraform output github_actions_role_arn
AWS_ACCOUNT_ID	Your 12-digit AWS account ID
GUARDOPS_S3_BUCKET	terraform output s3_reports_bucket_name

Required GitHub variables:

Variable	Value
HAS_EKS_CLUSTER	true (set after EKS is provisioned)
HAS_AWS_ROLE	true (set after OIDC terraform apply)

Optional secrets:

Secret	Purpose
HAS_ZAP_ENABLED	Set to true to enable ZAP DAST in CI deploy job
HAS_FALCO_ENABLED	Set to true to enable runtime gate in CI (Phase 7)
SEMGREP_APP_TOKEN	Semgrep cloud dashboard
SONAR_TOKEN	SonarQube
SONAR_HOST_URL	SonarQube

---

Configuration

GuardOps reads .guardops.yaml from your project directory.

project:
  name: my-app            # Used as Helm release name, image name, ingress host
  cloud: local            # local | aws

kubernetes:
  namespace: default
  cluster: guardops-local

docker:
  registry: ""            # ECR URL for prod (e.g. 123.dkr.ecr.ap-south-1.amazonaws.com)

security:
  fail_on_severity: HIGH  # LOW | MEDIUM | HIGH | CRITICAL  (pre-deploy gate)
  skip_sonarqube: false

  # DAST (Phase 6)
  tools:
    owasp_zap: false      # set true in prod to enable post-deploy ZAP scan
  zap_target_url: ""      # leave empty to auto-detect from deploy result
  zap_fail_on: CRITICAL   # severity that blocks and triggers auto-rollback
  zap_timeout_seconds: 300

monitoring:
  grafana_url: ''
  prometheus_url: ''
  loki_url: 'http://localhost:3100'   # set after setup-runtime-security.ps1

# Phase 7 — Runtime Security
runtime_security:
  enabled: true                # set true after setup-runtime-security.ps1
  falco_alert_window: 1h       # default --since for runtime-status
  alert_fail_on: CRITICAL      # default --fail-on for CI gate
  namespaces_to_watch: []      # empty = query all namespaces

# Phase 8 — Self-Healing
self_healing:
  enabled: false               # set true after terraform apply with enable_self_healing=true
  webhook_url: ''              # set to terraform output webhook_service_url

---

Commands Reference

guardops deploy

Options:
  --env [local|prod]     Target environment. Default: local
  --skip-scan            Skip security scans. Never use in prod.
  --skip-build           Reuse existing image.
  --skip-sonarqube       Skip SonarQube scan.
  --skip-trivy           Skip Trivy scans.
  --skip-dast            Skip OWASP ZAP DAST scan. Never use in prod.
  --fail-on [LOW|MEDIUM|HIGH|CRITICAL]
                         Severity threshold that blocks deploy. Default: HIGH
  --replicas INTEGER     Override replica count.

guardops scan

Runs the full pre-deploy security scan pipeline without deploying. Writes HTML and JSON reports to security/reports/.

guardops rollback

Options:
  --release TEXT         Helm release name. Default: reads from .guardops.yaml
  --revision INTEGER     Target revision. Default: 0 (previous release)
  --namespace TEXT       Kubernetes namespace. Default: default

guardops status

Shows pod phase, readiness, restart count, node placement, and service URL for the deployed release.

guardops logs

Streams logs from the running pod. Accepts --tail and --follow flags.

guardops runtime-status (Phase 7)

Options:
  --since [15m|30m|1h|3h|6h|12h|24h|7d]
                         Time window to query Loki. Default: 1h
  --namespace TEXT       Filter alerts to a specific Kubernetes namespace.
  --severity [LOW|MEDIUM|HIGH|CRITICAL]
                         Minimum severity to display. Default: LOW (show all)
  --tail INTEGER         Maximum number of alerts to display. Default: 50
  --loki-url TEXT        Loki base URL. Overrides monitoring.loki_url in config.
                         Default: http://localhost:3100
  --fail-on [LOW|MEDIUM|HIGH|CRITICAL]
                         Exit 1 if alerts at or above this severity are found.
                         Intended for CI post-deploy gates.

Prerequisites: Loki must be reachable. Run kubectl port-forward svc/loki 3100:3100 -n monitoring first.

guardops quarantine-status (Phase 8)

Options:
  --namespace, -n TEXT   Kubernetes namespace to check.
                         Defaults to kubernetes.namespace in .guardops.yaml.
                         Pass 'all' to check every namespace.
  --all-namespaces, -A   Check all namespaces (equivalent to kubectl -A).
  --release TEXT         Manually release a quarantined pod by name.
                         Deletes its NetworkPolicy and removes quarantine label.
                         Requires --namespace.
  --json-output          Print raw JSON (useful for CI/scripts).

Prerequisites: kubectl must be configured and the cluster reachable.

---

Local Kubernetes Setup (k3d)

# Create cluster with ingress port mapping
k3d cluster create guardops-local \
  --port "80:80@loadbalancer" \
  --port "443:443@loadbalancer" \
  --wait

# Install ingress-nginx
helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx --create-namespace \
  --set controller.service.type=NodePort \
  --set controller.admissionWebhooks.enabled=false \
  --wait --timeout 5m

# Add to hosts file (Windows: C:\Windows\System32\drivers\etc\hosts)
# 127.0.0.1  test-app.local

# Access app after deploy
kubectl port-forward -n ingress-nginx svc/ingress-nginx-controller 8080:80

Windows note: After recreating a k3d cluster, patch the kubeconfig — replace host.docker.internal with 127.0.0.1.

---

AWS Infrastructure

Infrastructure is fully defined in infra/terraform/. Remote state is stored in S3 with DynamoDB locking — no local .tfstate files.

infra/terraform/
    bootstrap/      S3 bucket + DynamoDB table for remote state (run once)
    modules/
        ecr/        ECR repository, scan-on-push, 10-image lifecycle policy
        s3/         Reports bucket, versioning, AES256, Glacier after 90 days
        vpc/        Public + private subnets, NAT, IGW, route tables
        iam/        EKS cluster role, node role, AmazonEBSCSIDriverPolicy
        iam_oidc/   GitHub OIDC provider + CI role (Phase 6, replaces IAM user)
        eks/        Managed node group (t3.large), CoreDNS, kube-proxy,
                    VPC CNI, EBS CSI driver, launch template (IMDSv2 hop limit=2)
        falco/      Falco + Loki + Promtail via Helm provider (Phase 7)
                    Requires live EKS + monitoring namespace. Use
                    enable_runtime_security=true in terraform.tfvars.
                    Alternative: scripts/setup-runtime-security.ps1
        alertmanager-webhook/   Phase 8 self-healing handler (FastAPI pod)
                    ServiceAccount + ClusterRole + Deployment + ClusterIP Service
                    Requires live EKS + monitoring namespace + webhook image in ECR.
                    Use enable_self_healing=true in terraform.tfvars.

Always-on (near-zero cost): ECR, S3, DynamoDB, remote state bucket, OIDC provider, IAM role.

Destroy nightly (~$5.28/day when running): EKS control plane ($0.10/hr), t3.large node ($0.075/hr), NAT gateways ($0.045/hr each).

# Bootstrap remote state (one-time only)
cd infra/terraform/bootstrap
terraform init && terraform apply -auto-approve

# Migrate existing state to S3
cd infra/terraform
terraform init -migrate-state

# Daily operations
terraform apply -auto-approve    # morning (~12 min)
terraform destroy -auto-approve  # evening (~8 min)

---

Helm Chart

The Helm chart at k8s/helm/guardops-app/ deploys with security defaults applied at the pod level:

securityContext:
  runAsNonRoot: true
  runAsUser: 1000
  readOnlyRootFilesystem: false
  capabilities:
    drop: ["ALL"]

Production values (values-prod.yaml) add:

• replicaCount: 2
• imagePullPolicy: Always
• HPA enabled (CPU-based autoscaling, 2-10 replicas)
• Ingress with TLS configuration
• monitoring.enabled: true — creates ServiceMonitor for Prometheus scraping

---

Dockerfile (Multi-stage)

Phase 4.5 replaced the single-stage build with a two-stage build:

# Stage 1: builder — installs deps into an isolated venv
FROM python:3.11 AS builder
RUN python -m venv /build/venv
COPY requirements.txt .
RUN pip install -r requirements.txt

# Stage 2: runtime — copies only the venv, no pip/wheel/setuptools
FROM python:3.11-slim AS runtime
COPY --from=builder /build/venv /venv
COPY app.py .
RUN useradd --uid 10001 --no-create-home --shell /sbin/nologin appuser
USER 10001

Result: pip, wheel, and all build tools are absent from the final image, significantly reducing the CVE surface area reported by Trivy.

Phase 8 adds Dockerfile.webhook, which layers fastapi, uvicorn, pyyaml, httpx, kubernetes, and kubectl on top of the app image to produce the self-healing handler image.

---

Development

# Clone and set up
git clone https://github.com/Bihan-Banerjee/GuardOps
cd GuardOps
python -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\Activate.ps1
pip install -e ".[dev]"

# Run tests
pytest tests/ -v --tb=short

# Lint and type check
ruff check cli/ backend/
mypy cli/ backend/ --ignore-missing-imports

Test coverage:

File	Tests	Covers
test_builder.py	15	Image naming, build success and failure paths, ECR tag format
test_config.py	12	YAML read/write, defaults, config existence checks
test_security.py	68	All 4 runners: skip, timeout, malformed JSON, severity mapping, report output
test_deployer.py	37	kubectl apply, k3d import, rollout wait, rollback, service URL
test_deployer_phase3.py	35	Helm deploy, rollback, release name sanitisation, chart path resolution
test_runtime_security.py	53	Falco priority mapping, FalcoQueryResult counts/filtering, Loki HTTP layer, --fail-on logic

---

Known Operational Notes

State lock after interrupted apply:

# If terraform hangs on "Acquiring state lock":
terraform force-unlock -force <LOCK_ID>

Subnet CIDR conflict after incomplete destroy:

# If terraform apply fails with InvalidSubnet.Conflict:
aws ec2 describe-subnets --filters "Name=cidrBlock,Values=10.0.1.0/24" `
    --query "Subnets[0].SubnetId" --output text
terraform import module.vpc.aws_subnet.public[1] <subnet-id>
terraform apply -auto-approve

EBS CSI driver / IMDS hop limit: EKS AL2023 nodes default to IMDSv2 hop limit of 1, which blocks pod-level AWS SDK calls. The launch template in modules/eks/main.tf sets http_put_response_hop_limit = 2 permanently. If the EBS CSI controller shows CrashLoopBackOff after a node replacement, verify the launch template is attached to the node group.

Prometheus not scraping test-app: Always use helm install (not helm upgrade --install) for kube-prometheus-stack on a fresh cluster. Upgrading over a previous release can silently preserve stale serviceMonitorNamespaceSelector settings that restrict scraping to the monitoring namespace only.

ZAP on Windows (local runs): --network host is not supported on Docker Desktop for Windows. The ZAP runner automatically omits this flag locally. Use host.docker.internal as the target hostname instead of localhost when port-forwarding a service for local DAST testing.

Loki chart 6.x — important values quirks:

• Use storageClass: gp2 not storageClassName: gp2 under singleBinary.persistence
• Must set chunksCache.enabled: false and resultsCache.enabled: false — defaults request ~11GB RAM, exceeding t3.large capacity
• Must set read.replicas: 0, write.replicas: 0, backend.replicas: 0 or chart validation fails in SingleBinary mode

Falco eBPF on EKS + t3.large: The Falco eBPF sensor requires kernel-level contiguous memory for perf ring buffer allocation. On a t3.large running the full monitoring stack, this allocation fails with unable to mmap the perf-buffer. The Falco simulator CronJob (k8s/falco/) provides identical JSON output for portfolio/dev use. See infra/terraform/modules/falco/ for production deployment documentation.

HCL multi-line strings: HCL does not support Python-style implicit string concatenation across lines inside parentheses. All description values in Terraform files must be single-line strings. The + operator is also not valid for string concatenation in HCL — use interpolation ("${var.a}.${var.b}") instead.

Terraform identity change error after manual kubectl deletes: If resources are deleted outside Terraform (e.g. kubectl delete deployment) and then terraform apply throws Unexpected Identity Change, run:

terraform state rm "module.alertmanager_webhook[0].kubernetes_deployment.webhook"
terraform apply -auto-approve

Webhook image uses /venv/bin/python, not system Python: The app image's CMD uses /venv/bin/python (an isolated virtualenv). All pip installs for the webhook handler in Dockerfile.webhook must target the venv: RUN /venv/bin/python -m pip install .... Installing via /usr/local/bin/pip writes to a different site-packages that the venv Python cannot see.

kubectl --short flag removed in v1.28+: The readyz endpoint in alertmanager_handler.py calls kubectl version --client. If your handler image uses kubectl v1.27 or earlier, remove --short from that call — the flag was removed and causes a non-zero exit code that makes /readyz return 503.

Nightly shutdown order matters:

helm uninstall kube-prometheus-stack -n monitoring  # triggers EBS volume deletion
helm uninstall loki -n monitoring
helm uninstall promtail -n monitoring
kubectl delete pvc --all -n monitoring              # ensures PVCs are removed
Start-Sleep -Seconds 30                             # wait for ec2:DeleteVolume
cd infra/terraform && terraform destroy -auto-approve

Skipping the Helm uninstall leaves orphaned EBS volumes that persist after terraform destroy and continue billing silently.

---

Release History

Version	Status	Description
v0.1.0	Published	CLI scaffold, k3d local deploy via kubectl
v0.2.0	Published	Security scanning pipeline, 154 tests, GitHub Actions CI
v0.3.0	Published	Helm deploy, rollback command, EKS Terraform modules
v0.4.0	Published	Full AWS pipeline: ECR push, EKS deploy, S3 report upload
v0.4.1	Published	Remote TF state, multi-stage Docker, subprocess hardening, Trivy CI cache
v0.5.0	Published	Prometheus + Grafana, /metrics endpoint, ServiceMonitor, EBS CSI, custom metrics
v0.6.0	Published	GitHub OIDC replaces static IAM keys, no more AWS_ACCESS_KEY_ID in CI
v0.6.1	Published	OWASP ZAP DAST post-deploy scan, auto-rollback on CRITICAL, ZAP image fix (ghcr.io), Windows Docker compat
v0.7.0	Published	Loki + Promtail log pipeline, Falco-format alert ingestion, guardops runtime-status CLI, CI runtime gate job, Falco rules + simulator
v0.8.0	Current	Self-healing: Alertmanager webhook handler, automatic pod quarantine via NetworkPolicy, auto-release on alert resolved, guardops quarantine-status CLI, Terraform alertmanager-webhook module, Dockerfile.webhook, PrometheusRule + AlertmanagerConfig wiring
v0.9.0	Planned	Multi-environment: staging + prod namespaces, blue-green deploy
v1.0.0	Planned	Real domain, TLS, ArgoCD GitOps, full runbooks

---

Planned Future Updates

Version	Status	Description
v1.1.0	Planned	SBOM + Cosign signing
v1.2.0	Planned	Kyverno admission control
v1.3.0	Planned	Scan metadata database
v1.4.0	Planned	Web dashboard
v1.5.0	Planned	Vulnerability waivers
v1.6.0	Planned	LLM-assisted triage
v1.7.0	Planned	Risk-based scoring (scoped)

---

License

MIT