DevOps AI Agent

An AI agent that investigates real DevOps problems. The end goal: ask it something like "Why is my Kubernetes pod in CrashLoopBackOff?" and have it gather evidence, reason about the evidence, identify the likely root cause, recommend remediation, and give verification steps.

Phase 5 broadens the agent from Kubernetes-only to the wider DevOps surface: Linux systemd services and journals, Docker containers, Terraform state/plan, and git/GitHub pull requests — every tool still read-only, still a fixed allowlisted argv template. Phase 6 adds the automation surface: a one-shot CLI mode (python main.py "problem" — exit-code driven, cron/CI-friendly) and a structured JSON export of the investigation report, so pipelines can act on the verdict instead of parsing markdown. Phase 7 adds persistence: the investigation record is auto-saved on every change to ~/.devops-ai-agent/investigations/ (overridable with --store-dir or AGENT_STORE_DIR), survives CLI exits, and the REPL resumes the newest in-progress record at startup. Phase 11 adds conversation persistence: the chat itself (user turns, replies, tool exchanges) is auto-saved to ~/.devops-ai-agent/conversations/ (AGENT_CHAT_DIR overrides) and restored at startup together with the investigation it was linked to — you continue where you left off instead of re-explaining the problem. Phase 12 adds streaming: in the REPL the model's answer appears as it is written, token by token (one-shot mode stays buffered for pipelines). Phase 8 broadens the tool surface to 47 read-only tools: Kubernetes depth (pod listing, kubectl top resource usage, HPAs, PVCs, contexts), Docker depth (networks, volumes, disk usage), GitHub Actions via gh (runs, run jobs, workflows), cloud identity (AWS/GCP/Azure), monitoring/logging (Prometheus, Loki, Grafana — endpoints from environment config), and Ansible listing (inventory, playbook tasks). Phase 9 adds 11 trending-market tools (58 total): New Relic (NRQL + open alerts over NerdGraph — credentials from env), Trivy image vulnerability scanning, Helm releases (list/status/history), Argo CD (GitOps app sync/health), Istio mesh proxy status, and Docker Compose (project and service listing). Phase 10 ships it as a product: the devopsiq package on PyPI and a prebuilt multi-arch Docker image (ghcr.io/devopsabhii/devops-ai-agent) — both cut automatically by pushing a v* tag (test gate → PyPI via Trusted Publishing + GHCR in parallel). No API key? The agent still runs: it starts in model-less mode — record commands (/report, /investigations) and the whole 61-tool layer work without a key; only questions to the model need one. The repository is git-tracked. Every phase still built from scratch — no LangChain, LangGraph, AutoGen, CrewAI, or MCP.

New here? The Install & run guide (PDF) walks you from zero to investigating — three install routes, credentials (or none at all), model choice, and every command.

Want to use this in your own project? See INTEGRATION.md for the three integration levels: drive it as a CLI from cron/CI (exit codes + --json), embed it as a Python library, or extend it with your own read-only tools.

1. What this project is

The agent is built intentionally: we own the core architecture — how the model is called, how conversation history flows, how tool selection + execution + evidence feedback work — instead of depending on a framework for it.

Today the agent:

• holds a conversation with GLM 5.3 through OpenRouter;
• has 61 real, read-only tools across fifteen domains: host facts, Kubernetes (12 tools), Linux system (4), Docker + Compose (10), Terraform (3), Helm (3), Argo CD (2), Istio (1), security/Trivy (1), git/GitHub + Actions (7), cloud identity (4), monitoring/logging (3), New Relic (2), Ansible (2), plus the 3 investigation tools;
• runs a tool-use loop: when the model decides a question needs evidence, it requests the tool, the application executes it locally, and the real output is fed back to the model, which then answers from it;
• runs an investigation loop: for a reported problem it plans a multi-step investigation — open a record with initial hypotheses, gather evidence with the right domain tools, update each hypothesis's verdict as evidence builds, conclude with root cause + remediation + verification, and end with a structured report. The tracked state lives in the application, so /investigation and /report show it directly.

It is read-only by design: every tool is a static, allowlisted command template — the model can never pass arbitrary command text, can never select a mutating verb (no delete/restart/edit/apply/scale/exec/run/rm/reset/ destroy anywhere), and no mutating capability exists. Object and unit names are validated before reaching any CLI; flags are injected only by fixed templates, never by model text. The investigation tools mutate only the agent's in-memory record. Any future mutating capability will only ever arrive behind an explicit human-approval gate.

2. Current architecture

You (terminal)
   │  plain text
   ▼
main.py ......................... CLI REPL, .env loading, error handling
   │
   ▼
agent/agent.py (DevOpsAgent) .... system prompt + conversation history
   │
   ▼  the tool-use loop, inside DevOpsAgent._complete():
   │
   │   attach tool schemas -> call model
   │        │
   │        ├─ model requests tool(s)? ── yes ─▶ tools/registry.execute_tool()
   │        │                                 │   per call, in order
   │        │                                 │      └─ allowlisted read-only
   │        │                                 │         command runs for real
   │        │                                 │      └─ result echoed back as
   │        │                                 │         a "tool" message
   │        │                                 ◀── loop calls the model again
   │        └─ model answers in plain text? ── yes ─▶ done
   │              (safety cap: max 10 tool-use turns, then abort)
   ▼
OpenRouter (https://openrouter.ai/api/v1)
   ▼
GLM 5.3 (z-ai/glm-5.3)

Module map:

The tool-use loop

All model calls funnel through DevOpsAgent._complete(). Each turn:

1. The full history is sent with every tool's schema attached (tools=).
2. If the reply contains tool calls, the assistant tool-request turn is echoed into history verbatim, and each call is executed locally via tools/registry.execute_tool(name, arguments).
3. Every real result is appended as a sibling {"role": "tool"} message pinned to its tool_call_id.
4. The loop calls the model again — giving it the evidence to reflect on — and repeats until it answers in plain text.
5. MAX_TOOL_ITERATIONS = 10 aborts runaway loops.

The investigation loop (Phase 4)

For a reported problem (a pod in CrashLoopBackOff, a broken rollout) the same loop carries a plan as well as evidence:

1. The model opens a record with investigation_begin (one-line problem + 2–4 initial hypotheses, which become H1, H2, …).
2. It plans what it needs, then calls the read-only tools one deliberate step at a time.
3. Every finding is recorded with investigation_record: evidence notes (linked to a hypothesis when they bear on one) and verdicts (supported / refuted / confirmed).
4. Each record call returns the live tracker (Problem, status, hypotheses with verdicts, evidence), so the model always knows where it stands without inspecting all of history.
5. When evidence is sufficient the model calls investigation_conclude with root cause, remediation recommendations (nothing is ever executed), verification steps, and confidence — then ends its answer with the structured report.

The record lives in the application, not just the conversation: the report is rendered deterministically from it (agent/investigation.py), and the CLI exposes it directly:

The report the agent ends with and /report render are kept consistent by construction: tool results confirm each record call and the tracker, and render_report() is the single report format.

Example (live, against a real cluster):

You: /investigate Pod crashloop-6f8c… is in CrashLoopBackOff in default ns.
     (…) investigate.
Agent: (investigation_begin → k8s_pod_status → k8s_pod_logs →
       k8s_deployment_status → verdicts on H1/H2 → investigation_conclude)
       ends with: Facts → Hypotheses → Root cause → Remediation (recommended)
       → Verification steps.

You: /report
Agent: # Investigation report  … (canonical, rendered from the record)

The tools (Phases 5, 8 and 9)

Every tool runs a fixed, allowlisted command — the model only picks arguments (names, counts, namespaces), never command text.

Notes:

• Output is the real CLI output — Python does not re-parse it; the model reads exactly what an engineer would see. Verbatim quoting is a hard prompt rule.
• docker stats is always --no-stream (the pinned flag is what keeps it from following forever and hanging the turn).
• terraform plan is a dry run — it computes the diff, mutates nothing; -input=false keeps it from ever prompting. Requires an initialized directory (terraform init); on a fresh dir the model reports terraform's own error, honestly.
• Terraform and git/gh tools operate on the current working directory the agent was launched from (no path parameters — that keeps traversal out).
• gh_prs needs the gh CLI authenticated and a GitHub remote; otherwise the exact CLI error is returned.
• kubectl top needs metrics-server in the cluster; where it is absent, kubectl's exact error is returned (that is honest behavior, not a bug).
• k8s_contexts is a listing of kubeconfig contexts only — the agent can never run config use-context and change which cluster is in use.
• Monitoring endpoints come from the environment, never the model: prom_query/loki_query/grafana_health read PROMETHEUS_URL / LOKI_URL / GRAFANA_URL; unset or non-http(s) values are honest ToolErrors naming the variable. The model supplies only the PromQL/LogQL text, which is percent-encoded into the query string; the pinned curl argv (--proto =https,http, GET only) blocks file:// and everything but the configured host.
• Cloud tools are identity/account level only and take no arguments at all; Ansible tools are listing modes only (--list, --list-tasks) with relative-path validation (no /, no leading -, no ..).
• Helm tools only read (list/status/history — never install/upgrade/ rollback/uninstall); Argo CD tools only list/get (never sync/rollback/ delete); istioctl_proxy_status takes no arguments at all; trivy_image_scan validates the image reference (no leading -, no spaces) and warns its first run may take minutes (CVE DB download).
• New Relic credentials are environment-configured, never model-chosen: NEW_RELIC_API_KEY (passed only as a curl header value) and NEW_RELIC_ACCOUNT_ID (digits only). The NRQL text travels inside a json.dumps-built payload as a GraphQL variable, so it can never escape its string slot; only NerdGraph queries are ever sent, never mutations.
• Log tails (--tail, -n, --limit) are bounded integers, validated 1–500 (git log 1–100, gh 1–50, Loki limit 1–1000, helm history 1–50).
• Any missing CLI / unreachable target returns the exact error — nothing is invented (verified live for kubectl, systemctl, journalctl, docker, terraform, git).

How read-only is enforced (defense in depth)

• The tool schemas only allow picking names/counts/namespaces from validated arguments — there is no way to pass command text to any CLI.
• Every tool is a fixed argv template. No sh -c anywhere, so nothing is ever parsed by a shell; flags the model might abuse (e.g. --no-stream for docker stats, -input=false for terraform plan) are hard-coded into the template and cannot be removed or added.
• Only read-only verbs exist per domain — get/logs/top and config get-contexts (kubectl), status/-u/ss/ps (system), ps/inspect/logs/stats/images/network ls/volume ls/ system df/compose ls/compose ps (docker), show/state list/ plan (terraform), list/status/history (helm), app list/ app get (argocd), proxy-status (istioctl), image --scanners vuln (trivy), status/log/diff/pr list/run list/run view/ workflow list (git/gh), --list/--list-tasks (ansible), GET-only curl with a pinned argv (monitoring), query-only NerdGraph with a pinned curl (New Relic). No delete, restart, edit, apply, scale, exec, run, rm, pull, push, commit, reset, merge, use-context, playbook-run, install, upgrade, rollback, uninstall, sync, destroy — by construction.
• Names are validated per domain before reaching any CLI: Kubernetes object names (DNS style), systemd unit names (no /, no leading -), Docker names (no /), GitHub run ids (digits only), Ansible sources (relative paths, no ..), and no path parameters at all for terraform/git/gh tools. This blocks flag and path injection.
• The executor re-validates every argument. Never trust the model.
• --request-timeout/timeouts bound slow or hanging commands, and logs are tail-bounded.
• Unrecognized tools/arguments return Tool error: ... to the model instead of executing.
• Tool output is truncated at 8,000 characters per result.

3. Why GLM 5.3

• GLM 5.3 (z-ai/glm-5.3 on OpenRouter) is a capable, cost-effective general model — with strong instruction-following and function-calling support, which the tool loop has demonstrated live in every phase.
• Served by Zhipu AI through a single OpenRouter endpoint, so there is no separate vendor API to manage.
• An investigation loop sends many tokens (system prompt, tool schemas, tool output, history); GLM 5.3 keeps that cost sustainable.

4. How OpenRouter fits

OpenRouter is a model gateway: one API key, one OpenAI-compatible endpoint, access to many models. We call

POST https://openrouter.ai/api/v1/chat/completions

with model z-ai/glm-5.3 and standard tools / tool messages. The official openai Python SDK works as our client with two config lines:

OpenAI(api_key=..., base_url="https://openrouter.ai/api/v1")

Swapping to another OpenRouter model later is a one-liner (env var, /model command, or --model flag); moving to any other OpenAI-compatible provider changes only agent/agent.py configuration.

Your model, your choice. The default is baked in as a fallback, never a restriction. Four ways to set the model — the first one set wins:

1. --model NAME flag (one run: devopsiq --model openai/gpt-5.2 "why?")
2. /model <name> in the REPL — saves it to ~/.devops-ai-agent/config.json so every future run uses it
3. OPENROUTER_MODEL (shell or .env)
4. built-in default (z-ai/glm-5.3)

export OPENROUTER_MODEL=anthropic/claude-sonnet-5   # or openai/gpt-5.2, google/gemini-2.5-pro, ...
devopsiq /model openai/gpt-5.2                      # or save a default from the REPL

Two things to weigh when picking: the agent is a tool-use loop, so choose a model with solid function-calling (a chat-only model will answer from imagination instead of gathering evidence); and an investigation makes several model calls per run, so price-per-call multiplies — that is why the default is a cheap, reliable tool-caller rather than the biggest model.

No OpenRouter at all? OPENROUTER_BASE_URL points the same client at any OpenAI-compatible endpoint — including a local one. Ollama, free and offline:

export OPENROUTER_BASE_URL=http://localhost:11434/v1
export OPENROUTER_MODEL=llama3.2          # any Ollama model that does tools
devopsiq "docker container checkout keeps exiting, investigate"

5. Installation

Three ways — pick one. Whatever you choose, the agent also needs the CLIs of the domains you use (kubectl, docker, gh, … — each tool reports its exact error if its CLI is missing, so a partial install is fine).

From PyPI (no clone):

pipx install devopsiq        # or: pip install devopsiq
export OPENROUTER_API_KEY=sk-or-...
devopsiq --json "why is api-5d6f crash-looping?"

The console command is devopsiq; the importable package is agent / tools / main (see INTEGRATION.md for library use).

Prebuilt Docker image (bundles kubectl, helm, gh, trivy, git, curl, docker CLI):

docker run --rm \
  -e OPENROUTER_API_KEY=sk-or-... \
  -v "$HOME/.kube:/home/agent/.kube:ro" \
  -v agent-records:/data \
  ghcr.io/devopsabhii/devops-ai-agent --json "why is api-5d6f crash-looping?"

Images are multi-arch (amd64 + arm64), published on every v* tag (:latest tracks the newest release). Add -v /var/run/docker.sock:/var/run/docker.sock for the Docker/Compose tools.

From source (development):

Requires Python 3.10+ (built-in venv). The tool layer itself uses only the standard library; the OpenAI SDK and python-dotenv are the only Python dependencies.

cd ~/devops-ai-agent
python3 -m venv .venv
.venv/bin/pip install --upgrade pip
.venv/bin/pip install -r requirements.txt

Recommended CLIs, by domain:

No API key? You still have the tools

The model is the only part that needs a key — the agent builds without one (model-less mode), and two things keep working:

• Record commands: devopsiq /report, devopsiq /investigations, and the same commands inside the REPL, all work with no key configured. Only actual questions exit 1 with the setup message naming OPENROUTER_API_KEY.
• The whole 61-tool layer via the Python library, with no model and no key (see INTEGRATION.md):

import agent.agent                            # registers every tool
from tools.registry import execute_tool

print(execute_tool("k8s_pods", '{"namespace": "prod"}'))

And the fully key-free agent path is a local model (Ollama example in section 4 above) — no cloud account needed at all.

6. Environment setup

cp .env.example .env     # then edit it

.env must contain your real key — get one at https://openrouter.ai/keys:

OPENROUTER_API_KEY=sk-or-...

Optional overrides (defaults shown):

OPENROUTER_BASE_URL=https://openrouter.ai/api/v1
OPENROUTER_MODEL=z-ai/glm-5.3
AGENT_STORE_DIR=~/.devops-ai-agent/investigations   # (Phase 7) where records are saved
AGENT_CHAT_DIR=~/.devops-ai-agent/conversations     # (Phase 11) where conversations are saved
PROMETHEUS_URL=http://prometheus:9090               # (Phase 8) enables prom_query
LOKI_URL=http://loki:3100                           # (Phase 8) enables loki_query
GRAFANA_URL=http://grafana:3000                     # (Phase 8) enables grafana_health
NEW_RELIC_API_KEY=NRAK-...                          # (Phase 9) NerdGraph user key
NEW_RELIC_ACCOUNT_ID=1234567                        # (Phase 9) numeric account id

.env is gitignored; the API key is never hard-coded in Python, printed, or logged. If OPENROUTER_API_KEY is already set in your shell, the shell value wins and .env is not consulted. Without any key the agent still starts in model-less mode (record commands + the tool layer — see "No API key?" in section 5). Kubernetes tools use kubectl's own config (~/.kube/config or KUBECONFIG); no agent-side config is needed.

7. How to run the agent

REPL (interactive):

cd ~/devops-ai-agent
.venv/bin/python main.py

One-shot (cron / CI / scripts): pass the problem as the first argument. The agent runs once, prints its answer, and exits 0 on success, 1 on setup/API errors — so it drops straight into a pipeline:

.venv/bin/python main.py "why is api-5d6f crash-looping?"
.venv/bin/python main.py --json "why is api-5d6f crash-looping?"   # structured report
.venv/bin/python main.py --resume --json "any update?"             # continue a prior run
.venv/bin/python main.py --store-dir /tmp/runs --out report.json "..."  # pipeline paths
.venv/bin/python main.py --model openai/gpt-5.2 "why is it down?"   # one-run model override

With --json the stdout is one JSON document (see render_report_json in agent/investigation.py): problem, status, hypotheses, evidence, and — once concluded — conclusion with root_cause, remediation, verification, confidence. If the model never opened an investigation, --json/--out fail with exit 1 rather than printing a malformed report. --out PATH additionally writes that JSON to an exact path for pipelines that want a known location; --store-dir DIR points persistence somewhere else for the run; --resume restores the newest conversation — messages replay into the model's context, and the investigation the conversation was linked to comes back too (with no open conversation, it falls back to the newest in-progress record). Slash commands also work one-shot: python main.py /report.

Persistence (Phase 7): every record mutation is auto-saved to ~/.devops-ai-agent/investigations/ — one JSON file per investigation, written atomically on every begin/record/conclude, so a crash mid-run loses nothing. /investigations lists everything on disk; /endinvestigation clears memory but keeps the saved file. A broken store (permissions, full disk) degrades to a warning in the tool result — it never interrupts an investigation.

Conversation persistence (Phase 11): every exchange is auto-saved to ~/.devops-ai-agent/conversations/ (AGENT_CHAT_DIR overrides) — one JSON file per conversation, written atomically after every answer, and linked to the investigation file that was active while it was written. At startup (and with --resume) the newest open conversation is restored into the model's context together with its linked investigation, so you continue where you left off. History is pruned at user-turn boundaries (system prompt + the newest 40 turns) so a restored session never replays an unbounded token load and a tool exchange is never split from its request. /conversations lists saved chats; /newchat clears history and closes the saved copy. Same best-effort guarantee as Phase 7: an unwritable directory degrades to a note, never an error.

Streaming (Phase 12): in the REPL the answer is printed as the model writes it (Agent: followed by live text), instead of appearing only after the full response. One-shot mode keeps the buffered behavior so pipelines get clean stdout; the tool-use loop, history, and persistence are identical in both modes.

Example session:

DevOps AI Agent (Phase 14 — 61 read-only tools, persistent investigations)
Model:   z-ai/glm-5.3
Backend: https://openrouter.ai/api/v1
Store:   /home/you/.devops-ai-agent/investigations
Chats:   /home/you/.devops-ai-agent/conversations
Tools:   ansible_inventory, ansible_playbook_tasks, aws_identity,
         az_account, az_groups, docker_disk_usage, docker_images,
         docker_inspect, docker_logs, docker_networks, docker_ps,
         docker_stats, docker_volumes, gcloud_identity, gh_prs,
         gh_run_view, gh_runs, gh_workflows, git_diff, git_log,
         git_repo_status, grafana_health, investigation_begin,
         investigation_conclude, investigation_record, k8s_contexts,
         k8s_deployment_status, k8s_events, k8s_hpa, k8s_nodes,
         k8s_pod_logs, k8s_pod_status, k8s_pods, k8s_pvc, k8s_services,
         k8s_top_nodes, k8s_top_pods, loki_query, prom_query,
         sys_open_ports, sys_service_logs, sys_service_status,
         sys_top_processes, system_info, tf_plan, tf_show, tf_state_list
Commands: /investigate <problem>, /investigation, /investigations, /report, /endinvestigation, /conversations, /newchat, /model [<name>]
One-shot: python main.py [--json] [--resume] [--out report.json] [--store-dir DIR] [--model NAME] "<problem>"
Type 'exit' to quit.

You: The checkout service container keeps exiting in Docker. Investigate.
Agent: (docker_ps → docker_inspect → docker_logs, tracks hypotheses, and
       ends with the structured report: the crash command, exit code, and
       the remediation recommendation)

You: /report
Agent: # Investigation report … (the canonical record, rendered from state)

You: exit

Type exit / quit, or press Ctrl-D / Ctrl-C to leave. Slash commands are handled locally and never reach the model.

8. Current limitations

• Each domain needs its CLI installed and reachable. Missing CLIs, unauthenticated gh, a dead docker daemon, an uninitialized terraform directory, or an unreachable cluster all return the exact error honestly — nothing is invented, but a tool can't produce data without its backend.
• terraform/git/gh/ansible tools are working-directory scoped. They read the directory the agent was launched from — no path arguments by design (keeps traversal out). To investigate another repo/module, launch the agent there.
• Monitoring endpoints are environment-configured by design. Unless the operator sets PROMETHEUS_URL / LOKI_URL / GRAFANA_URL, those tools fail with an honest message naming the variable; the agent never invents a URL.
• kubectl top needs metrics-server. Clusters without it return kubectl's exact error — honest, but no usage data.
• Cloud resource tools are compute-listing only. Phase 13 adds one listing per cloud (aws_ec2_instances, gcloud_compute_instances, az_vm_list — instances/VMs with state and IPs); storage, load balancers, databases, and IAM resources are not swept. Both identity and resource tools need their CLI installed and configured.
• Phase 9 tools need their CLIs. helm, argocd (plus argocd login), istioctl and trivy are not bundled; without them those tools return the exact "not found" error. New Relic tools need NEW_RELIC_API_KEY + NEW_RELIC_ACCOUNT_ID in the environment; without them they name the missing variable — nothing is invented.
• Trivy's first scan downloads the CVE database and can take a couple of minutes; subsequent scans are fast.
• The model can only select from the registered tools. It can never run an arbitrary verb of any CLI, or a mutating one — by construction.
• Raw CLI output to the model. Python does not re-parse pod/docs/state; the model interprets real output, truncated at 8,000 characters per result.
• Restored conversation history is bounded. Persistence keeps the system prompt plus the newest 40 user turns (Phase 11); older context is pruned at save time, not replayed.
• Hypothesis tracking is model-driven. The record is what the model chose to record through the investigation tools; the live tracker returned on every record call is designed to keep that complete, but it is still the model's discipline.
• Streaming has no mid-stream retry. Transient model-call failures are retried before chunks start flowing (Phase 12 context: the connect/auth phase); a connection drop mid-stream surfaces as an error, since a partially streamed answer cannot be replayed cleanly.
• Read-only is enforced by construction today. No mutating tool exists; the gate they must pass through is already in place (Phase 14): a tool declared mutating has every call confirmed by the human operator (default-deny when nobody can be asked), with a session audit under /approvals.

9. Planned future phases

• Phase 2 — tool architecture. ✅ Done. Tool contract, registry, tool-use loop, first read-only tool, offline + live verification.
• Phase 3 — Kubernetes tooling. ✅ Done. Read-only kubectl-backed tools (pod status/logs, deployment status), name validation, verb allowlist, offline + live verification.
• Phase 4 — investigation loop. ✅ Done. The agent plans a multi-step investigation, tracks hypotheses and evidence in a first-class in-memory record, and produces a structured report (facts → observations → hypotheses → conclusion → remediation → verification).
• Phase 5 — multi-domain tooling. ✅ Done. Linux system (systemd/ journal/ports/processes), Docker (ps/inspect/logs/stats/images), Kubernetes depth (events/nodes/services), Terraform (show/state/plan), git/GitHub (status/log/diff/PRs) — all read-only, offline + live verified.
• Phase 6 — automation surface. ✅ Done. One-shot CLI mode (python main.py [--json] "<problem>") with exit codes for cron/CI, and a structured JSON export of the investigation report (render_report_json) so pipelines can act on the verdict. Repository is git-tracked with a clean .gitignore (secrets and the venv never ride along).
• Phase 7 — persistence. ✅ Done. The investigation record survives CLI exits: auto-saved on every mutation to ~/.devops-ai-agent/investigations/ (--store-dir/AGENT_STORE_DIR to override), atomic in-place file updates, REPL auto-resume of the newest in-progress record, /investigations listing, one-shot --resume / --out, and lossless to_dict/from_dict round-tripping — all offline tested.
• Phase 8 — tool expansion. ✅ Done. 21 more read-only tools (26 → 47): Kubernetes depth (k8s_pods, k8s_top_pods, k8s_top_nodes, k8s_hpa, k8s_pvc, k8s_contexts), Docker depth (docker_networks, docker_volumes, docker_disk_usage), GitHub Actions (gh_runs, gh_run_view, gh_workflows), cloud identity (aws_identity, gcloud_identity, az_account, az_groups), monitoring/logging (prom_query, loki_query, grafana_health — endpoints from env only), and Ansible listing (ansible_inventory, ansible_playbook_tasks) — offline tested, live-verified where the host has the CLI.
• Phase 9 — trending-market tools. ✅ Done. 11 more read-only tools (47 → 58): New Relic (newrelic_nrql, newrelic_alerts — NerdGraph over curl, credentials env-only, GraphQL variables so query text can't escape), Trivy (trivy_image_scan), Helm (helm_list, helm_status, helm_history — reads only), Argo CD (argocd_apps, argocd_app_status), Istio (istioctl_proxy_status), Docker Compose (docker_compose_ls, docker_compose_ps — live-verified on this host).
• Phase 10 — distribution. ✅ Done. The devopsiq package on PyPI (console command devopsiq, MIT) and the prebuilt multi-arch GHCR image, cut by a tag-driven release workflow: an offline test gate, then PyPI (Trusted Publishing — no token in the repo) and GHCR (amd64 + arm64) in parallel. v0.1.1 added model-less mode: with no API key the agent still constructs — record commands (/report, /investigations) and the 61-tool layer work; only model questions exit 1 with the setup message. Verified end-to-end against the published package.
• Model choice + resilient calls (v0.1.2). ✅ Done. The model is set by a 4-rung ladder — --model flag > /model-saved config file (~/.devops-ai-agent/config.json) > OPENROUTER_MODEL > baked-in default — and transient model-call failures (timeouts, connection errors, 429s, 5xx) retry with exponential backoff (2s→4s→8s + jitter, honoring a 429's Retry-After); a rejected key fails immediately.
• Phase 11 — conversation persistence. ✅ Done. The chat survives CLI exits too: one JSON file per conversation in ~/.devops-ai-agent/conversations/ (AGENT_CHAT_DIR overrides), written atomically after every exchange and linked to the investigation file active while it was written. At startup (REPL and one-shot --resume) the newest open conversation replays into the model's context together with its linked investigation; with none open, the Phase 7 investigation fallback applies. History prunes at user-turn boundaries (system prompt + newest 40 turns); /conversations lists saved chats; /newchat clears history and closes the copy. All offline tested.
• Phase 12 — streaming. ✅ Done. In the REPL the answer streams to the terminal as the model writes it (agent.ask(..., stream=True)); the streamed chunks reassemble into the same message shape the loop already reads, so tool calls, history, persistence, and one-shot mode are byte-for-byte the same as before. Retries cover the connect/auth phase; a mid-stream drop surfaces as an error (a partial answer cannot be replayed cleanly).
• Phase 13 — region-scoped cloud resources. ✅ Done. Three more read-only tools (58 → 61): aws_ec2_instances (EC2 in one region, with a --query projection), gcloud_compute_instances (one GCP zone), and az_vm_list (one Azure resource group, with power state). The single model-chosen argument per tool (region/zone/group name) is validated against a strict pattern before any CLI runs; read-only verbs only.
• Phase 14 — human-approval gate. ✅ Done (scaffolding). No mutating tool exists yet, and now none can be added that bypasses approval: Tool(mutating=True) routes every call through tools/approval.py — the REPL asks the human at the terminal (y/N), one-shot mode and tests auto-deny, one approval covers identical actions for the session, and every decision lands in a session audit (/approvals) plus a JSONL file next to the investigation store.
• Later — deeper cloud/observability reads (storage, LBs, databases; New Relic dashboards/entities, Prometheus range queries) behind the same patterns; first real mutating tools behind the Phase 14 gate when a use case earns them.