ci-doctorr 2.0.1

Postmortem CI failure analyzer: deterministic phase attribution plus one LLM explanation. Read-only, air-gap friendly.

Root-cause analysis for CI/CD.
A postmortem step that runs when a pipeline fails, works out where and why it broke, and emits a structured report — without ever touching your code.

---

When a pipeline fails, the logs are long and loud. An LLM handed the whole trace will confidently blame the runner or a missing cache — because that text is alarming — even when the real failure was an exit code 1 three sections down. ci-doctor fixes that with one rule:

Deterministic code decides where the job failed. The LLM only explains why.

Phase attribution is a pure function of job metadata and log structure, computed before any model is called. A loud, non-fatal WARNING: can never outrank the actual failure. If the classifier is wrong, that's a bug with a failing test — not a prompt to tune.

Features

• Deterministic phase attribution — a pure, fully-tested classifier decides where a job broke (provision · prepare · fetch · script · post) from metadata and log structure. Non-fatal WARNING: lines can never be blamed.
• Read-only and safe — never edits code, commits, or opens MRs, and always exits 0, so it can't change your pipeline's status or hide the real failure.
• Useful with no LLM — ships a deterministic report out of the box: phase, reason, terminal command, evidence excerpt, and templated remediation.
• Bring-your-own model — optional LLM step via openai (any OpenAI-compatible endpoint), litellm (Bedrock/Vertex/Azure), or the local claude CLI, selected by config.
• GitLab & GitHub — one provider-neutral core; the GitHub adapter was added with zero changes to core.
• Air-gap friendly — no telemetry, no update checks, no runtime downloads; ship as a self-contained Docker image or an offline wheel bundle.
• Secret redaction — scrubs secrets from the prompt and the report (round-trip tested).
• Structured output — a rich terminal report, report.md + report.json artifacts (with a self-contained handoff_prompt you can paste into a coding agent), and one idempotent MR/PR note.

How it works

acquire → segment → attribute → denoise → extract → budget → (LLM) → render / deliver
          └── deterministic: decides WHERE it failed ──┘        └── explains WHY ──┘

1. Acquire the failed jobs and their logs (a missing log is valid data — the "never got a runner" case).
2. Segment the trace into sections (GitLab section_* markers / GitHub ##[group]).
3. Attribute the failure to a phase — the pure classifier, first-match-wins precedence ladder with a WARNING:-is-never-fatal rule.
4. Denoise / extract / budget the blamed section into a small, high-signal evidence slice (ANSI/CR/dedup denoising, anchored windows, token budgeting — every truncation is visible). Shipped matcher packs cover pytest/jest/go/maven/gradle/bazel, Rust, .NET, Ruby, PHP, node + npm/pnpm/yarn/bun, Playwright/Cypress, tsc/eslint/mypy, Docker and Terraform; add your own under extraction.matchers — they merge onto the shipped packs by id.
5. LLM (optional) explains the cause within the already-decided phase, returning a validated JSON report. Disabled or unreachable → deterministic report instead.
6. Render / deliver to the terminal, report.md/report.json, and an idempotent MR/PR note.

Install

On PyPI the distribution is ci-doctorr (the ci-doctor name was taken); the import package is ci_doctor and the command stays ci-doctor.

git clone https://github.com/fennet82/ci-doctor
cd ci-doctor
uv sync                       # or: pip install .

# optional LLM backends:
uv sync --extra litellm       # or: pip install '.[litellm]'   — only for Bedrock/Vertex/Azure

Or build the self-contained image:

docker build -t ci-doctor .

Quickstart

# Replay a captured log offline — no network, no LLM:
uv run ci-doctor analyze failing-job.log

# Against a live pipeline (reads $CI_PIPELINE_ID etc. inside CI):
uv run ci-doctor analyze "$CI_PIPELINE_ID"

# From your laptop, in a clone — the repo is taken from `git remote origin`
# (with a warning) when GITHUB_REPOSITORY / CI_PROJECT_ID isn't set:
uv run ci-doctor analyze 18234567890

ci-doctor config prints the effective config; --diff shows only what your layers changed, --schema emits the JSON Schema. Full CLI and config reference on the documentation site.

Configuration

Layered and pydantic-validated — defaults.yml < repo .ci-doctor.yml < CI_DOCTOR_* env < CLI flags, with nested env vars using __ (CI_DOCTOR_LLM__MODEL=…). Unknown keys are an error. Minimal config:

provider: gitlab

gitlab:
  base_url: https://gitlab.com        # default; override for self-hosted
  token_env: CI_DOCTOR_GITLAB_TOKEN   # or gitlab.token_file for a secret mount

llm:
  enabled: true                       # false => deterministic-only report
  backend: openai                     # openai | litellm | claude_code
  model: qwen2.5-coder:32b
  api_base: http://openai-compatible-endpoint.internal:8000/v1

Every knob, and the three LLM backends, are documented on the configuration page. The LLM step is optional throughout — disabled, unconfigured or unreachable, ci-doctor emits the deterministic report instead of failing.

Use it in CI

GitHub Actions — one step; run-id defaults to the run that just failed:

permissions: { actions: read, contents: read, pull-requests: write }
steps:
  - id: doctor
    uses: fennet82/ci-doctor@master   # or pin a release tag
    with:
      post-pr-note: 'true'
  - if: steps.doctor.outputs.is-infra-not-code == 'true'
    run: echo "::notice::Infrastructure, not the change."

Outputs phase, category, confidence, is-infra-not-code, and the two report paths, so a workflow can branch on the verdict. Inputs, PR comments and GitHub Enterprise are on the action page; the full workflow is in examples/github-actions.example.yml.

Listing on GitHub Marketplace is a one-time manual step and cannot be scripted: draft or edit a release in the GitHub UI, tick Publish this Action to the GitHub Marketplace, and accept the terms. action.yml already carries the metadata that requires. None of this affects uses: — that resolves straight from the repo either way.

GitLab (.gitlab-ci.yml) — runs only on failure, always exits 0:

ci-doctor:
  stage: .post
  image: registry.internal.example.com/ci-doctor:latest
  rules:
    - when: on_failure
  allow_failure: true
  variables:
    CI_DOCTOR_GITLAB_TOKEN: "$CI_DOCTOR_TOKEN"
  script:
    - ci-doctor analyze "$CI_PIPELINE_ID"
  artifacts:
    when: always
    paths: [report.md, report.json]

Both providers have a complete, commented workflow in examples/.

Air-gapped / offline

ci-doctor makes no network calls except to the GitLab/GitHub and LLM endpoints you configure — no telemetry, no update checks, no runtime model/rule/schema downloads. Build once where there is internet, then ship inside:

docker build -t ci-doctor .                              # a self-contained image
pip wheel . -w ./wheels                                  # ...or a wheel bundle
pip install --no-index --find-links ./wheels ci-doctorr  # on the air-gapped host

Custom CA bundles (gitlab.ca_bundle, github.ca_bundle, llm.ca_bundle) and the standard HTTPS_PROXY / NO_PROXY / REQUESTS_CA_BUNDLE variables are honoured.

Documentation

fennet82.github.io/ci-doctor — overview, requirements, configuration, usage, the GitHub Action, and CI/CD examples. Source in docs/site/ (Astro); mise run docs previews it locally.

Start with Concepts — the pipeline end to end, what each module decides, and what a phase, a matcher or a redaction pass actually is. Matchers lists every shipped pack; that page is generated from config/defaults.yml, so run mise run docs:data after changing a pack (a test fails if it drifts).

For contributors: how the code is built — where things go, the invariants, how to write tests — is GUIDELINES.md; how to get a change in — setup, commits, pre-push checks — is CONTRIBUTING.md. The original design spec, kept for provenance, is docs/PLAN.md.

Development

mise run setup       # uv sync + the repo's git hooks  (or just: uv sync)
mise run check       # everything CI runs: tests, lint, guardrails, secret scan

The suite blocks real sockets and never calls an LLM, so it runs in seconds.

License

MIT © Elad Cohen