shkit 1.2.16

Deterministic + AI hybrid engine that converts Databricks pipeline failures into prioritized, explainable incident intelligence

Incident Intelligence Core (IIC)

A deterministic + AI hybrid engine that converts raw Databricks pipeline failures into structured, prioritized, explainable incident intelligence.

Product model (current): install from PyPI (shkit==X.Y.Z) and configure via a Databricks secret scope (iic) in your own workspace — see docs/INSTALL.md. The old operator path (publish-tenant, pull/push-antibodies, the DBX_TOKENS secret, per-tenant configs in git) is DEPRECATED and kept for one release only — see docs/MIGRATION.md.

Not "AI that fixes pipelines." Not a DevOps automation tool. IIC is a structured intelligence engine that explains why a pipeline failed, what it affects, and how urgent it is — in business and technical terms — and only spends an LLM call when the structure says it's worth it.

---

Design principles

1. Deterministic first, AI second. Stages 1–8 never call an LLM. The model runs only at stage 9, and only when the router decides it's warranted.
2. Every failure becomes a structured object. The whole system is organised around a handful of dataclasses that flow stage to stage.
3. No external workflow dependencies. Databricks + an optional logs webhook. No Jira, ServiceNow, or Airflow lock-in.
4. No "chatty AI." The diagnosis engine returns structured output only — it interprets the pre-built IncidentDNA, it does not classify or converse.
5. Every decision is traceable from evidence. Each IncidentDNA records the exact signals it fired on; each ImpactScore exposes its term-by-term breakdown.

---

The 11-stage pipeline

 1  Event ingestion        Databricks Jobs/Runs API · logs webhook
 2  Normalize              → NormalizedFailureEvent   (source-agnostic)
 3  Context builder        logs + notebook + metadata + schema snapshot
 4  Dependency analyzer    upstream/downstream blast radius
 5  Change detector        diff failed run vs last successful run
 6  Incident DNA builder   structured failure fingerprint   ← the heart (rules)
 7  Impact engine          deterministic severity + business risk   ← NO LLM
 8  Model router           none / lightweight / powerful, by impact
 9  Diagnosis engine       root cause + reasoning + fix      ← LLM only if routed
10  Report generator       one IncidentReport (JSON + markdown)
11  Notifier               severity-ranked Teams card (optional)

Stages 3–8 are pure/deterministic and fully unit-tested. The LLM is invoked at most once per incident, and never for cache hits or derived dependency failures.

---

The core data models (the moat)

Everything flows through these, in order (src/iic/models/):

Model	Stage	What it captures
NormalizedFailureEvent	2	source-agnostic failure (pipeline, task, error, run)
IncidentContextBundle	3	logs, notebook source, cluster/job metadata, schema, lineage
ChangeDiffObject	5	schema / config / code / runtime / deployment changes since last success
IncidentDNA	6	failure_type, affected_layer, root_signal, pattern_id, confidence_signature, signals
ImpactScore	7	severity, business_risk, blast radius, transparent breakdown
RoutingDecision	8	tier (none/lightweight/powerful) + reason
DiagnosisResult	9	root_cause, confidence, suggested_fix, evidence, produced_by
IncidentReport	10	the final product — to_dict() and to_markdown()

IncidentDNA — deterministic classification

The heart of the system. A rule engine maps the raw error + context + change diff onto a canonical taxonomy (SCHEMA_DRIFT, DATA_QUALITY, MISSING_DATA, PERMISSION, DEPENDENCY, RESOURCE, TIMEOUT, CONFIG, CODE_ERROR, UNKNOWN) and records exactly which signals fired. Rules — not an LLM — because the classification drives the cost decision (routing), so it must be free, instant, predictable, and traceable.

Impact engine — deterministic scoring (NO LLM)

score = downstream_jobs * 2
      + affected_tables * 1.5
      + dashboard_impact * 3        # a broken exec dashboard outranks an internal job
      + recurrence_score * 2

Bands → LOW / MEDIUM / HIGH / CRITICAL. A business-facing layer (gold / mart) bumps severity one band. The full term contribution is exposed in breakdown, so every score is explainable.

---

Example output

# Incident INC-555-1 — MEDIUM
Pipeline: customer_ingestion · Task: validate_schema · 2026-06-10T10:02:00+00:00

Summary: MEDIUM schema drift in customer_ingestion.validate_schema (silver layer)
         — 2 downstream job(s), 0 dashboard(s) at risk; business risk MODERATE.

Root cause (85%): The failing task hit a schema mismatch — a column it expects is
                  missing or changed type.

Failure DNA:  type=SCHEMA_DRIFT · layer=silver · pattern=schema_drift__silver_v1
Suggested fix: Reconcile the ingestion mapping with the new source schema, then re-run.
Diagnosed by:  rules   (deterministic — no tokens spent)

---

Project structure

src/iic/                      ← the engine (packaged as a wheel, fully tested)
  models/                       6 core dataclasses + enums
  ingestion/                    Databricks source + logs-webhook normalizer
  context/                      ContextBuilder
  dependency/                   DependencyAnalyzer (blast radius)
  change/                       ChangeDetector (pure diff core + I/O wrapper)
  dna/                          IncidentDNABuilder  ← the heart
  impact/                       ImpactEngine  ← deterministic, no LLM
  routing/                      IncidentModelRouter
  diagnosis/                    DiagnosisEngine (structured LLM, post-DNA)
  report/                       ReportGenerator
  notify/                       TeamsNotifier (optional)
  runtime/                      IncidentEngine orchestrator + notebook entrypoint
src/healing_kit/              ← shared utility library reused by IIC
                                (Databricks/Teams clients, sql_safety, dependency
                                 graph, hmac, error hashing, auth, query guard)
notebooks/incident_engine.py  ← thin Databricks driver (imports the wheel)
onboarding/                   ← provisioner / preflight / rollback / CLI
tests/unit/test_iic_*.py      ← 66 unit tests for the deterministic core
resources/incident_workflows.yml  ← single consolidated DAB job

What was removed in the rebuild: the old multi-notebook self-healing fan-out (event_aggregator, context_agent, diagnosis_engine, resolution_executor, token_budget_enforcer, healing_agent_v2) and the old orchestrator — all redundant duplicate logic. The pipeline is now one consolidated, tested engine.

---

Quick start

git clone https://github.com/yogasathyandrun/Self-healing-kit.git
cd Self-healing-kit
pip install -r requirements.txt
python setup.py

setup.py runs preflight checks, provisions the schema + the single [IIC] Incident Intelligence Engine job, uploads the thin driver, and wires a failure-triggered task onto the pipeline you want to observe.

After setup, run your pipeline normally. When a task fails you get one severity-ranked incident report in Teams.

---

Development

pip install -e ".[dev]"
pytest tests/unit/ -q          # 66 IIC tests + existing healing_kit tests
ruff check src/ onboarding/

The deterministic core has no Spark/Databricks dependency, so the whole engine is testable with plain fakes — see tests/unit/test_iic_engine.py for the full 11-stage flow exercised end-to-end.

---

What still needs a live workspace

The deterministic core (stages 3–8, 10) is fully tested here. These touch live Databricks and are exercised against a real workspace, not in CI unit tests:

• Unity Catalog lineage for table-level blast radius (falls back to task-level DAG when unavailable).
• Model Serving for the LLM diagnosis path (the engine is fully functional without it — it falls back to deterministic templates).
• Provisioner / DAB deployment of the consolidated job and trigger wiring.

---

v4 — Databricks-native Cluster Agent SDK (recommended)

The intelligence core ships as a Databricks-native SDK (a Python wheel) that auto-activates on import. Install it as a cluster library and restart — that's the whole setup. No external infrastructure, no notebook changes:

merge to main → CI auto-publishes the rolling "latest" release + copyable URL
             → install on cluster from URL (init script / %pip) → restart cluster
             → import iic → auto_bootstrap() → excepthook + reconciler active

Every merge to main refreshes a rolling latest release and prints the install URL in the run summary (no tagging needed): …/releases/download/latest/iic-1.0.0-py3-none-any.whl. Pushing a version tag (v1.2.0) additionally cuts an immutable, versioned release.

Uncaught Python errors in any job on that cluster are then captured, analyzed by the engine, and reported. A Jobs API reconciler backstops the rest, and an optional one-line with iic_monitor(): gives guaranteed immediate capture for critical sections. Importing iic stays fully inert off-cluster (tests/local). See docs/V4_ARCHITECTURE.md.

Author

Sathya | Data Engineer Rebuilt as the Incident Intelligence Core, June 2026. GitHub: https://github.com/yogasathyandrun/Self-healing-kit