clusop 0.0.15

Photon Fallback Analyzer — zero-touch Databricks cost forensics (.pth self-arm + bundled JVM listener)

clusop — Photon Fallback Analyzer

pip install clusop

clusop finds the money Databricks Photon quietly burns. When Photon hits an operator it can't run (a Python UDF, a struct-IN filter, an unsupported Delta feature), it silently falls back to the JVM — you keep paying the 2–2.9× Photon DBU premium while getting JVM speed. clusop detects those fallbacks from the executed plan, estimates the wasted spend, and proposes a fix. It never touches your job.

One install, two halves, zero config

A single pip install clusop ships both halves and arms itself:

• a Python .pth (clusop_autoload.pth) that Python's site machinery runs at interpreter startup → imports clusop.runtime.bootstrap → arms automatically. You do not import clusop anywhere in your job.
• a bundled Scala JAR (clusop/jars/photon_listener.jar) — the actual QueryExecutionListener. The bootstrap addJars it and registers it over Py4J.

Install it as a cluster/job library (production path) so every interpreter that starts already has it. %pip install clusop + dbutils.library.restartPython() works for dev.

Everything is auto-detected at runtime — cloud (from the instance type), DBR (from DATABRICKS_RUNTIME_VERSION), cluster shape, and whether the JVM is reachable. No per-user setup; a new customer pip-installs and it works.

What it costs you: nothing if it can't run safely

• Fail-open everywhere. Any error in arming, listening, parsing, or dispatch is swallowed. clusop never breaks, slows, or blocks a customer query.
• Needs a SINGLE_USER / dedicated cluster for the JVM listener. On USER_ISOLATION (Shared) clusters the JVM is sealed behind Spark Connect — clusop detects this and stays dormant rather than failing.
• Propose-never-apply. clusop emits a signal and a Teams card. A human decides.

How a signal is born

1. The JVM listener sees a finished query and reads its executed plan.
2. The structural parser (src/clusop/analysis/parser.py, mirrored in Scala) decides if this is a real fallback. The hard part: a clean Photon query always ends in one terminal ColumnarToRow (the normal result boundary) — that is not a fallback. Real fallback is mid-plan ColumnarToRow, RowToColumnar round-trips, or BatchEvalPython/ArrowEvalPython. Counting raw occurrences false-positives; clusop doesn't.
3. The waste model (waste.py) sizes the loss: runtime × Σ(node DBU/hr) × $rate × (photon_premium−1) × fallback_weight.
4. Confidence (confidence.py) is decomposed into four legs — parse / diagnosis / cost / recommendation. A fix (rewrite the UDF) needs only parse+diagnosis; a disable-Photon recommendation is a dollar decision and is capped by the cost leg.
5. The signal flows to a driver batcher (dedup by signature) → central aggregator (idempotent upsert, suppression, prioritization) → Teams card.

Cost: modeled now, billed if granted

clusop always works with a modeled cost (public price table → MEDIUM cost confidence). If the workspace can read system.billing.usage, the CostTierResolver reconciles the estimate to billed dollars (HIGH confidence). Detection never depends on system tables — they only upgrade the cost layer. Prices are reference data, not something clusop invents per-row.

Certifying it's real

harness/certify.py runs on a dedicated cluster, captures real executed plans for known clean / known-fallback fixtures, and reports catch-rate plus a DBR-stamped Delta feature-support matrix — derived, never hand-maintained.

Layout

src/clusop/
  runtime/      bootstrap (arming), detect (cloud/DBR/shape/jvm)
  analysis/     parser · waste · confidence · signal
  pricing/      price_table.json · provider
  service/      aggregator · resolver · suppression · teams · onboard
  jars/         photon_listener.jar  (baked in by the release workflow)
scala/          the QueryExecutionListener (sbt; mirror of the Python parser)
harness/        certify.py  (catch-rate + Delta matrix, run on a dedicated cluster)
tests/          parser / waste / confidence

See SPEC.md for the full design and invariants.

Release

Push-button: the Publish workflow auto-versions (max(PyPI, tags)+1), builds the JAR with sbt, bakes it into the wheel, lints+tests, gates on (.pth at wheel root AND JAR bundled), then publishes via PyPI Trusted Publishing (OIDC — no stored token) and tags the bump.