SkillJab

Your AI writes the analysis. SkillJab makes sure it's right before you spend the six hours finding out.

pipx install skilljab  ·  claude --plugin-dir ./plugin  ·  /skilljab:jab

See it work · Why · How it works · The report · Install · What you get · Under the hood · The stones · Limits

SkillJab is a Claude Code plugin and CLI for people who do data analysis with an AI at their side. It builds a miniature of your analysis with a planted truth, sabotages it with realistic perturbations, and shows you the failures nothing would have warned you about — then turns every one of them into a checker, a fix, and a heads-up inside a SKILL.md that Claude loads whenever it touches that analysis again.

• 🧪 Proves your pipeline can recover a truth you planted — before any real data
• 🥊 Finds the silent failures — the result moved and no check fired
• 🧬 Keeps the antibodies — your AI gets better at your pipeline, permanently, with the reason on record
• 📊 Crash-test report — where it's strong, where it's fragile, how long it will take at full size

🎯 See it work

The prompt: "Here's a CSV of our customers — tenure, monthly charges, support tickets, contract, plan, churned. Build a churn model and tell me which factors matter."

Claude wrote dropna → get_dummies → logistic regression on every column. Clean, idiomatic code. On clean data it recovers every planted effect — AUC 0.73, factors ranked correctly. Nothing looks wrong.

Then SkillJab threw ten stones at it, one per round, each a thing that happens to real customer tables. Eight broke the answer, and nothing warned anyone:

Then the antibodies. For each silent failure the explainer wrote why in the domain's own words, a checker at the stage boundary, and a heads-up. Same ten stones again: six caught; two still silent — and for those two the skill says plainly that the data as delivered cannot reveal them, and which columns to ask for.

Then the hold-out — the part that makes it evidence. Ten perturbations the checkers were never written for: tenure in years, a different column going missing, a very noisy leak, the outcome itself missing, a stone dropped between stages… Nine of ten flagged by antibodies written for something else; the one miss is the batch the skill already declares unguardable. The first pass of that hold-out exposed two real holes in the antibodies — and fixing them is in the replay too.

Full write-up, the pipeline, and a 4-minute replay script →  ·  the rendered report and the skill it produced →

💡 Why

You ask Claude to write your pipeline. It writes clean, competent code. You run it on the real data, wait hours, and the result is wrong — because of something that was known: the region where every variant is correlated, the column secretly computed from the outcome, the join that doubled half the rows, the spreadsheet that exported 1,234 as text.

The AI knew about all of these. It didn't apply them, because nothing reminded it at the moment it was writing that step. You didn't catch it, because the code looked right. And by the time the mistake was visible, the compute was spent.

SkillJab gives the AI the scars before you pay for them. The name says the mechanism: a jab is a vaccine shot and a boxing punch — a small, deliberate hit that makes you stronger before the real fight.

⚙️ How it works

1. Plan it several ways. Planners with different personas — a statistician, a domain veteran, the person who runs the cluster, Reviewer 2 — each write a plan. Where they disagree is where the risk lives.
2. Plant a truth and prove recovery. A tiny dataset with a known answer, the whole pipeline run on it. If the answer doesn't come back, the pipeline is wrong before any real data.
3. Throw stones. Generic perturbations, dosed to bite. Only silent failures are reported — the result moved and nothing fired.
4. Explain. Each silent failure forces the AI to say why, which is when it recalls the domain knowledge it had but wasn't using.
5. Keep the antibody. Explanation → checker + mitigation + heads-up → rendered into SKILL.md, with provenance.
6. Repeat until a round is clean. Then run the real thing — with a hook that warns you if the skill has gone stale.

Every step is a slash command:

📊 The report

One sentence at the top tells you the worst thing it found; everything below shows where and how much. A single self-contained HTML file — no server, no external requests, safe to email. These are from the churn test.

Slow-motion replay — your own estimates against the planted truth, with a slider for the dose. Watch the coefficient leave the green band and see the point where nothing would have told you.	Where it breaks — dose against error per stone; the knee is the fragility. "How much missingness can I survive?" gets a number, not a warning. Green dots: a check caught it.
Impact points — stone × stage, colored by the worst outcome ever seen: red silent, green caught, grey harmless. The reds that remain are the ones the skill says it cannot guard without more columns.	Lap times — predicted runtime per stage at your real N, extrapolated from three miniature sizes, with ranges, a flag for anything superlinear, and where to put a cheap check so a bad input dies at minute two instead of hour six.

Immunity record — rounds × stones; red turning green is the skill learning. Below it, every antibody with its provenance, and how often the judge picked the right plan in the lineup.

🚀 Install

pipx install skilljab                                  # the engine — a CLI, so pipx (no fight with system Python)
git clone https://github.com/AlsammanAlsamman/skilljab
claude --plugin-dir ./skilljab/plugin                  # the Claude Code side

sudo apt install pipx && pipx ensurepath        # Debian / Ubuntu
brew install pipx                               # macOS
python3 -m venv ~/.skilljab && ~/.skilljab/bin/pip install skilljab && export PATH=~/.skilljab/bin:$PATH   # plain venv
pipx install git+https://github.com/AlsammanAlsamman/skilljab.git   # bleeding edge

Quick start — in Claude Code

/skilljab:build  pipeline.yaml
/skilljab:jab
/skilljab:report

Quick start — CLI only

cd skilljab/examples/churn_ai_pipeline
./run_demo.sh                                         # baseline, 30 rounds, 7 sweeps, report
open .claude/skills/churn/history/report.html
cat  .claude/skills/churn/SKILL.md

skilljab init     --name mine --pipeline pipeline.yaml --spec spec.yaml --skill-dir .claude/skills/mine
skilljab baseline --skill .claude/skills/mine --target-n 2000000     # proves clean recovery, fits timing
skilljab round new    --skill .claude/skills/mine
skilljab round stones --skill .claude/skills/mine target_leakage mnar_missing --level 0.6
skilljab round run    --skill .claude/skills/mine                     # -> silent | caught | harmless | …
skilljab sweep  --skill .claude/skills/mine --stone mnar_missing --levels 6
skilljab antibody add --skill .claude/skills/mine --file antibody.json
skilljab report --skill .claude/skills/mine                           # history/report.html
skilljab status --skill .claude/skills/mine                           # jabbed: true/false, and why

📦 What you get

In your repo, per pipeline:

.claude/skills/<your-pipeline>/
├── SKILL.md            what Claude reads before touching this analysis — rendered, never hand-edited
├── antibodies.json     every entry traces to a round, a stone, or something you recognized
├── tree.json           the decisions the planners disagreed on, with the evidence on each branch
├── checks/             the checkers that now run at every stage boundary
└── history/            rounds · sweeps · timing · lineup log · graveyard · the report

A heads-up in the skill reads like this:

ab-004 · dropna removed the churners — stage prepare · decision missing_handling · round 3, stone mnar_missing, worst error 67%

• Trigger: missingness of any column differs by > 5 points between churned and not
• Heads-up: support_tickets was blank for the customers with the most tickets — their ticket history is purged when an account closes. dropna removed a biased slice of churners, and the effect of support tickets came out 67% too small.
• Do: report missingness per column split by outcome before dropping anything; use a missing indicator or imputation.
• Check: checks/missingness_by_outcome.py runs after stage prepare

Three evidence classes, kept apart: proven by simulation · reported by you · hypothesis only.

🔬 Under the hood

Two halves, and each fact lives in exactly one:

Five blind roles. Subagents that cannot see each other's files: planners (one per persona) · a judge that runs the lineup — "one of these plans failed; which?" — and is scored against planted culprits so you know how much to trust it · a saboteur that picks stones · a blind analyst that runs the pipeline with the current skill loaded · an explainer, the only one allowed to see what was planted.

The funnel. Search wide and cheap, prove narrow and expensive:

Elicitation proposes; simulation disposes. The AI can suspect anything; only a stone that breaks the miniature — or something you recognized yourself — earns a place in the skill.

🪨 The stones

Deliberately domain-neutral. A generic stone is enough to make the AI's own knowledge do the specialization: "a correlated block broke PCA" becomes "exclude chr6:25–35 Mb before PCA" in the explainer's hands.

🔌 Your pipeline's contract

Stages as shell commands — any language, any tool — and a last stage that writes the estimates:

name: gwas
stages:
  - id: qc      cmd: "plink2 --bfile {in} --geno 0.02 --make-bed --out {out}"   out: qc.csv
  - id: prune   cmd: "Rscript prune.R {in} {out}"                              out: pruned.csv
  - id: fit     cmd: "python fit.py {in} {out}"                                 out: result.json

result.json = {"estimates": {"beta_x1": 0.79, ...}}, with the same names as the planted truth in sim/spec.yaml. Checkers are scripts called at stage boundaries — python3 check.py <stage_output> <stage_input> — that exit 1 or print {"fired": true, "message": "..."}.

Three generators ship: tabular_regression, tabular_classification, two_group_lift — with named features, categorical effects, and a planted intercept. Adding one is a function that returns (DataFrame, {estimand: truth}).

⚠️ Limits

• Miniatures can lie. Some failures only appear at scale. The timing page extrapolates and says so; treat ranges as optimistic.
• You need a truth to plant. Quantitative analyses, yes. Design and strategy, no — there the simulation would just be the AI's own assumptions fed back to itself.
• Stones are stones. The churn test proves the mechanism and its generalization; it has not yet caught a problem in a real dataset that nobody planted. That's the next test, and the one that matters most.
• The graveyard is local. A shared, anonymized graveyard of real past failures is the obvious next step.

🛠️ Development

git clone https://github.com/AlsammanAlsamman/skilljab && cd skilljab
python3 -m venv .venv && . .venv/bin/activate && pip install -e ".[dev]"
python -m pytest -q                              # 47 tests, ~35 s
claude plugin validate plugin --strict

skilljab/            engine: simulate · stones/ · inject · runner · check · project · plandiff · tree · lineup · timing · render_skill · graveyard · report · pack/
plugin/              Claude Code plugin: commands/ · agents/ · skills/skilljab-core · hooks/
examples/            churn_ai_pipeline — the test above, replayable · toy_regression — the minimal example the tests use
assets/flowchart/    the loop diagram — hand-drawn SVG in JavaScript, rendered to PNG by scripts/render_flowchart.sh
docs/DESIGN.md       the design and the reasoning behind every choice
docs/sessions/       transcripts of the design discussions

The thinking — the pre-mortem, the recoverability test, why generic stones beat a hand-written pitfall library, the lineup, the tricks for making an AI say "oh yeah, I should have told you" — is in docs/DESIGN.md.

Alsamman M. Alsamman · aalsamman100@gmail.com · github.com/AlsammanAlsamman · MIT