Demyst

de·mys·ti·fy /dēˈmistəˌfī/ — to make less obscure or confusing

A scientific linter for research code. Like black for formatting and mypy for types, demyst checks scientific logic.

pip install demyst
demyst analyze ./src

Status

What It Catches

Try It in 30 Seconds

git clone https://github.com/Hmbown/demyst.git
cd demyst
pip install -e .
demyst leakage examples/ml_data_leakage.py

You'll see demyst catch the classic ML mistake: preprocessing data before splitting it.

Sample Output

$ demyst leakage examples/ml_data_leakage.py

──────────────────────────── Data Leakage Detected ─────────────────────────────

CRITICAL Line 47 in examples/ml_data_leakage.py
  fit_transform() called BEFORE train_test_split.
  Preprocessing learns from test data — your benchmark is invalid.

  45   X, y = load_medical_data()
  46   scaler = StandardScaler()
❱ 47   X_scaled = scaler.fit_transform(X)  # LEAKS TEST INFO
  48   X_train, X_test, y_train, y_test = train_test_split(X_scaled, y)

  Fix: Split first, then fit on train only:
       X_train, X_test = train_test_split(X)
       X_train = scaler.fit_transform(X_train)
       X_test = scaler.transform(X_test)

Summary: 1 critical issue

Quick Examples

Leakage — the #1 ML benchmarking error:

# WRONG: Leaks test statistics into training
scaler.fit_transform(X)
X_train, X_test = train_test_split(X_scaled)

# CORRECT
X_train, X_test = train_test_split(X)
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

P-Hacking — uncorrected multiple comparisons:

# 20 tests at α=0.05 expects 1 false positive
for condition in conditions:
    if ttest(a[condition], b[condition]).pvalue < 0.05:
        print(f"{condition} significant!")  # No correction applied

Usage

# Full analysis
demyst analyze your_code.py

# Individual guards
demyst mirage model.py
demyst leakage train.py
demyst hypothesis stats.py
demyst units physics.py
demyst tensor network.py

# Auto-fix mirages
demyst mirage model.py --fix

# CI mode
demyst ci . --strict

Why Mirages Matter

These documented cases show how np.mean() hides critical information:

Run demyst mirage examples/real_world_mirages.py to see detection in action.

CI/CD

GitHub Actions:

name: Demyst
on: [push, pull_request]
jobs:
  demyst:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - run: pip install demyst
      - run: demyst ci . --strict

Pre-commit:

If you're already using black + mypy + ruff, drop this in next to them:

repos:
  - repo: https://github.com/Hmbown/demyst
    rev: v0.1.0a1
    hooks:
      - id: demyst

See examples/configs/ for more templates.

Suppressing False Positives

Use inline comments to suppress specific warnings:

# Suppress all demyst warnings on this line
mean_value = np.mean(data)  # demyst: ignore

# Suppress only mirage warnings
dashboard_avg = np.mean(daily_views)  # demyst: ignore-mirage

# Suppress only leakage warnings  
scaler.fit_transform(X)  # demyst: ignore-leakage

Available suppressions: ignore, ignore-mirage, ignore-leakage, ignore-hypothesis, ignore-tensor, ignore-unit, ignore-all

Configuration

Create .demystrc.yaml:

profile: default  # Or: biology, physics, chemistry, economics

rules:
  mirage:
    enabled: true
    severity: critical
  leakage:
    enabled: true
    severity: critical

ignore_patterns:
  - "**/tests/**"

Programmatic API

from demyst import TensorGuard, LeakageHunter, HypothesisGuard, UnitGuard

source = open('model.py').read()
result = LeakageHunter().analyze(source)

if result['summary']['critical_count'] > 0:
    print("DATA LEAKAGE DETECTED")

Design Principles

Silent failures in research code don't crash — they produce wrong numbers that look right. A model trains, metrics look good, paper gets submitted... then someone discovers the test set leaked into training. Demyst catches these before they become retractions.

Detection capabilities:

• Mirage: Detects 80+ NumPy array creators, tracks variable flow, checks for nearby variance operations
• Leakage: Tracks fit/fit_transform calls relative to train_test_split/cross_val_score
• Hypothesis: Counts statistical tests, checks for correction methods, detects p-value conditionals
• Tensor: Analyzes layer sequences for gradient death patterns, normalization misuse
• Units: Dimensional analysis via variable naming conventions and explicit annotations

References

Resources

• Quick Start Guide
• Interactive Notebook
• Configuration Templates
• Full Documentation

License

MIT — See LICENSE

"The first principle is that you must not fool yourself—and you are the easiest person to fool." — Richard Feynman