ReproGuard

Pre-production risk scanner for data science notebooks and ML repositories.

ReproGuard analyzes Jupyter notebooks and Python scripts for reproducibility risks, data leakage, privacy violations, missing dependencies, and handoff readiness — before you share, review, or promote work toward production.

Positioning: SonarQube-style review for data science work. Not a replacement for MLflow, DVC, Databricks, DataHub, or data observability platforms.

What's New in v0.3

v0.3.0 doubles the check inventory (29 → 73) and adds a dedicated GenAI risk category plus a full supporting toolchain:

Every feature is documented with runnable examples below.

Why This Exists

Data science projects often work on the author's machine but fail during review or production handoff because of:

• 📍 Local data paths — C:\Users\... or /Users/... that don't exist on another machine
• 📦 Missing dependencies — No requirements.txt or unpinned packages causing environment drift
• 🔄 Out-of-order execution — Notebook cells run in a non-linear order, hiding stateful assumptions
• 🔓 Hidden PII & secrets — Email addresses, API keys, or credentials buried in code or output
• 📊 Data leakage — Preprocessing before train/test split, test data in fit calls, target-like features
• 📝 Missing handoff docs — No clear statement of objective, data source, assumptions, or instructions

ReproGuard is local-first — scans happen on your machine without uploading data or notebooks to any third-party service.

Features

Detects 73 risk patterns across 7 categories

Output Formats

• Terminal — Color-coded summary with severity breakdown
• JSON — Structured data for programmatic consumption
• HTML — Styled standalone report with issue grouping
• SARIF 2.1.0 — Compatible with GitHub Code Scanning and VS Code

Scoring

Penalty-based scoring from 0–100. Status: ready_with_caution (≥75), needs_review (50–74), or not_ready (<50 or any CRITICAL issue). All penalties and weights are configurable.

Installation

pip install reproguard

From source

git clone https://github.com/vipulgote1999/ReproGuard.git
cd ReproGuard
pip install -e .

Development install

pip install -e .[dev]

Privacy extra (Presidio support — future)

pip install reproguard[privacy]

Quick Start

Scan a single notebook:

reproguard scan examples/risky_customer_churn.ipynb

Scan an entire project directory:

reproguard scan .

Scan with HTML report and fail CI on low score:

reproguard scan . --format html --fail-under 50

Enable clean notebook execution:

reproguard scan notebook.ipynb --execute

Usage Examples

Basic scan

reproguard scan examples/risky_customer_churn.ipynb

Output:

ReproGuard score: 27/100 (not_ready)
Files scanned: 1 | Issues: 8
Critical: 1 | High: 3 | Medium: 2 | Low: 2

 Severity   Code     Issue                          Location
 ────────   ────     ─────                          ────────
 CRITICAL   LEAK001  Possible preprocessing before…  risky_customer_churn.ipynb:cell 6
 HIGH       DATA001  Local machine path detected     risky_customer_churn.ipynb:line 2
 HIGH       LEAK002  Future/target-like column nam…  risky_customer_churn.ipynb:cell 4
 HIGH       LEAK007  Exception traceback found in…   risky_customer_churn.ipynb:cell 9
 MEDIUM     PII001   Email address detected          risky_customer_churn.ipynb:cell 8
 MEDIUM     REP001   Non-deterministic code witho…   risky_customer_churn.ipynb:cell 6
 LOW        NB001    Notebook cells were executed…   risky_customer_churn.ipynb
 LOW        NB002    Notebook output exists witho…   risky_customer_churn.ipynb:cell 9

Scan a GenAI project

ReproGuard detects LLM configuration risks, prompt-injection bait, and agent structure issues statically — no API keys or network access needed:

# GenAI-tuned weights: genai findings penalize 35x, handoff only 5x
reproguard scan . --profile genai

Real output from scanning a LangGraph + FastAPI agent repository:

ReproGuard score: 25/100 (not_ready)
Files scanned: 29 | Issues: 43
Points deducted by category: genai: -35, handoff: -0, privacy: -10, reproducibility: -30

 Severity   Code     Issue                       Location
 ────────   ────     ─────                       ────────
 HIGH       AGENT004 Tool accepts free-form      src/tools/research_tool.py:33
                     input without an allowlist
 HIGH       REPO001  '.env' file is present      .env
                     and not gitignored
 LOW        AGENT002 Agentic graph without       src/agents/base_agent.py:74
                     explicit recursion limit
 LOW        LLMC003  Model ID is not version-    src/config/settings.py:19
                     pinned (gpt-4o-mini)
 LOW        LLMC004  High temperature on         src/config/settings.py:24
                     agentic model (0.7)

Run --format json for machine-readable findings, or --model-card for a Markdown handoff summarizing risks and recommended actions.

Custom rules

Add org-specific checks in .reproguard.yml — they behave like built-ins (scored, reported, suppressible via checks.disabled/checks.ignore):

# .reproguard.yml
rules:
  - code: NOSAMPLE      # Flag sampling without a fixed random_state
    title: Sample without seed
    pattern: 'sample\('   # plain regex matched against source text
    category: reproducibility
    severity: medium
    confidence: 0.8
    file_patterns:
      - 'src/**/*.py'
      - '*.ipynb'
  - code: TODO001       # Track tech-debt markers
    title: TODO left in code
    pattern: '# TODO'
    severity: low

Generate reports

# All report formats
reproguard scan . --format all

# JSON only
reproguard scan . --format json

# SARIF for GitHub Code Scanning
reproguard scan . --format sarif

# Custom output directory
reproguard scan . --output-dir scan-reports

CI integration

# Fail the build if the score is too low
reproguard scan . --fail-under 50
echo $?  # Exit code 1 when score < 50 or any CRITICAL issue

GitHub Action

Scan in CI with the reusable action:

steps:
  - uses: actions/checkout@v4
  - uses: actions/setup-python@v5
    with:
      python-version: "3.11"
  - uses: vipulgote1999/ReproGuard/.github/actions/reproguard-scan@v0.3.0
    with:
      target: "."
      fail-under: 50
      profile: genai

Auto-fixes, model cards, git history

# Safe, idempotent remediations:
#   gitignore    — append .env protection to .gitignore
#   clear-counts — reset notebook execution counts (clean handoff)
#   seed         — inject missing random/numpy seeds into notebooks
reproguard scan . --fix all          # or pick one: --fix seed

# Markdown handoff artifact (risk summary + recommended actions)
reproguard scan . --model-card

# gitleaks-lite: scan git history diffs for committed secrets
reproguard scan . --git-history

# Validate llm.yaml model IDs against the OpenRouter catalog
reproguard scan . --manifest-online

# GenAI-tuned category weights (or 'profile: genai' in .reproguard.yml)
reproguard scan . --profile genai

# Parallel clean execution of notebooks (kernelspec-aware)
reproguard scan . --execute --parallel --max-workers 4

The LLM manifest validated by --manifest-online lives at the repo root:

# llm.yaml
models:
  - id: gpt-4o-2024-08-06     # pinned — never 'gpt-4o' or '*-latest'
    provider: openai
prompts:
  - id: rag
    path: prompts/rag.md      # must exist on disk

Prompt files (prompts/*.md|txt|yaml|json) are scanned as whole prompts — injection-bait phrasing, PII, and oversized prompts are flagged.

Regression detection (baseline diff)

Gate CI on new issues while existing debt is paid down gradually:

# First run: save the baseline
reproguard scan . --format json --output-dir .reproguard

# Later runs: block on new issues only
reproguard scan . --baseline .reproguard/reproguard-report.json --fail-new 0
reproguard scan . --baseline .reproguard/reproguard-report.json --fail-new-critical

New and resolved issues are printed in the terminal summary and recorded in the JSON report metadata. Exit codes: 1 when the gate trips, 2 for invalid baselines.

Scan with privacy disabled

reproguard scan . --no-privacy

Custom execution timeout

reproguard scan notebook.ipynb --execute --execution-timeout 300

Understanding Reports

Score interpretation

Report files

Reports are written to .reproguard/ by default:

.reproguard/
├── reproguard-report.json      # Structured data
├── reproguard-report.html      # Styled HTML report
└── reproguard-report.sarif     # GitHub Code Scanning compatible

Configuration

Create a .reproguard.yml in your project root:

# .reproguard.yml
exclude_paths:
  - "archive/**"
  - "tests/**"
exclude_dirs:
  - "scratch"
fail_under: 50
checks:
  disabled:
    - "LEAK005"     # Disable large tabular output check
    - "PII004"      # Disable base64 image check

Configuration is discovered by walking up from the scan path (like git). See docs/CONFIGURATION.md for the full reference.

Pre-commit Hook

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/vipulgote1999/ReproGuard
    rev: v0.3.0
    hooks:
      - id: reproguard-scan
        args: ["--fail-under", "75"]

The hook scans the entire repository on each commit and blocks the commit when the score falls below the threshold.

CI/CD Integration

GitHub Actions (with SARIF upload)

name: ReproGuard
on: [push, pull_request]
jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install reproguard
      - run: reproguard scan . --format sarif --fail-under 50
      - uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: .reproguard/reproguard-report.sarif

GitLab CI

reproguard:
  stage: test
  script:
    - pip install reproguard
    - reproguard scan . --format html --fail-under 50
  artifacts:
    paths:
      - .reproguard/

Project Structure

ReproGuard/
├── src/reproguard/
│   ├── cli.py              # Typer CLI entry point
│   ├── scanner.py          # Scan orchestrator
│   ├── models.py           # Core data models & scoring
│   ├── config.py           # .reproguard.yml loader (profiles, rules)
│   ├── python_analysis.py  # Python source analysis
│   ├── notebook.py         # Notebook parser + hidden-state analysis
│   ├── leakage.py          # ML data leakage heuristics
│   ├── privacy.py          # PII / secret scanning
│   ├── dependency.py       # Dependency file analysis
│   ├── execution.py        # Clean execution (parallel, kernel-aware)
│   ├── repo.py             # Repository hygiene checks (REPO001-007)
│   ├── genai.py            # LLM config / prompt / agent checks
│   ├── unsafe.py           # Bandit-lite unsafe Python checks
│   ├── rules.py            # Custom rule engine
│   ├── manifest.py         # llm.yaml manifest validation
│   ├── fixes.py            # --fix auto-remediation
│   ├── githistory.py       # --git-history secret scan
│   ├── modelcard.py        # --model-card generation
│   ├── report.py           # JSON/HTML report generators
│   ├── sarif.py            # SARIF 2.1.0 report generator
│   ├── plugin.py           # Check registry & filtering
│   └── utils.py            # Shared helpers
├── examples/
│   ├── risky_customer_churn.ipynb  # Notebook with intentional issues
│   ├── clean_analysis.py           # Clean script example
│   └── requirements.txt            # Example dependency file
├── docs/
│   ├── ARCHITECTURE.md     # Internal design & module docs
│   ├── CHECKS.md           # Complete issue code reference
│   ├── CONFIGURATION.md    # Configuration file reference
│   ├── GUIDES.md           # Usage guides & integrations
│   └── ROADMAP.md          # Future plans
├── pyproject.toml          # Build & tool config
└── README.md               # This file

Check Reference

See docs/CHECKS.md for full details on every check.

Design Principles

1. Local-first — Scans run entirely on your machine. No data or notebooks leave your environment.
2. Explainable rules — Every issue has a code, severity, evidence, confidence score, and suggested fix. No black boxes.
3. Low friction — CLI-first design with a single reproguard scan <target> command. Pre-commit hook, CI integration, and GitHub Action out of the box.
4. Conservative scoring — The score is transparent (penalty-based, weighted by severity and category). You can customize all penalties and weights.
5. Narrow wedge — Focused on catching pre-production risks before work enters heavier MLOps pipelines.

Limitations

ReproGuard v0.3 (alpha) uses heuristics. It flags likely risks but cannot prove every issue is real. Treat it as a review assistant, not a final governance decision.

• Leakage detection is heuristic — expect false positives (use checks.ignore to silence known-safe locations)
• Dependency parsing is intentionally lightweight (regex-based for requirements/Pipfile, YAML for conda env files, TOML for pyproject/lock files — no full resolver)
• Privacy scanning uses regex rules by default (Presidio support planned)
• Clean notebook execution depends on local kernel and dependency availability
• Data file existence checks are limited to paths referenced from notebooks/scripts

Development

# Install dev dependencies
pip install -e .[dev]

# Lint
ruff check .

# Test
pytest

# Build release artifacts and validate metadata
python -m build
python -m twine check dist/*

Releases follow the procedure in docs/RELEASING.md — see the checklist there before tagging. Changes are tracked in CHANGELOG.md.

Roadmap

• v0.2 ✅ — correctness hardening (magic handling, conda/lock dependency parsing), path-scoped ignore rules, baseline diffing, extended coverage
• v0.3 ✅ — GenAI category (27 checks), repository hygiene, custom rule engine, profiles, auto-fixes, git-history secret scan, LLM manifests, model cards, parallel + kernel-aware execution, GitHub Action
• v0.4 (planned): prompt-suite drift detection, --fix for more codes, SARIF-to-CodeQL grouping, notebook diffs, license scanning
• v1.0+: ML pipeline scanning, differential scans, team dashboard, API

See docs/ROADMAP.md for the full roadmap.

License

MIT