sparkguardian 0.1.2

Production-grade PySpark auditing, optimization, and governance framework — static + dynamic analysis with local AI assistance

SparkGuardian

Production-grade PySpark auditing, optimization, and governance framework

Tech stack

---

SparkGuardian detects PySpark anti-patterns, analyzes execution plans, generates prioritized remediation plans, and integrates into CI/CD pipelines. All 100% offline, no cloud APIs, no source code exfiltration

Table of Contents

• Features
• Installation
• Quick Start
• Rules
• Ignore System
• Configuration
• Local AI Setup
• Cloud Cost Estimator
• Audit History
• SARIF Output
• CI/CD Integration
• Licensing & Web Portal
• Deployment
• Architecture
• Development

---

Features

Feature	Description
Static AST analysis	libcst, never executes your code — 21 rules
Execution-plan analysis	shuffles, CartesianProduct, AQE from df.explain()
Safe auto-refactor	backup + dry-run preview
Local AI (Ollama)	explanations, remediation, offline RAG
Cloud cost estimator	$ waste on EMR / Databricks / GCP / Azure
Score + history	0–100 score, regression detection
Reports	terminal, JSON, Markdown, HTML, SARIF v2.1.0
CI/CD	exit codes 0/1/2 — GitHub Actions, GitLab, Jenkins
Licensing	trial tokens, offline activation, Stripe billing

Installation

pip install sparkguardian
pip install "sparkguardian[ai]"    # + local Ollama

Python 3.11+.

Quick Start

sparkguardian activate             # paste SG-… token (7-day free trial)
sparkguardian analyze app.py       # --json --html --md --sarif
sparkguardian fix app.py --dry-run
sparkguardian cost app.py --cloud databricks
sparkguardian explain-plan plan.txt
sparkguardian remediation app.py   # AI (needs Ollama)
sparkguardian license              # status

Rules

21 rules — static AST (16) + execution-plan (5). List them:

sparkguardian list-rules

Ignore System

Suppress inline or by block, with audit trail:

df.collect()                       # cy:ignore CY001
# cy:ignore-start CY008
...
# cy:ignore-end CY008

Configuration

.sparkguardian.toml — enable/disable rules, override severity, ignore paths, CI thresholds.

Local AI Setup

Optional, 100% local via Ollama:

pip install "sparkguardian[ai]"
ollama pull llama3.1
sparkguardian ai-status

Cloud Cost Estimator

Turns anti-patterns into $ waste per cloud:

sparkguardian cost app.py --cloud databricks
# 💸 Estimated waste: $2,407/month

Audit History

Track score over time, flag regressions:

sparkguardian analyze app.py --history
sparkguardian history app.py

SARIF Output

GitHub Code Scanning ready:

sparkguardian analyze src/ --sarif results.sarif

CI/CD Integration

- run: pip install sparkguardian
- run: sparkguardian analyze src/ --sarif results.sarif

Exit 0 clean · 1 critical · 2 warnings. GitLab / Jenkins samples in ci/.

Licensing & Web Portal

application/ — FastAPI landing page + license portal + Stripe billing. Live: https://sparkguardian-portal.fly.dev

• 7-day free-trial tokens (name + email → signed SG-…)
• CLI activates online once, caches locally, checks expiry offline
• Types: trial / monthly / lifetime / enterprise
• Stripe Checkout ($10/mo Pro) + webhook auto-issues Pro
• Admin dashboard: list / revoke / stats

cd application && ./run.sh          # http://localhost:8000  (/admin, /docs)

Deployment

Full app on any container host:

• Fly.io — cd application && fly deploy (fly.toml included)
• Render — Blueprint via render.yaml

See application/README.md.

Architecture

sparkguardian/   parser · rules · rewriter · ai · cost · scoring · history · reporting · licensing · cli
application/     backend (FastAPI)  ·  frontend (static portal)

Development

pip install -e ".[dev]"
pytest                 # 375 tests
ruff check . && mypy sparkguardian

License

MIT — see LICENSE.