Instalação portátil: docs/installation/quickstart.md — clone → setup → install. Delegação agêntica: forge.agentic.json (SpecialistAgenticManifest/v1). Docs: commands · skills · agents · tutorial · economy · troubleshooting

Forge Doctor Data

Deterministic diagnostics for data engineering projects. One command inspects your repository, Python environment, packaging, dependencies, git hygiene, PySpark code, and local AWS setup — no daemons, no network calls, no AI.

Why it exists

Data projects drift: a collect() slips into a hot path, requirements.txt and poetry.lock start coexisting, a .env gets committed, the Python in the venv no longer matches requires-python. These failures are cheap to catch and expensive to debug. Forge Doctor Data is a fast, offline, deterministic sensor that surfaces them with file:line precision — and explains when each pattern is actually fine.

Installation

pipx install forge-doctor-data

New here? Walk through docs/getting-started.md — install to a gated CI scan in five steps.

Or from source:

git clone https://github.com/EdgarSocrates98/forge-doctor-data
cd forge-doctor-data
pipx install .

Usage

forge-doctor-data scan .                 # full scan
forge-doctor-data scan . --check spark   # one category
forge-doctor-data scan . --ignore SPARK001
forge-doctor-data scan . --format json   # stable machine-readable contract
forge-doctor-data scan . --format html   # self-contained shareable report
forge-doctor-data scan . --format sarif  # GitHub Code Scanning
forge-doctor-data scan . --format agent  # compact bundle for agent consumers
forge-doctor-data scan . --quiet         # only problems
forge-doctor-data scan . --fail-on warning
forge-doctor-data scan . --profile security   # severity profile
forge-doctor-data scan . --files a.py b.py    # only these files (pre-commit)
forge-doctor-data scan . --no-plugins         # skip external plugins
forge-doctor-data scan . --watch         # re-scan on every file change
forge-doctor-data scan . --save-baseline .fd-baseline.json
forge-doctor-data scan . --baseline .fd-baseline.json   # only NEW findings fail
forge-doctor-data scan . --baseline .fd.json --new-only # just the new ones
forge-doctor-data scan . --save-baseline main       # named: .forge-doctor-data/baselines/main.json
forge-doctor-data scan . --baseline main --new-only # per-branch baselines
forge-doctor-data scan . --evidence-out audit/      # dated audit bundle (report+suppressions+packs)
forge-doctor-data scan . --record --keep 30         # append a history snapshot, prune to 30
forge-doctor-data history .              # recorded snapshots
forge-doctor-data history diff --last    # new/resolved findings, entity+capability+drift deltas
forge-doctor-data history trend          # finding counts over the series + debt trajectory
forge-doctor-data diff old.json new.json     # diff two saved reports
forge-doctor-data diff HEAD~1...HEAD         # diff across git refs
forge-doctor-data compatibility --to 6.0     # Glue env + migration risks
forge-doctor-data migrate glue --from 3.0 --to 4.0  # migration intelligence
forge-doctor-data workspace .                # discover projects in a monorepo
forge-doctor-data workspace scan --path .    # scan every nested project
forge-doctor-data workspace inspect .        # cross-repo graph links (defines/implements/invokes)
forge-doctor-data fleet inspect fleet.yml    # manifest-driven estate graph over many repos
forge-doctor-data fleet query fleet.yml runtimes|dependents|findings|capability
forge-doctor-data fleet report fleet.yml     # estate census + findings roll-up
forge-doctor-data fleet portfolio fleet.yml  # platforms, duplication, complexity — facts not scores
forge-doctor-data fleet regressions fleet.yml # regression dimensions shared across repos
forge-doctor-data diff base...head --semantic  # entity diff + blast radius + capability/migration intel
forge-doctor-data remediate .              # deterministic fix plans per finding/root cause
forge-doctor-data fix .                    # preview/apply safety-classified text fixes
forge-doctor-data root-cause .             # correlate findings into causal clusters
forge-doctor-data runtime inspect .        # offline runtime evidence (exported artifacts)
forge-doctor-data architecture drift .     # drift vs .forge-doctor-data/contract.yml
forge-doctor-data contract validate .      # verify a platform contract file
forge-doctor-data capabilities list .      # platform capability registry report
forge-doctor-data capabilities graph .     # capability→evidence provenance subgraph
forge-doctor-data twin inspect .           # digital twin: graph + invariant report
forge-doctor-data twin export .            # deterministic twin snapshot artifact
forge-doctor-data advise .                 # ranked, cited actions (decision intelligence)
forge-doctor-data optimize .               # ranked optimization candidates + validation commands
forge-doctor-data what-if . --change glue-version=5.1   # hypothetical-change evaluation
forge-doctor-data what-if . --change platform=bigquery  # cross-platform re-target evaluation
forge-doctor-data migrate plan .           # named deterministic migration plans
forge-doctor-data migrate plan . --from snowflake --to bigquery  # cross-platform plan
forge-doctor-data policy list .            # organization policy packs
forge-doctor-data policy report .          # compliance: violations + suppression audit
forge-doctor-data lab run                  # Forge Lab scenario suites + ground truth
forge-doctor-data incident inspect         # incident windows over recorded runtime history
forge-doctor-data reliability path .       # critical paths over data-flow edges + segment coverage
forge-doctor-data golden run               # golden-repo snapshot regression
forge-doctor-data bench run .              # performance & scale benchmark
forge-doctor-data schema contracts         # JSON Schemas for output artifacts
forge-doctor-data ontology                 # canonical vocabulary (entity/rel kinds, planes)
forge-doctor-data ontology validate .      # graph conformance vs the vocabulary
forge-doctor-data snowflake inspect .      # Snowflake warehouse model + vendor objects
forge-doctor-data bigquery inspect .       # BigQuery datasets, partitions, slots, jobs
forge-doctor-data redshift inspect .       # Redshift clusters, dist/sort keys, WLM, datashares
forge-doctor-data dbt inspect .            # dbt models, sources, tests, ref/source lineage
forge-doctor-data trino inspect .          # Trino catalogs, connectors, coordinator config, 3-part refs
forge-doctor-data analytical inspect .     # ClickHouse/Pinot/Druid tables, engines, ingestion, indexes
forge-doctor-data search inspect .         # OpenSearch/Elasticsearch templates, policies, domains
forge-doctor-data catalog inspect .        # DataHub/OpenMetadata/Glue/Unity datasets, drift, recipes
forge-doctor-data quality inspect .        # Deequ/GX/SodaCL/dbt suites, coverage, gate wiring
forge-doctor-data cloud inspect .          # vendor-neutral multi-cloud abstraction view
forge-doctor-data export . -f handoff      # portable bundle for downstream Forge tools
forge-doctor-data contracts verify b.json  # validate a bundle against published contracts
forge-doctor-data info                   # project stats, no checks
forge-doctor-data init --name my-etl     # scaffold a new project
forge-doctor-data repo                   # category shortcuts
forge-doctor-data plugins                # built-in + external checks
forge-doctor-data plugins init forge-doctor-data-x   # scaffold a plugin package
forge-doctor-data plugins lock && forge-doctor-data plugins verify  # integrity pinning
forge-doctor-data plugins install forge-doctor-data-x --dry-run     # pipx/pip wrapper
forge-doctor-data checks                 # every rule id and title
forge-doctor-data explain SPARK001       # why / when it's OK / how to fix
forge-doctor-data explain SPARK001 --json    # rule metadata for agents
forge-doctor-data trace SPARK001 src/job.py:42   # why this finding fired
forge-doctor-data diagnose driver.log      # fingerprint log errors (offline)
forge-doctor-data spark eventlog dir/      # executor loss, skew, spill, GC
forge-doctor-data lineage                # static reads/writes graph
forge-doctor-data schema diff old.avsc new.avsc  # breaking/compatible changes
forge-doctor-data graph                  # project intelligence graph
forge-doctor-data platform graph .       # canonical platform graph census (+ --json)
forge-doctor-data platform blast-radius <entity> .  # semantic impact traversal
forge-doctor-data suppressions           # audit governed exceptions
forge-doctor-data iceberg inspect .      # Iceberg model summary (ops/maintenance/risks)
forge-doctor-data iceberg merge .        # MERGE posture: source, ON cols, partition pruning
forge-doctor-data iceberg files .        # write APIs + small-file risk
forge-doctor-data iceberg compatibility  # runtime x Iceberg compat from the packs
forge-doctor-data controlm inspect .     # Control-M workflows-as-code (folders/jobs/events/risks)
forge-doctor-data airflow inspect .      # Airflow DAGs/tasks/sensors/providers + parse-time risks
forge-doctor-data terraform inspect .    # providers/modules/backends/reference graph
forge-doctor-data parquet inspect .      # Parquet dataset stats + codec/write risks
forge-doctor-data stepfunctions inspect .# ASL machines: states, integrations, graph risks
forge-doctor-data streaming inspect .    # streaming queries: source→sink, checkpoint, watermark
forge-doctor-data streaming diagnose .   # progress-metrics diagnostics (rate/state/watermark)
forge-doctor-data streaming semantics .  # derived delivery semantics per query
forge-doctor-data dynamodb inspect .     # DynamoDB tables + access patterns
forge-doctor-data neptune inspect .      # Neptune clusters + query shapes
forge-doctor-data lakeformation inspect .# governance, cross-account shares, tag usage
forge-doctor-data emr inspect .          # EMR EC2/Serverless/EKS deep model
forge-doctor-data databricks inspect .   # jobs/clusters/DBR/UC + bundles
forge-doctor-data delta inspect .        # Delta tables, ops, protocol features
forge-doctor-data athena inspect .       # workgroups, queries, result locations
forge-doctor-data lambda inspect .       # functions, triggers, runtimes, data-plane calls
forge-doctor-data kafka inspect .        # MSK clusters/topics/consumer groups
forge-doctor-data kinesis inspect .      # streams, EFO, Firehose
forge-doctor-data flink inspect .        # Flink jobs, state, checkpoints, savepoints
forge-doctor-data data-model inspect .   # cross-domain access-style inspection
forge-doctor-data iac                    # IaC checks (Terraform/CloudFormation)
forge-doctor-data cache                  # incremental-analysis stats
forge-doctor-data sbom                   # CycloneDX 1.5 of the project
forge-doctor-data knowledge verify       # pack freshness/provenance
forge-doctor-data knowledge audit        # fresh/stale/expired/source status
forge-doctor-data doctor                 # environment self-check
forge-doctor-data mcp                    # JSON-RPC stdio server for agents
forge-doctor-data hosts                  # observe AI host CLIs; `hosts <h> --launch` opens one (TTY)
forge-doctor-data portal                 # optional local web UI over the real services (127.0.0.1)
forge-doctor-data agent manifest .       # compact domains, risks, capabilities, refs
forge-doctor-data agent context . --budget 8000
forge-doctor-data agent delta . --since context.json
forge-doctor-data agent evidence entity:<id> .
forge-doctor-data collector validate evidence.json
forge-doctor-data inspect iceberg .      # convergent: inspect <domain> (alias of `iceberg inspect`)
forge-doctor-data project status         # generated status; --check gates doc drift
forge-doctor-data lsp                    # editor diagnostics (pip install .[lsp])
forge-doctor-data version

One scan can emit several reports — --emit is repeatable and takes FMT (stdout) or FMT:PATH:

forge-doctor-data scan . --emit text --emit sarif:report.sarif --emit html:report.html

Example output

╭─ Forge Doctor Data ───────────────────────╮
│ Project  /home/me/etl                │
│ Version  1.0.0                       │
│  Checks  51                          │
╰──────────────────────────────────────╯

Python
  ✓ PY001 Python version
  ⚠ PY002 requires-python pyproject.toml
      not declared
      → Declare requires-python in pyproject.toml

Spark
  ⚠ SPARK001 collect() src/jobs/customer.py:182 NEW
      collect() moves all rows to the driver; verify the volume is bounded.
      [confidence: high]
      > df_final.collect()

╭─────────────── Summary ───────────────╮
│ Passed 17   Info 3   Warnings 4   Errors 0   │
│ 4 warning(s)                                 │
│ baseline: +1 new, -2 fixed, 33 pre-existing  │
╰──────────────────────────────────────────────╯

Baselines

Save a scan once, then compare every later run against it — new findings get a NEW marker and --fail-on only counts those, so pre-existing debt never breaks CI:

forge-doctor-data scan . --save-baseline .forge-doctor-data-baseline.json
forge-doctor-data scan . --baseline .forge-doctor-data-baseline.json --fail-on warning

HTML reports

forge-doctor-data scan . --format html --output report.html

Produces a single self-contained HTML file — same content as the terminal report, shareable with the team. --no-color (or NO_COLOR=1) disables ANSI colors; --output also works for plain text reports.

Checks

Full per-rule documentation (including when it's OK) lives in docs/checks.md.

JSON output

forge-doctor-data scan . --format json

{
  "tool": {"name": "forge-doctor-data", "version": "1.0.0"},
  "schema_version": "3.0",
  "version": "1.0.0",
  "project": {"name": "etl"},
  "summary": {"passed": 17, "info": 3, "warnings": 4, "errors": 0},
  "results": [
    {"check_id": "SPARK001", "severity": "warning", "category": "spark",
     "file": "src/jobs/customer.py", "line": 182, "message": "...", "recommendation": "...",
     "fingerprint": "7998e28bb13c13ac", "confidence": "high",
     "evidence": "df_final.collect()", "tags": ["performance"],
     "is_new": true}
  ],
  "baseline": {"new": 1, "fixed": 2, "existing": 33}
}

The JSON contract is stable and meant for CI, GitHub, and agent consumers — forge-doctor-data schema contracts emits the JSON Schemas for every public artifact, forge-doctor-data contracts verify validates artifacts against them, and docs/api.md + docs/contracts.md record the versioning and interop rules. Every finding carries a stable fingerprint; optional fields (confidence, evidence, evidence_kind, tags, docs_uri, source, fixable, column, end_line, end_column) appear only when set. is_new/baseline appear only when --baseline is in use. evidence_kind classifies the fact's source plane (static/config/observed_metadata/runtime/derived).

SARIF & GitHub Code Scanning

forge-doctor-data scan . --format sarif -o forge-doctor-data.sarif

SARIF 2.1.0 with rules, locations, snippets, partialFingerprints and fixes — upload via github/codeql-action/upload-sarif or use the composite action in action.yml, which installs the CLI, scans, and uploads SARIF in one step.

Pre-commit

.pre-commit-hooks.yaml ships a forge-doctor-data hook; pre-commit feeds it the changed filenames via --files, so only the files under review produce findings.

Profiles

--profile re-maps severities per audience: default, strict (info → warning), security (supply-chain findings escalate), spark-performance, glue-migration, production (strict + security).

Compatibility reports

forge-doctor-data compatibility . --to 6.0
forge-doctor-data migrate glue --from 4.0 --to 6.0

Detects the project's Glue/Spark/Python/Java/Iceberg environment and lists migration risks from bundled knowledge packs (src/forge_doctor_data/knowledge/*/), so version facts evolve without engine changes. migrate glue adds real signals: version pins in code, DynamicFrame usage, dependencies, and Terraform/CloudFormation pins.

Runtime diagnosis

forge-doctor-data diagnose driver.log       # known-error fingerprinting
forge-doctor-data spark eventlog dir/       # executor loss, skew, spill, GC
forge-doctor-data spark plan plan.txt       # pathological physical-plan operators
forge-doctor-data spark logs executor.log   # signatures + runtime patterns

Deterministic and offline: error signatures live in knowledge/errors/*.json; event logs/plans are parsed with the stdlib.

Data intelligence

forge-doctor-data lineage                   # datasets read/written per job
forge-doctor-data lineage --format openlineage
forge-doctor-data schema diff old.avsc new.avsc
forge-doctor-data schema diff HEAD~1...HEAD --path .
forge-doctor-data graph --format mermaid    # jobs + datasets + infra + DAGs

Static lineage and the intelligence graph are built from the shared semantic index — one ast.parse per file feeds Spark checks, Glue checks, lineage, and trace. No target code is ever imported or executed.

Incremental analysis & policy

--cache (default on locally, off in CI unless explicit) stores per-file analysis facts in the platform user cache (%LOCALAPPDATA%\forge-doctor-data\cache, ~/Library/Caches/forge-doctor-data, $XDG_CACHE_HOME/forge-doctor-data; override with FORGE_DOCTOR_DATA_CACHE_DIR) — never inside the scanned repo, so a hostile checkout can't poison it. Entries are keyed by repo + tool + analyzer schema version and carry dependency provenance: editing a producer module re-analyzes its importers. forge-doctor-data cache shows stats; cache clean removes it. --stats reports per-check timings and cache hit rate on stderr.

[tool.forge-doctor-data.policy]
extends = "strict"          # base profile
[tool.forge-doctor-data.policy.rules.SPARK001]
severity = "error"          # per-rule override
[tool.forge-doctor-data.policy.rules.CI002]
enabled = false             # kill switch

[[tool.forge-doctor-data.suppressions]]
rule = "SPARK001"
path = "src/legacy/**"      # glob-scoped
reason = "migration in progress"
owner = "@data"
expires = "2026-12-31"      # past expiry reactivates the finding
approved_by = "@sec-lead"   # governance sign-off (POLICY011 if a pack requires it)

forge-doctor-data suppressions audits every exception as ACTIVE / EXPIRED / UNUSED with owner and approver; expired ones emit POLICY001 warnings instead of suppressing. A policy pack with require_approval: true makes unapproved suppressions emit POLICY011 findings — and packs can extends: a shared org pack (child rules override by id).

Workspace

forge-doctor-data workspace --path <dir> discovers nested pyproject.toml projects; workspace scan runs each and aggregates with a project prefix; workspace diff base...head compares findings per subproject.

Agent bundle

--format agent emits a minimal {id, sev, loc, fp} per finding — built for LLM/agent consumers where every token counts; pair with forge-doctor-data explain <ID> --json for full rule metadata on demand.

Configuration

Zero configuration works. Optional, in the scanned project's pyproject.toml:

[tool.forge-doctor-data]
exclude = ["tests/fixtures/**"]
ignore = ["SPARK001"]

# Trust model: `trusted` gates BEFORE plugin code loads (distribution or
# entry-point names only - check ids can't gate code that hasn't run).
# `allow` additionally accepts check ids as a post-load filter.
[tool.forge-doctor-data.plugins]
trusted = ["forge-doctor-data-databricks"]

# Post-load per-check filter (requires the plugin to load first).
[tool.forge-doctor-data.plugins.checks]
enabled = ["DBX001", "DBX002"]

[tool.forge-doctor-data.aws]
ignore = ["AWS002"]

Exit codes

0 clean · 1 errors found (or --fail-on warning hit) · 2 internal/usage error.

Integrations

• MCP (forge-doctor-data mcp [--root DIR]): zero-dependency JSON-RPC stdio server — scan_project, explain_rule, check_compatibility, get_lineage, diagnose_log, diff_findings, plus behavioral tools get_execution_baseline, get_regressions, get_runtime_correlations, get_incident_explanation, get_critical_path, get_capacity_signals, get_portfolio_summary, and forge-doctor-data://rules/ID / forge-doctor-data://knowledge/D/N resources. --root sandboxes every tool path argument to that tree.
• LSP (pipx inject forge-doctor-data pygls lsprotocol, forge-doctor-data lsp): publishes diagnostics on open/change/save, using the workspace root and unsaved-buffer contents (debounced; clears resolved findings).
• SBOM (forge-doctor-data sbom): CycloneDX 1.5 covering all locked dependencies (declared + transitive) with a dependency graph, plugins, knowledge packs and forge-doctor-data itself; deterministic serial number.

Architecture

cli/ (Typer) → ScanService → runner → checks → semantic index/analyzers
     → cache → renderers (Rich/JSON/JSONL/HTML/SARIF/agent)

beside it: platform graph → capabilities / semantic diff + change intel /
migration / what-if / workspace / fleet / history / contracts / lab /
golden / bench / policy packs

Checks are small classes implementing a Check protocol (id/title/category/run(ctx) -> list[CheckResult]). They never print and never see the terminal. See docs/architecture.md.

Plugins

External packages contribute checks via the forge_doctor_data.checks entry-point group — install with pipx inject forge-doctor-data forge-doctor-data-<ext>. A broken plugin degrades to a warning, never a crash.

Plugins run in-process with the same privileges as the CLI — treat them as trusted code. --no-plugins disables them per run; [tool.forge-doctor-data.plugins].allow restricts which ones may load. Plugin findings carry a source field naming their distribution.

Development

git clone https://github.com/EdgarSocrates98/forge-doctor-data
cd forge-doctor-data
poetry install
poetry run forge-doctor-data scan .
poetry run pytest
poetry run ruff check .
poetry run mypy

Contributing

See CONTRIBUTING.md. New checks: implement the Check protocol, pick a stable id, register it in the category module's CHECKS list, add tests and a docs/checks.md entry.

License

MIT — see LICENSE.

Graph Studio

forge-doctor-data graph view|ui --path . projects the evidence graph through forge/ForgeGraphView/v1 and serves the embedded local explorer (declinable at install via --components). Full docs: the-forge/docs/graph-studio/.