glue-doctor

Audit CLI for AWS Glue jobs. It reads job definitions, run history and the S3 lifecycle rules of Spark UI event logs (read-only), reports configuration risks, and exits non-zero so it can gate CI.

Install

uvx glue-doctor audit --region eu-central-1
# or
uv tool install glue-doctor

Requires Python 3.13 or newer.

Quick start

glue-doctor audit --profile my-profile --region eu-central-1
glue-doctor audit --format json | jq '.summary'
glue-doctor audit --job "etl-*" --fail-on error

Example output:

┏━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Severity ┃ Job        ┃ Check                ┃ Message                                                             ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ warning  │ etl-orders │ timeout-too-generous │ Timeout is 2880 min but the median successful run takes 12 min      │
│          │            │                      │ (240x)                                                              │
│          │            │                      │ Set the timeout closer to the observed runtime so a hung run is     │
│          │            │                      │ stopped early                                                       │
│ info     │ etl-legacy │ glue-version         │ Glue 4.0 (Spark 3.3.0, Python 3.10) is older than 5.1               │
│          │            │                      │ Upgrade to Glue 5.1 or newer. Glue 6.0 runs Spark 4 (ANSI SQL mode  │
│          │            │                      │ on by default) and costs less, so validate your job before          │
│          │            │                      │ migrating                                                           │
└──────────┴────────────┴──────────────────────┴─────────────────────────────────────────────────────────────────────┘
2 finding(s) in 34 job(s)

Checks

Streaming jobs are skipped by the timeout checks; Python shell jobs are skipped by the version, observability and Spark UI checks. glue-doctor audit --help lists every check.

Known limits of spark-ui-no-lifecycle:

• A rule counts as expiring when it has an Expiration by days or date. On a bucket with versioning enabled that only adds delete markers: the data stays as noncurrent versions unless the rule also sets NoncurrentVersionExpiration, which the check does not inspect (it would need s3:GetBucketVersioning).
• An Expiration date far in the future still counts as expiring.
• A bucket in another region is followed through S3's redirect. If the region cannot be resolved without an extra HeadBucket call (which the tool never makes), that bucket is reported as a collect-error with the code BucketRegionUnresolved, and the rest of the audit goes on.
• Reports contain the event-log location of each job (bucket and prefix). Glue's default event-log bucket name includes the account ID, so do not paste unredacted reports in public places.

Exit codes

Configuration

Optional. glue-doctor reads ./glue-doctor.toml (or --config PATH); flags override the file. Unknown keys are errors, so a typo never silently disables a check.

fail_on = "warning"          # error | warning | info | never
region  = "eu-central-1"
profile = "my-profile"

[jobs]
include = ["*"]
exclude = ["scratch-*"]

[checks.timeout-too-generous]
severity    = "warning"
max_ratio   = 10.0           # timeout / median of successful runs
history     = 20             # last N successful runs
min_runs    = 5              # fewer runs: fall back to cap_minutes
cap_minutes = 1440
min_timeout_minutes = 120    # ignore timeouts up to this value, whatever the ratio

[checks.timeout-too-tight]
p95_ratio = 0.8

[checks.glue-version]
warn_below = "5.0"
info_below = "5.1"

[checks.observability-off]
enabled = false

[[suppress]]
check  = "timeout-too-generous"
job    = "nightly-*"
reason = "Highly variable runtime"   # required
until  = 2027-01-31                  # optional; after this date the finding returns

Any check accepts enabled and severity (error, warning or info); a severity override applies to every finding of that check. The region is read from AWS_DEFAULT_REGION when neither the flag nor the file sets it.

Permissions

glue-doctor calls glue:GetJobs, glue:GetJobRuns and s3:GetLifecycleConfiguration (plus sts:GetCallerIdentity, which needs no permission). The S3 call reads the lifecycle configuration of each bucket that holds Spark UI event logs; the tool never lists or reads objects. A minimal policy:

{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": ["glue:GetJobs", "glue:GetJobRuns"], "Resource": "*" },
    {
      "Effect": "Allow",
      "Action": "s3:GetLifecycleConfiguration",
      "Resource": "arn:aws:s3:::example-event-logs-bucket"
    }
  ]
}

If you do not want to grant the S3 permission, disable the check; no S3 call is then made:

[checks.spark-ui-no-lifecycle]
enabled = false

Without the permission and with the check enabled, every job with Spark UI enabled is reported as a collect-error, because a failed read must never look like a clean audit. S3 directory buckets are not supported and are reported the same way. Any other AWS operation is blocked inside the tool. Only Glue's own special job arguments are read from a job definition; your custom arguments (which can hold secrets) never enter a report.

JSON output

--format json prints a document validated by glue_doctor/schemas/report-v1.json (schema_version: 1) on stdout; warnings and errors go to stderr.

Development

make sync    # install dependencies
make check   # lint + format + test

License

Apache-2.0.