AWS Glue Toolkit

A streamlined CLI utility designed to simplify the AWS Glue development lifecycle.

Disclaimer: This is an independent, community-maintained project. It is not affiliated with, endorsed by, or sponsored by Amazon Web Services (AWS). AWS, AWS Glue, and related marks are trademarks of Amazon.com, Inc. or its affiliates.

Installation

Requires Python 3.11+ and Docker for run, test, and --mode container on check/build (never installed by gtk). Default gtk check / gtk build (--mode host) need no Docker. Installing the package adds the gtk command and a compatible pip release.

pip install aws-glue-toolkit

Example

A Glue job is a directory with a pyproject.toml and a source tree. Given this layout:

my-glue-job/
  pyproject.toml
  src/
    __main__.py
  tests/
    test_example.py

configure the job in pyproject.toml:

[project]
name = "my-glue-job"
version = "0.1.0"
dependencies = ["pandas>=2"]

[tool.aws-glue-toolkit]
glue_version = "5.1"
source = "src"
script = "__main__.py"
tests = "tests"

then run:

cd my-glue-job
gtk check .
gtk build .
gtk run .
gtk test .

Configuration

Each job is a directory containing pyproject.toml. Unknown keys are ignored. The source directory and entry script must exist before gtk runs.

Pip configuration

For container pip (check/build --mode container, and dep install on run/test), gtk snapshots the host’s effective pip configuration (files + PIP_*, env wins) into a temporary pip.conf, mounts it, and sets PIP_CONFIG_FILE. Host-local keys (cache-dir, cert, target, and similar) are omitted. Default --mode host uses host pip config as-is.

# ~/.config/pip/pip.conf  or:
export PIP_EXTRA_INDEX_URL="https://my.company/simple"
gtk check .

Commands

[JOB-DIR] is the job directory path (positional or --job-dir; default .). Docker is required for run, test, and --mode container (Glue image pulled on first use). Default --mode host does not. gtk never installs Docker.

build|check --mode chooses where pip runs: host (default) local pip for worker-oriented resolve/packaging (no Docker); container worker-arch Glue/build container (may need QEMU on ARM).
run|test --platform chooses which Glue image arch to run: native (default) matches your machine; worker matches Glue job workers (may need QEMU on ARM).

For run / test: job dir mounted at /home/hadoop/workspace; non-empty job dependencies install into an ephemeral PYTHONPATH target (Glue pins as constraints); stdio pass through; exit code is the container command’s (or pip’s if install fails). Same dependency forms as check / build.

check

Verifies job dependencies against Glue runtime pins. Does not write zip artifacts.

• --mode host (default): same gluewheels recipe as build --mode host (temp dir, discarded). No Docker.
• --mode container: pip install --dry-run in the Glue image (worker-arch; QEMU on ARM).

build

Writes a dependencies zip (host: .py under source) and a gluewheels zip.

• --mode host (default): host packaging (no Docker). Path/VCS via pip wheel --no-deps; other packages via pip download --platform --only-binary=:all:, or sdist→wheel on the host when no compatible wheel exists. Wheels must be any or the Glue pip_platform (use --mode container for compiled packages that need a worker-arch build). VCS needs network and git on the host.
• --mode container: pip wheel in the Glue image (worker-arch). On ARM hosts this needs QEMU (or similar) unless you use --mode host for portable wheels.

Path deps must be installable packages (pyproject.toml or setup.py); loose job modules belong under source. Both zips are always written; gluewheels omits packages already pinned on the image at the same version.

Dependency forms

Editable installs (-e) are rejected.

Relative file: paths are not a portable PEP 508 form for other tools. Prefer them under [tool.aws-glue-toolkit].dependencies (merged after [project].dependencies):

[project]
name = "my-glue-job"
version = "0.1.0"
dependencies = ["pandas>=2"]

[tool.aws-glue-toolkit]
glue_version = "5.1"
source = "src"
script = "__main__.py"
dependencies = [
  "my-lib @ file:./libs/my-lib",
  "shared @ file:../packages/shared",
]

You may still list relative file: specs under [project].dependencies; gtk accepts either location.

run

spark-submit on the entry script. Passes --JOB_NAME from project.name unless overridden. Extra --key value tokens after [JOB-DIR] go to getResolvedOptions (pass an explicit [JOB-DIR] when using .). Shuts down the Spark driver so the container returns.

• --platform native (default): omit Docker --platform so the official multi-arch Glue image matches the host (arm64 on Apple Silicon / ARM CI). Local CPU may differ from cloud Glue workers (amd64).
• --platform worker: force the Glue worker Docker platform from runtime metadata (today linux/amd64). On ARM this needs QEMU (or an amd64 machine); there is no fallback to native.

test

python3 -m pytest. PYTHONPATH includes source (and the install target when deps are present). Default target is the configured tests dir; if the first forwarded token starts with -, that dir is still passed first (gtk test . -v); if it is a path, only those tokens go to pytest.

Same --platform native|worker as run. Prefer --platform worker (or amd64 CI / real Glue) when validating native libs or Spark bits that can differ by CPU.

Exit codes

Success is 0; unhandled errors are 1. For run / test, a successful Docker launch returns the container command’s exit code (not limited to the table above).

Architecture

Two-layer module layout (core and shell) is documented in docs/architecture.md.

Contributing

Contributions are welcome. See CONTRIBUTING.md for setup and workflow. Participants are expected to follow the Code of Conduct. Report security vulnerabilities privately via SECURITY.md.

License

This project is licensed under the MIT License — see the LICENSE file for details.