AWS Glue Toolkit

A streamlined CLI utility designed to simplify the AWS Glue development lifecycle.

Disclaimer: This is an independent, community-maintained project. It is not affiliated with, endorsed by, or sponsored by Amazon Web Services (AWS). AWS, AWS Glue, and related marks are trademarks of Amazon.com, Inc. or its affiliates.

Installation

Requires Python 3.11+ and Docker for check, run, test, and gtk build --mode container (assumed installed, never installed by gtk). Default gtk build (--mode host) does not need Docker. Installing the package adds the gtk command and a compatible pip release.

pip install aws-glue-toolkit

Example

A Glue job is a directory with a pyproject.toml and a source tree. Given this layout:

my-glue-job/
  pyproject.toml
  src/
    __main__.py
  tests/
    test_example.py

configure the job in pyproject.toml:

[project]
name = "my-glue-job"
version = "0.1.0"
dependencies = ["pandas>=2"]

[tool.aws-glue-toolkit]
glue_version = "5.1"
source = "src"
script = "__main__.py"
tests = "tests"

then run:

cd my-glue-job
gtk check .
gtk build .
gtk run .
gtk test .

Configuration

Each job is a directory containing pyproject.toml. Unknown keys are ignored. The source directory and entry script must exist before gtk runs.

Pip configuration

For container pip (check, build --mode container, and dep install on run / test), gtk snapshots the host’s effective pip configuration (files + PIP_*, env wins) into a temporary pip.conf, mounts it, and sets PIP_CONFIG_FILE. Host-local keys (cache-dir, cert, target, and similar) are omitted. With gtk build (default --mode host), host pip config applies directly.

# ~/.config/pip/pip.conf  or:
export PIP_EXTRA_INDEX_URL="https://my.company/simple"
gtk check .

Commands

[JOB-DIR] is the job directory path, passed as a positional argument or with --job-dir [JOB-DIR] (default: .). Docker must be available for check, run, test, and gtk build --mode container (Glue image pulled on first use). Default gtk build (--mode host) does not need Docker. gtk never installs Docker.

For run / test: job dir mounted at /home/hadoop/workspace; non-empty project.dependencies install into an ephemeral PYTHONPATH target (Glue pins as constraints); stdio pass through; exit code is the container command’s (or pip’s if install fails). Same dependency forms as check / build.

check

Resolves project.dependencies in the Glue image for glue_version against bundled pins. Does not write files.

build

Writes a dependencies zip (host: .py under source) and a gluewheels zip.

• --mode host (default): host packaging (no Docker). Path/VCS via pip wheel --no-deps; other packages via pip download --platform --only-binary=:all:, or sdist→wheel on the host when no compatible wheel exists. Wheels must be any or the Glue pip_platform (use --mode container for compiled packages that need a worker-arch build). VCS needs network and git on the host.
• --mode container: pip wheel in the Glue image (linux/amd64). On ARM hosts this needs QEMU (or similar) unless you use --mode host for portable wheels.

Path deps must be installable packages (pyproject.toml or setup.py); loose job modules belong under source. Both zips are always written; gluewheels omits packages already pinned on the image at the same version.

Dependency forms

Editable installs (-e) are rejected.

run

spark-submit on the entry script. Passes --JOB_NAME from project.name unless overridden. Extra --key value tokens after [JOB-DIR] go to getResolvedOptions (pass an explicit [JOB-DIR] when using .). Shuts down the Spark driver so the container returns.

test

python3 -m pytest. PYTHONPATH includes source (and the install target when deps are present). Default target is the configured tests dir; if the first forwarded token starts with -, that dir is still passed first (gtk test . -v); if it is a path, only those tokens go to pytest.

Exit codes

Success is 0; unhandled errors are 1. For run / test, a successful Docker launch returns the container command’s exit code (not limited to the table above).

Architecture

Two-layer module layout (core and shell) is documented in docs/architecture.md.

Contributing

Contributions are welcome. See CONTRIBUTING.md for setup and workflow. Participants are expected to follow the Code of Conduct. Report security vulnerabilities privately via SECURITY.md.

License

This project is licensed under the MIT License — see the LICENSE file for details.