Gluekit

Gluekit works with one active AWS Glue job in a local workspace. Activate a job, edit its local definition or arguments, convert its script and notebook, then push or pull files through AWS Glue and S3.

gluekit activate my-job --profile my-aws-profile
gluekit status
gluekit settings show
gluekit settings set --worker-type G.1X --number-of-workers 5
gluekit params set --temp-dir s3://my-bucket/tmp/
gluekit script convert
gluekit notebook convert
gluekit push --to aws --dry-run
gluekit push --to aws

Run directly from PyPI with uvx gluekit --help, or install the command with uv tool install gluekit. Contributors can run the checkout with uv run gluekit.

Commands

Command	Purpose
activate <job>	Select one job and optionally an AWS profile for later commands. activate --clear clears the selection.
settings show/set	Inspect or change local Glue job fields such as Role, WorkerType, and Timeout.
params show/set	Inspect or change DefaultArguments and NonOverridableArguments.
params save <key> <value>	Save a reusable value for the active job; add --profile or --global for that scope. params saved [job] inspects these values.
script convert / notebook convert	Convert to the other representation.
script sync / notebook sync	Refresh generated metadata from the local job definition.
script import-settings / notebook import-settings	Import represented metadata into the local job definition.
push --to aws / pull --from aws	Transfer the full selected job. Pass a job name explicitly to override the selection; pass '*' explicitly for all jobs.
script push/pull / notebook push/pull	Transfer only that artifact.
init, build, status	Set up profile values, build local dependencies, and inspect the active job.
local, run	Set up local fixtures and run Glue scripts locally.

The default transfer endpoint is AWS. The --to and --from flags accept aws; a GitHub transfer backend is planned separately. Transfer commands use local files as the other endpoint, and do not need --include or --exclude. To inspect the available options, use gluekit <command> --help.

Script and notebook conversion

uvx gluekit notebook convert my-job  # notebook → clean Python script
uvx gluekit script convert my-job    # script → notebook with Glue magics

Omit my-job to use the active job. --artifact-only skips the local job configuration and generated magics; --dry-run writes neither artifacts nor state.

Notebook exports contain Python and commented markdown, with no percent cell markers, generated Glue configuration header, or commented pyproject/PEP 723 metadata. Existing percent markers are removed as well. Python indentation and user comments are retained; Gluekit does not reformat executable code. Non-Python magics such as SQL remain as commented documentation, not executable Python.

Cell boundaries live separately under notebook_cells["my-job"] in .gluekit/state.json. The markers object maps 1-based script line numbers to the marker inserted before that line when reconstructing notebook cells:

{"markers": {"1": "#%%", "9": "#%%", "15": "#%%[markdown]"}}

Each entry also records script/notebook paths and line hashes. At conversion time, Python's standard-library difflib relocates boundaries after insertions, replacements, and deletions; ast moves unsafe boundaries to the start of the containing top-level statement. Functions, decorators, classes, loops, with, try/except/finally, multiline expressions, and match blocks stay intact. Markdown is restored from its script comments without accumulating comment prefixes. The map updates on conversion, not continuously while editing; no watcher or extra dependency is needed. Line-diff matching is best-effort for moved or repeated code, but AST validation still prevents a boundary from splitting a statement.

For a script without saved boundaries or explicit percent markers, Gluekit groups global setup and starts cells around top-level definitions and compound statements. Nested blocks remain in their containing cell. Existing explicit markers take precedence over saved boundaries and are also checked for scope safety. Invalid Python causes conversion to fail before writing the notebook or its saved map.

Script-to-notebook conversion generates the existing Glue session magics from the local job configuration, but omits %iam_role. It replaces script-only job.commit() statements with no-ops so otherwise empty suites remain valid. notebook sync also omits the role, and importing a notebook without a role leaves the configured job role intact. Explicitly authored role magics can still be imported. See the AWS interactive-session magic reference.

script sync remains an explicit opt-in to the older commented Glue configuration representation; a clean notebook export does not contain settings for script import-settings. Use notebook import-settings or edit the job configuration instead. Saved cell metadata is local workspace state, never an AWS job parameter.

Workspace configuration

Gluekit owns three validated files with distinct responsibilities:

• .gluekit/config.json stores declarative workspace, job, and profile overrides. Values use nested native Glue fields; each narrower scope merges over the prior one by mapping key. Lists replace as a whole, and a JSON null explicitly removes an inherited field.
• .gluekit/state.json stores one active job, its profile and configuration directory, the independently selected local fixture file, and notebook round-trip metadata.
• .gluekit/local.json stores local S3 mounts and explicit objects plus SSM fixtures.

Activation only selects context; it never materializes profile values into the base job JSON. Commands that need a runnable definition resolve the effective configuration in memory as workspace defaults, job defaults, profile defaults, and profile/job overrides. Explicit job patterns and "*" are resolved independently of the active job. A plain build does not rewrite job files, scripts, notebooks, or cell metadata; use --package-whl for explicit package integration and --push for AWS transfer.

Existing .gluekit/glue_set.json files are validated and migrated once into the new files without modifying the legacy file. Invalid JSON, unknown fields in the new contract, or invalid values fail with their file and field path; Gluekit does not reset the file. Writes validate the complete candidate and replace the target atomically. The old checkout command and deprecated aliases remain unregistered.

CLI reference

Auto-generated from live CLI --help output.

gluekit --help

Usage: gluekit [OPTIONS] COMMAND [ARGS]...

Options:
  --install-completion  Install completion for the current shell.
  --show-completion     Show completion for the current shell, to copy it or
                        customize the installation.
  --help                Show this message and exit.

Commands:
  activate  Replace the active job.
  build     Build local artifacts; only touches AWS when --push is used.
  init      Initialize local repo-level Gluekit settings from AWS CLI...
  pull      Transfer the full active job between local files and AWS.
  push      Transfer the full active job between local files and AWS.
  run       Run a Glue script locally with emulated Glue libraries and...
  status    Show the active job and its local-to-S3 artifact mappings.
  settings  Edit the active Glue job definition.
  params    Edit the active job's Glue arguments.
  script    Work with the active job's script.
  notebook  Work with the active job's notebook.
  local     Local Glue fixtures and setup.

gluekit activate --help

Usage: gluekit activate [OPTIONS] [JOB_NAME]

  Replace the active job.

Arguments:
  [JOB_NAME]  Glue job name to activate for later commands.

Options:
  --config-dir PATH   Directory containing Glue job config files.  [default:
                      glue/configs]
  -p, --profile TEXT  AWS CLI credential profile name to store as the active
                      gluekit profile scope for saved params.
  --dry-run           Show gluekit profile-scoped config updates without writing
                      files.
  --clear             Remove the active job selection without deleting saved
                      params or local files.
  --help              Show this message and exit.

  **Examples**

  - `gluekit activate my-job`

  - `gluekit activate --clear`

gluekit status --help

Usage: gluekit status [OPTIONS]

  Show the active job and its local-to-S3 artifact mappings.

Options:
  --config-dir PATH  Directory containing Glue job config files.
  --help             Show this message and exit.

  **Examples**

  - `gluekit status`

  - `gluekit status --config-dir glue/configs`

gluekit settings --help

Usage: gluekit settings [OPTIONS] COMMAND [ARGS]...

  Edit the active Glue job definition.

Options:
  --help  Show this message and exit.

Commands:
  show  Show the active job's local Glue definition.
  set   Manage local AWS Glue job-definition properties.

gluekit settings show --help

Usage: gluekit settings show [OPTIONS] [JOB_NAME]

  Show the active job's local Glue definition.

Arguments:
  [JOB_NAME]

Options:
  --config-dir PATH
  --help             Show this message and exit.

gluekit settings set --help

Usage: gluekit settings set [OPTIONS] [JOB_NAME]

  Manage local AWS Glue job-definition properties.

Arguments:
  [JOB_NAME]  Glue job name. Defaults to the active job selection.

Options:
  --description TEXT              Set the job description.
  --role TEXT                     Set the IAM role for the job.
  --glue-version TEXT             Set the Glue version.
  --worker-type TEXT              Set the worker type.
  --number-of-workers INTEGER     Set the number of workers.
  --timeout INTEGER               Set the job timeout in minutes.
  --max-retries INTEGER           Set the max retry count.
  --execution-class TEXT          Set the execution class.
  --max-concurrent-runs INTEGER   Set the max concurrent runs.
  --job-run-queuing-enabled / --no-job-run-queuing-enabled
                                  Enable or disable job run queuing.
  --security-configuration TEXT   Set the security configuration name.
  --maintenance-window TEXT       Set the maintenance window.
  --log-uri TEXT                  Set the log URI.
  --command-name TEXT             Set Command.Name.
  --python-version TEXT           Set Command.PythonVersion.
  --runtime TEXT                  Set Command.Runtime.
  --remote-script-location TEXT   Set Command.ScriptLocation.
  --local-script-location TEXT    Set SourceControlDetails.ScriptLocation.
  --notebook-location TEXT        Set SourceControlDetails.NotebookLocation and
                                  NotebookPath.
  --source-control-provider TEXT  Set SourceControlDetails.Provider.
  --source-control-owner TEXT     Set SourceControlDetails.Owner.
  --source-control-repository TEXT
                                  Set SourceControlDetails.Repository.
  --source-control-branch TEXT    Set SourceControlDetails.Branch.
  --source-control-folder TEXT    Set SourceControlDetails.Folder.
  --add-connection TEXT           Add one or more Glue connections.
  --remove-connection TEXT        Remove one or more Glue connections.
  --dry-run                       Show what would be updated without writing
                                  files.
  --config-dir PATH               Directory containing Glue job config files.
  --help                          Show this message and exit.

  **Examples**

  - `gluekit settings set --description 'Updated job description' --glue-version
  5.0`

  - `gluekit settings set my-job --worker-type G.1X --number-of-workers 5`

gluekit params --help

Usage: gluekit params [OPTIONS] COMMAND [ARGS]...

  Edit the active job's Glue arguments.

Options:
  --help  Show this message and exit.

Commands:
  show   Show the active job's local Glue arguments.
  set    Manage local AWS Glue runtime and job arguments.
  save   Save a reusable value for the active job, profile, or all jobs.
  saved  Show stored local Glue config params.

gluekit params show --help

Usage: gluekit params show [OPTIONS] [JOB_NAME]

  Show the active job's local Glue arguments.

Arguments:
  [JOB_NAME]

Options:
  --config-dir PATH
  --help             Show this message and exit.

gluekit params set --help

Usage: gluekit params set [OPTIONS] [JOB_NAME]

  Manage local AWS Glue runtime and job arguments.

Arguments:
  [JOB_NAME]  Glue job name. Defaults to the active job selection.

Options:
  --add-additional-python-modules TEXT
                                  Add one or more values to --additional-python-
                                  modules.
  --remove-additional-python-modules TEXT
                                  Remove one or more values from --additional-
                                  python-modules.
  --add-extra-py-files TEXT       Track local relative Python files in
                                  SourceControlDetails.ExtraPyFiles and --extra-
                                  py-files.
  --remove-extra-py-files TEXT    Remove local Python files from
                                  SourceControlDetails.ExtraPyFiles and --extra-
                                  py-files.
  --add-extra-jars TEXT           Add one or more values to --extra-jars.
  --remove-extra-jars TEXT        Remove one or more values from --extra-jars.
  --add-additional-python-files TEXT
                                  Track local paths in
                                  SourceControlDetails.AdditionalPythonFiles and
                                  --extra-py-files.
  --remove-additional-python-files TEXT
                                  Remove local paths from
                                  SourceControlDetails.AdditionalPythonFiles and
                                  --extra-py-files.
  --add-extra-files TEXT          Track local paths in
                                  SourceControlDetails.ExtraFiles and --extra-
                                  files.
  --remove-extra-files TEXT       Remove local paths from
                                  SourceControlDetails.ExtraFiles and --extra-
                                  files.
  --add-default-argument TEXT     Set a DefaultArguments entry using KEY=VALUE.
                                  Repeat as needed.
  --remove-default-argument TEXT  Remove a DefaultArguments entry by key. Repeat
                                  as needed.
  --add-non-overridable-argument TEXT
                                  Set a NonOverridableArguments entry using
                                  KEY=VALUE. Repeat as needed.
  --remove-non-overridable-argument TEXT
                                  Remove a NonOverridableArguments entry by key.
                                  Repeat as needed.
  --enable-metrics / --disable-metrics
                                  Set DefaultArguments.--enable-metrics to true
                                  or false.
  --enable-observability-metrics / --disable-observability-metrics
                                  Set DefaultArguments.--enable-observability-
                                  metrics to true or false.
  --enable-glue-datacatalog / --disable-glue-datacatalog
                                  Set DefaultArguments.--enable-glue-datacatalog
                                  to true or false.
  --enable-continuous-cloudwatch-log / --disable-continuous-cloudwatch-log
                                  Set DefaultArguments.--enable-continuous-
                                  cloudwatch-log to true or false.
  --enable-spark-ui / --disable-spark-ui
                                  Set DefaultArguments.--enable-spark-ui to true
                                  or false.
  --enable-job-insights / --disable-job-insights
                                  Set DefaultArguments.--enable-job-insights to
                                  true or false.
  --enable-auto-scaling / --disable-auto-scaling
                                  Set DefaultArguments.--enable-auto-scaling to
                                  true or false.
  --job-bookmark-option TEXT      Set DefaultArguments.--job-bookmark-option.
  --temp-dir TEXT                 Set DefaultArguments.--TempDir.
  --spark-event-logs-path TEXT    Set DefaultArguments.--spark-event-logs-path.
  --datalake-formats TEXT         Set DefaultArguments.--datalake-formats.
  --spark-conf TEXT               Set DefaultArguments.--conf.
  --continuous-log-log-group TEXT
                                  Set DefaultArguments.--continuous-log-
                                  logGroup.
  --continuous-log-log-stream-prefix TEXT
                                  Set DefaultArguments.--continuous-log-
                                  logStreamPrefix.
  --continuous-log-conversion-pattern TEXT
                                  Set DefaultArguments.--continuous-log-
                                  conversionPattern.
  --executor-cores INTEGER        Set DefaultArguments.--executor-cores.
  --customer-driver-env-vars TEXT
                                  Set DefaultArguments.--customer-driver-env-
                                  vars.
  --customer-executor-env-vars TEXT
                                  Set DefaultArguments.--customer-executor-env-
                                  vars.
  --python-modules-installer-option TEXT
                                  Set DefaultArguments.--python-modules-
                                  installer-option.
  --dry-run                       Show what would be updated without writing
                                  files.
  --config-dir PATH               Directory containing Glue job config files.
  --help                          Show this message and exit.

  **Examples**

  - `gluekit params set --temp-dir s3://bucket/tmp --enable-metrics`

  - `gluekit params set my-job --add-default-argument survey_id=test`

gluekit params save --help

Usage: gluekit params save [OPTIONS] KEY VALUE

  Save a reusable value for the active job, profile, or all jobs.

Arguments:
  KEY    Saved parameter name.  [required]
  VALUE  Saved parameter value.  [required]

Options:
  --job TEXT          Override the active job.
  --global            Save params as global defaults for all jobs.
  -p, --profile TEXT  Save params under this gluekit profile scope, usually
                      matching an AWS CLI profile name; does not contact AWS.
  --help              Show this message and exit.

  **Examples**

  - `gluekit params save default_arguments.--TempDir s3://my-bucket/tmp/`

  - `gluekit params save additional_python_modules "pydantic==2.11.7" --global`

gluekit params saved --help

Usage: gluekit params saved [OPTIONS] [JOB_NAME]

  Show stored local Glue config params.

Arguments:
  [JOB_NAME]  Optional Glue job name to view merged params (global + job).

Options:
  -p, --profile TEXT  Include params from this gluekit profile scope, usually
                      matching an AWS CLI profile name; does not contact AWS.
  --help              Show this message and exit.

  **Examples**

  - `gluekit params saved my-job`

  - `gluekit params saved`

gluekit script --help

Usage: gluekit script [OPTIONS] COMMAND [ARGS]...

  Work with the active job's script.

Options:
  --help  Show this message and exit.

Commands:
  convert          Convert the script to a notebook.
  sync             Refresh generated script metadata from the local job...
  import-settings  Import represented script metadata into the local job...
  push             Upload the local script to AWS.
  pull             Download the remote script from AWS.

gluekit script convert --help

Usage: gluekit script convert [OPTIONS] [JOB_NAME]

  Convert the script to a notebook.

Arguments:
  [JOB_NAME]

Options:
  --use-config / --artifact-only  [default: use-config]
  --dry-run
  --config-dir PATH
  --help                          Show this message and exit.

gluekit script sync --help

Usage: gluekit script sync [OPTIONS] [JOB_NAME]

  Refresh generated script metadata from the local job config.

Arguments:
  [JOB_NAME]

Options:
  --dry-run
  --config-dir PATH
  --help             Show this message and exit.

gluekit script import-settings --help

Usage: gluekit script import-settings [OPTIONS] [JOB_NAME]

  Import represented script metadata into the local job config.

Arguments:
  [JOB_NAME]

Options:
  --dry-run
  --config-dir PATH
  --help             Show this message and exit.

gluekit script push --help

Usage: gluekit script push [OPTIONS] [JOB_NAME]

  Upload the local script to AWS.

Arguments:
  [JOB_NAME]

Options:
  --to TEXT                       [default: aws]
  --dry-run
  --config-dir PATH
  -p, --profile TEXT
  --auto-login / --no-auto-login  [default: auto-login]
  --help                          Show this message and exit.

gluekit script pull --help

Usage: gluekit script pull [OPTIONS] [JOB_NAME]

  Download the remote script from AWS.

Arguments:
  [JOB_NAME]

Options:
  --from TEXT                     [default: aws]
  --dry-run
  --config-dir PATH
  -p, --profile TEXT
  --auto-login / --no-auto-login  [default: auto-login]
  --help                          Show this message and exit.

gluekit notebook --help

Usage: gluekit notebook [OPTIONS] COMMAND [ARGS]...

  Work with the active job's notebook.

Options:
  --help  Show this message and exit.

Commands:
  convert          Convert the notebook to a script.
  sync             Refresh generated notebook metadata from the local job...
  import-settings  Import represented notebook metadata into the local job...
  push             Upload the local notebook to AWS.
  pull             Download the remote notebook from AWS.

gluekit notebook convert --help

Usage: gluekit notebook convert [OPTIONS] [JOB_NAME]

  Convert the notebook to a script.

Arguments:
  [JOB_NAME]

Options:
  --use-config / --artifact-only  [default: use-config]
  --dry-run
  --config-dir PATH
  --help                          Show this message and exit.

gluekit notebook sync --help

Usage: gluekit notebook sync [OPTIONS] [JOB_NAME]

  Refresh generated notebook metadata from the local job config.

Arguments:
  [JOB_NAME]

Options:
  --dry-run
  --config-dir PATH
  --help             Show this message and exit.

gluekit notebook import-settings --help

Usage: gluekit notebook import-settings [OPTIONS] [JOB_NAME]

  Import represented notebook metadata into the local job config.

Arguments:
  [JOB_NAME]

Options:
  --dry-run
  --config-dir PATH
  --help             Show this message and exit.

gluekit notebook push --help

Usage: gluekit notebook push [OPTIONS] [JOB_NAME]

  Upload the local notebook to AWS.

Arguments:
  [JOB_NAME]

Options:
  --to TEXT                       [default: aws]
  --dry-run
  --config-dir PATH
  -p, --profile TEXT
  --auto-login / --no-auto-login  [default: auto-login]
  --help                          Show this message and exit.

gluekit notebook pull --help

Usage: gluekit notebook pull [OPTIONS] [JOB_NAME]

  Download the remote notebook from AWS.

Arguments:
  [JOB_NAME]

Options:
  --from TEXT                     [default: aws]
  --dry-run
  --config-dir PATH
  -p, --profile TEXT
  --auto-login / --no-auto-login  [default: auto-login]
  --help                          Show this message and exit.

gluekit push --help

Usage: gluekit push [OPTIONS] [JOB_NAME] [TARGET_JOB]

  Transfer the full active job between local files and AWS.

Arguments:
  [JOB_NAME]    Glue job name or pattern to push. Use "*" for all configs.
  [TARGET_JOB]  Optional remote Glue job name to push to.

Options:
  --dry-run                       Show what would be uploaded without writing
                                  files.
  --update-config / --no-update-config
                                  Update Glue job configuration after uploading
                                  files.  [default: update-config]
  --build                         Build local package artifacts in the workspace
                                  root before push.
  --build-tool TEXT               Build frontend for --build: auto, uv, or
                                  build.  [default: auto]
  -v, --verbose                   Print local build command(s) before execution.
  --config-dir PATH               Directory containing Glue job config files.
  -p, --profile TEXT              AWS CLI credential profile used for real AWS
                                  Glue/S3 API calls; not a gluekit local test
                                  profile.
  --auto-login / --no-auto-login  For real AWS profiles, automatically run 'aws
                                  sso login' when credentials are missing or
                                  expired.  [default: auto-login]
  --to TEXT                       Transfer endpoint (aws).  [default: aws]
  --help                          Show this message and exit.

  **Examples**

  - `gluekit push my-job --profile my-sso-profile`

  - `gluekit push source-job target-job --profile my-sso-profile`

  - `gluekit push "my-job-\*"`

  - `gluekit push my-job --build --build-tool auto --dry-run`

gluekit pull --help

Usage: gluekit pull [OPTIONS] [JOB_NAME] [TARGET_JOB]

  Transfer the full active job between local files and AWS.

Arguments:
  [JOB_NAME]    Glue job name or pattern to pull. Use "*" for all jobs.
  [TARGET_JOB]  Optional local job name to write pulled artifacts to.

Options:
  -f, --force                     Download even if the local config is newer or
                                  equal to LastModifiedOn.
  --dry-run                       Show what would be downloaded without writing
                                  files.
  --config-dir PATH               Directory containing Glue job config files.
  --write-job-list / --no-write-job-list
                                  Write glue/glue_full_job_list.csv during the
                                  run.  [default: write-job-list]
  -p, --profile TEXT              AWS CLI credential profile used for real AWS
                                  Glue/S3 API calls; not a gluekit local test
                                  profile.
  --auto-login / --no-auto-login  For real AWS profiles, automatically run 'aws
                                  sso login' when credentials are missing or
                                  expired.  [default: auto-login]
  --from TEXT                     Transfer endpoint (aws).  [default: aws]
  --help                          Show this message and exit.

  **Examples**

  - `gluekit pull my-job --profile my-sso-profile`

  - `gluekit pull source-job target-job --profile my-sso-profile`

  - `gluekit pull "my-job-\*" --force`

  - `gluekit pull "\*" --dry-run --profile my-sso-profile --no-auto-login`

gluekit init --help

Usage: gluekit init [OPTIONS]

  Initialize local repo-level Gluekit settings from AWS CLI profiles.

Options:
  --aws-config PATH               Local AWS CLI config file to scan for
                                  credential profile names; defaults to
                                  AWS_CONFIG_FILE or ~/.aws/config.
  --config-dir PATH               Directory containing Glue job config files.
                                  [default: glue/configs]
  --auto-login / --no-auto-login  For selected real AWS profiles, automatically
                                  run 'aws sso login' when credentials are
                                  missing or expired.  [default: auto-login]
  --detect-roles / --no-detect-roles
                                  Fetch the selected real AWS Glue job in each
                                  profile and save its Role as local gluekit
                                  params.  [default: detect-roles]
  --detect-buckets / --no-detect-buckets
                                  Compare S3 URIs in real AWS Glue job configs
                                  across profiles and save local gluekit profile
                                  mappings.  [default: detect-buckets]
  --match-threshold INTEGER       Minimum rapidfuzz score for non-exact S3 key
                                  matches.  [default: 85]
  --baseline-profile TEXT         Selected AWS CLI profile to store as the first
                                  gluekit activate scope and S3 comparison
                                  baseline.
  --profile-job TEXT              Real AWS Glue job to inspect for an AWS CLI
                                  profile, as PROFILE=GLUE_JOB_NAME. May be
                                  repeated.
  --help                          Show this message and exit.

  **Examples**

  - `gluekit init`

gluekit build --help

Usage: gluekit build [OPTIONS]

  Build local artifacts; only touches AWS when --push is used.

Options:
  --job-name TEXT                 Active or explicit local Glue job to update
                                  when using --package-whl or --push.
  --dry-run                       Show what would be built and zipped without
                                  running build commands.
  --bump-version, --bump TEXT     Bump pyproject.toml [project].version before
                                  building: major, minor, patch, or X.Y.Z.
  --build-tool TEXT               Build frontend to use: auto, uv, or build.
                                  [default: auto]
  --out-dir PATH                  The output directory to which distributions
                                  should be written.
  --no-version-suffix             Remove the version suffix from generated zip
                                  artifact names.
  -v, --verbose                   Print build commands before execution.
  --package-whl                   After building, replace the tracked dist/*.whl
                                  in the selected Glue config.
  --push                          Push the selected Glue config and artifacts
                                  after building.
  --update-config / --no-update-config
                                  For --push, update Glue job configuration
                                  after uploading files.  [default: update-
                                  config]
  --config-dir PATH               Directory containing Glue job config files.
  -p, --profile TEXT              AWS CLI credential profile for profile-scoped
                                  config params and, with --push, real AWS
                                  Glue/S3 API calls.
  --auto-login / --no-auto-login  For --push with a real AWS profile,
                                  automatically run 'aws sso login' when
                                  credentials are missing or expired.  [default:
                                  auto-login]
  --help                          Show this message and exit.

  **Examples**

  - `gluekit build`

  - `gluekit build --build-tool uv`

  - `gluekit build --out-dir build/artifacts --no-version-suffix`

  - `gluekit build --build-tool build --dry-run`

gluekit local --help

Usage: gluekit local [OPTIONS] COMMAND [ARGS]...

  Local Glue fixtures and setup.

Options:
  --help  Show this message and exit.

Commands:
  setup   Create or update the local-only development setup.
  status  Show the local development fixture configuration.
  s3      Manage local mocked S3 fixture mappings; does not call AWS S3.
  ssm     Manage local mocked SSM Parameter Store values; does not call AWS...

gluekit run --help

Usage: gluekit run [OPTIONS] SCRIPT_PATH

  Run a Glue script locally with emulated Glue libraries and mocked AWS
  services.

Arguments:
  SCRIPT_PATH  Path to the Glue Python script to execute locally.  [required]

Options:
  --glue-arg TEXT       Glue script argument in KEY=VALUE form; passed through
                        as --KEY VALUE.
  --job-name TEXT       Default JOB_NAME used when not provided in script args.
                        [default: local-glue-job]
  --create-bucket TEXT  Create a mocked S3 bucket only inside the local run.
                        Repeat as needed.
  --ssm-parameter TEXT  Seed a mocked SSM parameter only inside the local run as
                        NAME=VALUE. Repeat as needed.
  --aws-region TEXT     Mock AWS region used for local boto3 clients. Defaults
                        to us-east-1.
  --config-file FILE    Local fixture config file for mocked S3/SSM. Defaults to
                        .gluekit/local.json when local settings are active.
  --runtime TEXT        Local DataFrame runtime: sqlframe (fast default) or
                        pyspark (real local Spark).  [default: sqlframe]
  --spark-master TEXT   Spark master URL used when --runtime=pyspark.  [default:
                        local[2]]
  --help                Show this message and exit.

  **Examples**

  - `gluekit run ../ppsc-survey/glue/scripts/job_a.py --create-bucket my-input-
  bucket`

  - `gluekit run glue/scripts/job-a.py --ssm-parameter /app/env=dev --glue-arg
  report_date=2026-06-18`

  - `gluekit run glue/scripts/job-a.py -- --JOB_NAME custom-local-job
  --report_date 2026-06-18`

Package releases

Install the repository task runner with uv tool install poethepoet==0.48.0, then run poe release-check for validation and poe build for a local wheel and source distribution. One-off usage is uvx --from poethepoet==0.48.0 poe release-check.

The publish-to-pypi.yml workflow has three jobs: prepare, publish, and release. Pushes to dev start or increment a patch development version (1.0.0 → 1.0.1.dev1 → 1.0.1.dev2). Pushes to main promote a prerelease to stable, or increment the patch when the source already has a stable version. After a stable tag exists, dev starts the next patch series. Normal releases should not edit the version manually. Merge updated main back into dev when needed to keep the branches' version baselines aligned.

poe release --branch dev --source <full-commit-sha> runs scripts/release.py in a clean checkout of that source. It uses uv version --no-sync to update pyproject.toml and uv.lock, invokes poe build, and creates a local release commit and annotated tag. This is the CI preparation command: it changes the local checkout but does not push or publish. The workflow first saves the built distributions, then atomically pushes the release commit and tag. Both branches share one release concurrency group; a stale run fails rather than overwriting newer work. Rapid pushes may supersede pending runs; the latest source should be released. Workflow pushes use GITHUB_TOKEN and do not recursively trigger another push workflow.

PyPI publication uses the pypi GitHub environment and Trusted Publishing bound to this repository and publish-to-pypi.yml. No long-lived PyPI token is needed. Only stable main versions get a GitHub Release, after PyPI succeeds. The branch rules must permit the workflow's version commit; rejected pushes leave both remote refs unchanged.

For a failed publish, use Re-run failed jobs on the original Actions run. The publish job downloads the saved wheel and source distribution without rebuilding. Identical PyPI uploads can be retried, including a partially completed upload. A full rerun recognizes the tagged source/branch before calculating a version and reuses its original artifact. Existing GitHub Releases are left intact. If preparation failed before the atomic push, a fresh attempt can rebuild and replace that run's unpublished artifact. If the original artifact has expired or been deleted after the push, stop and recover those exact files; the workflow deliberately does not rebuild a published version.

poe release-test exercises the release helper with temporary local Git remotes and real uv version changes; distribution builds are mocked. No test publishes packages or contacts cloud providers.

Documentation development

Install Poe with uv tool install poethepoet==0.48.0, or use uvx --from poethepoet==0.48.0 poe <task> for one-off commands.

• poe docs-update: regenerate committed documentation; review the diff and commit it.
• poe docs-check: detect stale generated documentation without changing it.
• poe docs-build: build the site with MkDocs strict mode.
• poe docs-validate: run both documentation checks and the strict build.
• poe docs: regenerate documentation and build the site for local development.

Documentation tasks use dependencies from uv.lock with --frozen; update the lockfile intentionally when changing dependencies. CI calls the same Poe tasks, never regenerates or auto-commits documentation, and validates docs through poe release-check for both pull requests and package releases. Documentation hosting remains a separate deployment concern.