Gluekit

Gluekit works with one active AWS Glue job in a local workspace. Activate a job, edit its local definition or arguments, convert its script and notebook, then push or pull files through AWS Glue and S3.

gluekit activate my-job --profile my-aws-profile
gluekit status
gluekit settings show
gluekit settings set --worker-type G.1X --number-of-workers 5
gluekit params set --temp-dir s3://my-bucket/tmp/
gluekit script convert
gluekit notebook convert
gluekit push --to aws --dry-run
gluekit push --to aws

Install with uv tool install . or run from this repository with uv run gluekit.

Commands

Command	Purpose
activate <job>	Select one job and optionally an AWS profile for later commands. activate --clear clears the selection.
settings show/set	Inspect or change local Glue job fields such as Role, WorkerType, and Timeout.
params show/set	Inspect or change DefaultArguments and NonOverridableArguments.
params save <key> <value>	Save a reusable value for the active job; add --profile or --global for that scope. params saved [job] inspects these values.
script convert / notebook convert	Convert to the other representation.
script sync / notebook sync	Refresh generated metadata from the local job definition.
script import-settings / notebook import-settings	Import represented metadata into the local job definition.
push --to aws / pull --from aws	Transfer the full selected job. Pass a job name explicitly to override the selection; pass '*' explicitly for all jobs.
script push/pull / notebook push/pull	Transfer only that artifact.
init, build, status	Set up profile values, build local dependencies, and inspect the active job.
local, run	Set up local fixtures and run Glue scripts locally.

The default transfer endpoint is AWS. The --to and --from flags accept aws; a GitHub transfer backend is planned separately. Transfer commands use local files as the other endpoint, and do not need --include or --exclude. To inspect the available options, use gluekit <command> --help.

Script and notebook conversion

uvx gluekit notebook convert my-job  # notebook → clean Python script
uvx gluekit script convert my-job    # script → notebook with Glue magics

Omit my-job to use the active job. --artifact-only skips the local job configuration and generated magics; --dry-run writes neither artifacts nor state.

Notebook exports contain Python and commented markdown, with no percent cell markers, generated Glue configuration header, or commented pyproject/PEP 723 metadata. Existing percent markers are removed as well. Python indentation and user comments are retained; Gluekit does not reformat executable code. Non-Python magics such as SQL remain as commented documentation, not executable Python.

Cell boundaries live separately under notebook_cells["my-job"] in .gluekit/glue_set.json. The markers object maps 1-based script line numbers to the marker inserted before that line when reconstructing notebook cells:

{"markers": {"1": "#%%", "9": "#%%", "15": "#%%[markdown]"}}

Each entry also records script/notebook paths and line hashes. At conversion time, Python's standard-library difflib relocates boundaries after insertions, replacements, and deletions; ast moves unsafe boundaries to the start of the containing top-level statement. Functions, decorators, classes, loops, with, try/except/finally, multiline expressions, and match blocks stay intact. Markdown is restored from its script comments without accumulating comment prefixes. The map updates on conversion, not continuously while editing; no watcher or extra dependency is needed. Line-diff matching is best-effort for moved or repeated code, but AST validation still prevents a boundary from splitting a statement.

For a script without saved boundaries or explicit percent markers, Gluekit groups global setup and starts cells around top-level definitions and compound statements. Nested blocks remain in their containing cell. Existing explicit markers take precedence over saved boundaries and are also checked for scope safety. Invalid Python causes conversion to fail before writing the notebook or its saved map.

Script-to-notebook conversion generates the existing Glue session magics from the local job configuration, but omits %iam_role. It replaces script-only job.commit() statements with no-ops so otherwise empty suites remain valid. notebook sync also omits the role, and importing a notebook without a role leaves the configured job role intact. Explicitly authored role magics can still be imported. See the AWS interactive-session magic reference.

script sync remains an explicit opt-in to the older commented Glue configuration representation; a clean notebook export does not contain settings for script import-settings. Use notebook import-settings or edit the job configuration instead. Saved cell metadata is local workspace state, never an AWS job parameter.

State migration

Existing .gluekit/glue_set.json files with a checkout entry are read as the active selection. The next state write stores it under active. The old checkout command and other deprecated aliases are no longer registered. The existing local and run behavior remains available.

CLI reference

Auto-generated from live CLI --help output.

gluekit --help

Usage: gluekit [OPTIONS] COMMAND [ARGS]...

Options:
  --install-completion  Install completion for the current shell.
  --show-completion     Show completion for the current shell, to copy it or
                        customize the installation.
  --help                Show this message and exit.

Commands:
  activate  Replace the active job.
  build     Build local artifacts; only touches AWS when --push is used.
  init      Initialize local repo-level Gluekit settings from AWS CLI...
  pull      Transfer the full active job between local files and AWS.
  push      Transfer the full active job between local files and AWS.
  run       Run a Glue script locally with emulated Glue libraries and...
  status    Show the active job and its local-to-S3 artifact mappings.
  settings  Edit the active Glue job definition.
  params    Edit the active job's Glue arguments.
  script    Work with the active job's script.
  notebook  Work with the active job's notebook.
  local     Local Glue fixtures and setup.

gluekit activate --help

Usage: gluekit activate [OPTIONS] [job_name]

  Replace the active job.

Arguments:
  job_name  Glue job name to activate for later commands.

Options:
  --config-dir <path>  Directory containing Glue job config files.  [default:
                       glue/configs]
  -p, --profile <str>  AWS CLI credential profile name to store as the active
                       gluekit profile scope for saved params.
  --dry-run            Show gluekit profile-scoped config updates without
                       writing files.
  --clear              Remove the active job selection without deleting saved
                       params or local files.
  --local              Check out the single local development settings file in
                       .gluekit/local.json.
  --help               Show this message and exit.

  **Examples**

  - `gluekit activate my-job`

  - `gluekit activate --local`

  - `gluekit activate --clear`

gluekit status --help

Usage: gluekit status [OPTIONS]

  Show the active job and its local-to-S3 artifact mappings.

Options:
  --config-dir <path>  Directory containing Glue job config files.  [default:
                       glue/configs]
  --help               Show this message and exit.

  **Examples**

  - `gluekit status`

  - `gluekit status --config-dir glue/configs`

gluekit settings --help

Usage: gluekit settings [OPTIONS] COMMAND [ARGS]...

  Edit the active Glue job definition.

Options:
  --help  Show this message and exit.

Commands:
  show  Show the active job's local Glue definition.
  set   Manage local AWS Glue job-definition properties.

gluekit settings show --help

Usage: gluekit settings show [OPTIONS] [job_name]

  Show the active job's local Glue definition.

Arguments:
  job_name

Options:
  --config-dir <path>  [default: glue/configs]
  --help               Show this message and exit.

gluekit settings set --help

Usage: gluekit settings set [OPTIONS] [job_name]

  Manage local AWS Glue job-definition properties.

Arguments:
  job_name  Glue job name. Defaults to the active job selection.

Options:
  --description <str>             Set the job description.
  --role <str>                    Set the IAM role for the job.
  --glue-version <str>            Set the Glue version.
  --worker-type <str>             Set the worker type.
  --number-of-workers <int>       Set the number of workers.
  --timeout <int>                 Set the job timeout in minutes.
  --max-retries <int>             Set the max retry count.
  --execution-class <str>         Set the execution class.
  --max-concurrent-runs <int>     Set the max concurrent runs.
  --job-run-queuing-enabled / --no-job-run-queuing-enabled
                                  Enable or disable job run queuing.
  --security-configuration <str>  Set the security configuration name.
  --maintenance-window <str>      Set the maintenance window.
  --log-uri <str>                 Set the log URI.
  --command-name <str>            Set Command.Name.
  --python-version <str>          Set Command.PythonVersion.
  --runtime <str>                 Set Command.Runtime.
  --remote-script-location <str>  Set Command.ScriptLocation.
  --local-script-location <str>   Set SourceControlDetails.ScriptLocation.
  --notebook-location <str>       Set SourceControlDetails.NotebookLocation and
                                  NotebookPath.
  --source-control-provider <str>
                                  Set SourceControlDetails.Provider.
  --source-control-owner <str>    Set SourceControlDetails.Owner.
  --source-control-repository <str>
                                  Set SourceControlDetails.Repository.
  --source-control-branch <str>   Set SourceControlDetails.Branch.
  --source-control-folder <str>   Set SourceControlDetails.Folder.
  --add-connection <str>          Add one or more Glue connections.
  --remove-connection <str>       Remove one or more Glue connections.
  --dry-run                       Show what would be updated without writing
                                  files.
  --config-dir <path>             Directory containing Glue job config files.
                                  [default: glue/configs]
  --help                          Show this message and exit.

  **Examples**

  - `gluekit settings set --description 'Updated job description' --glue-version
  5.0`

  - `gluekit settings set my-job --worker-type G.1X --number-of-workers 5`

gluekit params --help

Usage: gluekit params [OPTIONS] COMMAND [ARGS]...

  Edit the active job's Glue arguments.

Options:
  --help  Show this message and exit.

Commands:
  show   Show the active job's local Glue arguments.
  set    Manage local AWS Glue runtime and job arguments.
  save   Save a reusable value for the active job, profile, or all jobs.
  saved  Show stored local Glue config params.

gluekit params show --help

Usage: gluekit params show [OPTIONS] [job_name]

  Show the active job's local Glue arguments.

Arguments:
  job_name

Options:
  --config-dir <path>  [default: glue/configs]
  --help               Show this message and exit.

gluekit params set --help

Usage: gluekit params set [OPTIONS] [job_name]

  Manage local AWS Glue runtime and job arguments.

Arguments:
  job_name  Glue job name. Defaults to the active job selection.

Options:
  --add-additional-python-modules <str>
                                  Add one or more values to --additional-python-
                                  modules.
  --remove-additional-python-modules <str>
                                  Remove one or more values from --additional-
                                  python-modules.
  --add-extra-py-files <str>      Track local relative Python files in
                                  SourceControlDetails.ExtraPyFiles and --extra-
                                  py-files.
  --remove-extra-py-files <str>   Remove local Python files from
                                  SourceControlDetails.ExtraPyFiles and --extra-
                                  py-files.
  --add-extra-jars <str>          Add one or more values to --extra-jars.
  --remove-extra-jars <str>       Remove one or more values from --extra-jars.
  --add-additional-python-files <str>
                                  Track local paths in
                                  SourceControlDetails.AdditionalPythonFiles and
                                  --extra-py-files.
  --remove-additional-python-files <str>
                                  Remove local paths from
                                  SourceControlDetails.AdditionalPythonFiles and
                                  --extra-py-files.
  --add-extra-files <str>         Track local paths in
                                  SourceControlDetails.ExtraFiles and --extra-
                                  files.
  --remove-extra-files <str>      Remove local paths from
                                  SourceControlDetails.ExtraFiles and --extra-
                                  files.
  --add-default-argument <str>    Set a DefaultArguments entry using KEY=VALUE.
                                  Repeat as needed.
  --remove-default-argument <str>
                                  Remove a DefaultArguments entry by key. Repeat
                                  as needed.
  --add-non-overridable-argument <str>
                                  Set a NonOverridableArguments entry using
                                  KEY=VALUE. Repeat as needed.
  --remove-non-overridable-argument <str>
                                  Remove a NonOverridableArguments entry by key.
                                  Repeat as needed.
  --enable-metrics / --disable-metrics
                                  Set DefaultArguments.--enable-metrics to true
                                  or false.
  --enable-observability-metrics / --disable-observability-metrics
                                  Set DefaultArguments.--enable-observability-
                                  metrics to true or false.
  --enable-glue-datacatalog / --disable-glue-datacatalog
                                  Set DefaultArguments.--enable-glue-datacatalog
                                  to true or false.
  --enable-continuous-cloudwatch-log / --disable-continuous-cloudwatch-log
                                  Set DefaultArguments.--enable-continuous-
                                  cloudwatch-log to true or false.
  --enable-spark-ui / --disable-spark-ui
                                  Set DefaultArguments.--enable-spark-ui to true
                                  or false.
  --enable-job-insights / --disable-job-insights
                                  Set DefaultArguments.--enable-job-insights to
                                  true or false.
  --enable-auto-scaling / --disable-auto-scaling
                                  Set DefaultArguments.--enable-auto-scaling to
                                  true or false.
  --job-bookmark-option <str>     Set DefaultArguments.--job-bookmark-option.
  --temp-dir <str>                Set DefaultArguments.--TempDir.
  --spark-event-logs-path <str>   Set DefaultArguments.--spark-event-logs-path.
  --datalake-formats <str>        Set DefaultArguments.--datalake-formats.
  --spark-conf <str>              Set DefaultArguments.--conf.
  --continuous-log-log-group <str>
                                  Set DefaultArguments.--continuous-log-
                                  logGroup.
  --continuous-log-log-stream-prefix <str>
                                  Set DefaultArguments.--continuous-log-
                                  logStreamPrefix.
  --continuous-log-conversion-pattern <str>
                                  Set DefaultArguments.--continuous-log-
                                  conversionPattern.
  --executor-cores <int>          Set DefaultArguments.--executor-cores.
  --customer-driver-env-vars <str>
                                  Set DefaultArguments.--customer-driver-env-
                                  vars.
  --customer-executor-env-vars <str>
                                  Set DefaultArguments.--customer-executor-env-
                                  vars.
  --python-modules-installer-option <str>
                                  Set DefaultArguments.--python-modules-
                                  installer-option.
  --dry-run                       Show what would be updated without writing
                                  files.
  --config-dir <path>             Directory containing Glue job config files.
                                  [default: glue/configs]
  --help                          Show this message and exit.

  **Examples**

  - `gluekit params set --temp-dir s3://bucket/tmp --enable-metrics`

  - `gluekit params set my-job --add-default-argument survey_id=test`

gluekit params save --help

Usage: gluekit params save [OPTIONS] {key} {value}

  Save a reusable value for the active job, profile, or all jobs.

Arguments:
  key    Saved parameter name.  [required]
  value  Saved parameter value.  [required]

Options:
  --job <str>          Override the active job.
  --global             Save params as global defaults for all jobs.
  -p, --profile <str>  Save params under this gluekit profile scope, usually
                       matching an AWS CLI profile name; does not contact AWS.
  --help               Show this message and exit.

  **Examples**

  - `gluekit params save default_arguments.--TempDir s3://my-bucket/tmp/`

  - `gluekit params save additional_python_modules "pydantic==2.11.7" --global`

gluekit params saved --help

Usage: gluekit params saved [OPTIONS] [job_name]

  Show stored local Glue config params.

Arguments:
  job_name  Optional Glue job name to view merged params (global + job).

Options:
  -p, --profile <str>  Include params from this gluekit profile scope, usually
                       matching an AWS CLI profile name; does not contact AWS.
  --help               Show this message and exit.

  **Examples**

  - `gluekit params saved my-job`

  - `gluekit params saved`

gluekit script --help

Usage: gluekit script [OPTIONS] COMMAND [ARGS]...

  Work with the active job's script.

Options:
  --help  Show this message and exit.

Commands:
  convert          Convert the script to a notebook.
  sync             Refresh generated script metadata from the local job...
  import-settings  Import represented script metadata into the local job...
  push             Upload the local script to AWS.
  pull             Download the remote script from AWS.

gluekit script convert --help

Usage: gluekit script convert [OPTIONS] [job_name]

  Convert the script to a notebook.

Arguments:
  job_name

Options:
  --use-config / --artifact-only  [default: use-config]
  --dry-run
  --config-dir <path>             [default: glue/configs]
  --help                          Show this message and exit.

gluekit script sync --help

Usage: gluekit script sync [OPTIONS] [job_name]

  Refresh generated script metadata from the local job config.

Arguments:
  job_name

Options:
  --dry-run
  --config-dir <path>  [default: glue/configs]
  --help               Show this message and exit.

gluekit script import-settings --help

Usage: gluekit script import-settings [OPTIONS] [job_name]

  Import represented script metadata into the local job config.

Arguments:
  job_name

Options:
  --dry-run
  --config-dir <path>  [default: glue/configs]
  --help               Show this message and exit.

gluekit script push --help

Usage: gluekit script push [OPTIONS] [job_name]

  Upload the local script to AWS.

Arguments:
  job_name

Options:
  --to <str>                      [default: aws]
  --dry-run
  --config-dir <path>             [default: glue/configs]
  -p, --profile <str>
  --auto-login / --no-auto-login  [default: auto-login]
  --help                          Show this message and exit.

gluekit script pull --help

Usage: gluekit script pull [OPTIONS] [job_name]

  Download the remote script from AWS.

Arguments:
  job_name

Options:
  --from <str>                    [default: aws]
  --dry-run
  --config-dir <path>             [default: glue/configs]
  -p, --profile <str>
  --auto-login / --no-auto-login  [default: auto-login]
  --help                          Show this message and exit.

gluekit notebook --help

Usage: gluekit notebook [OPTIONS] COMMAND [ARGS]...

  Work with the active job's notebook.

Options:
  --help  Show this message and exit.

Commands:
  convert          Convert the notebook to a script.
  sync             Refresh generated notebook metadata from the local job...
  import-settings  Import represented notebook metadata into the local job...
  push             Upload the local notebook to AWS.
  pull             Download the remote notebook from AWS.

gluekit notebook convert --help

Usage: gluekit notebook convert [OPTIONS] [job_name]

  Convert the notebook to a script.

Arguments:
  job_name

Options:
  --use-config / --artifact-only  [default: use-config]
  --dry-run
  --config-dir <path>             [default: glue/configs]
  --help                          Show this message and exit.

gluekit notebook sync --help

Usage: gluekit notebook sync [OPTIONS] [job_name]

  Refresh generated notebook metadata from the local job config.

Arguments:
  job_name

Options:
  --dry-run
  --config-dir <path>  [default: glue/configs]
  --help               Show this message and exit.

gluekit notebook import-settings --help

Usage: gluekit notebook import-settings [OPTIONS] [job_name]

  Import represented notebook metadata into the local job config.

Arguments:
  job_name

Options:
  --dry-run
  --config-dir <path>  [default: glue/configs]
  --help               Show this message and exit.

gluekit notebook push --help

Usage: gluekit notebook push [OPTIONS] [job_name]

  Upload the local notebook to AWS.

Arguments:
  job_name

Options:
  --to <str>                      [default: aws]
  --dry-run
  --config-dir <path>             [default: glue/configs]
  -p, --profile <str>
  --auto-login / --no-auto-login  [default: auto-login]
  --help                          Show this message and exit.

gluekit notebook pull --help

Usage: gluekit notebook pull [OPTIONS] [job_name]

  Download the remote notebook from AWS.

Arguments:
  job_name

Options:
  --from <str>                    [default: aws]
  --dry-run
  --config-dir <path>             [default: glue/configs]
  -p, --profile <str>
  --auto-login / --no-auto-login  [default: auto-login]
  --help                          Show this message and exit.

gluekit push --help

Usage: gluekit push [OPTIONS] [job_name] [target_job]

  Transfer the full active job between local files and AWS.

Arguments:
  job_name    Glue job name or pattern to push. Use "*" for all configs.
  target_job  Optional remote Glue job name to push to.

Options:
  --dry-run                       Show what would be uploaded without writing
                                  files.
  --update-config / --no-update-config
                                  Update Glue job configuration after uploading
                                  files.  [default: update-config]
  --build                         Build local package artifacts in the workspace
                                  root before push.
  --build-tool <str>              Build frontend for --build: auto, uv, or
                                  build.  [default: auto]
  -v, --verbose                   Print local build command(s) before execution.
  --config-dir <path>             Directory containing Glue job config files.
                                  [default: glue/configs]
  -p, --profile <str>             AWS CLI credential profile used for real AWS
                                  Glue/S3 API calls; not a gluekit local test
                                  profile.
  --auto-login / --no-auto-login  For real AWS profiles, automatically run 'aws
                                  sso login' when credentials are missing or
                                  expired.  [default: auto-login]
  --to <str>                      Transfer endpoint (aws).  [default: aws]
  --help                          Show this message and exit.

  **Examples**

  - `gluekit push my-job --profile my-sso-profile`

  - `gluekit push source-job target-job --profile my-sso-profile`

  - `gluekit push "my-job-\*"`

  - `gluekit push my-job --build --build-tool auto --dry-run`

gluekit pull --help

Usage: gluekit pull [OPTIONS] [job_name] [target_job]

  Transfer the full active job between local files and AWS.

Arguments:
  job_name    Glue job name or pattern to pull. Use "*" for all jobs.
  target_job  Optional local job name to write pulled artifacts to.

Options:
  -f, --force                     Download even if the local config is newer or
                                  equal to LastModifiedOn.
  --dry-run                       Show what would be downloaded without writing
                                  files.
  --config-dir <path>             Directory containing Glue job config files.
                                  [default: glue/configs]
  --write-job-list / --no-write-job-list
                                  Write glue/glue_full_job_list.csv during the
                                  run.  [default: write-job-list]
  -p, --profile <str>             AWS CLI credential profile used for real AWS
                                  Glue/S3 API calls; not a gluekit local test
                                  profile.
  --auto-login / --no-auto-login  For real AWS profiles, automatically run 'aws
                                  sso login' when credentials are missing or
                                  expired.  [default: auto-login]
  --from <str>                    Transfer endpoint (aws).  [default: aws]
  --help                          Show this message and exit.

  **Examples**

  - `gluekit pull my-job --profile my-sso-profile`

  - `gluekit pull source-job target-job --profile my-sso-profile`

  - `gluekit pull "my-job-\*" --force`

  - `gluekit pull "\*" --dry-run --profile my-sso-profile --no-auto-login`

gluekit init --help

Usage: gluekit init [OPTIONS]

  Initialize local repo-level Gluekit settings from AWS CLI profiles.

Options:
  --aws-config <path>             Local AWS CLI config file to scan for
                                  credential profile names; defaults to
                                  AWS_CONFIG_FILE or ~/.aws/config.
  --config-dir <path>             Directory containing Glue job config files.
                                  [default: glue/configs]
  --auto-login / --no-auto-login  For selected real AWS profiles, automatically
                                  run 'aws sso login' when credentials are
                                  missing or expired.  [default: auto-login]
  --detect-roles / --no-detect-roles
                                  Fetch the selected real AWS Glue job in each
                                  profile and save its Role as local gluekit
                                  params.  [default: detect-roles]
  --detect-buckets / --no-detect-buckets
                                  Compare S3 URIs in real AWS Glue job configs
                                  across profiles and save local gluekit profile
                                  mappings.  [default: detect-buckets]
  --match-threshold <int>         Minimum rapidfuzz score for non-exact S3 key
                                  matches.  [default: 85]
  --baseline-profile <str>        Selected AWS CLI profile to store as the first
                                  gluekit activate scope and S3 comparison
                                  baseline.
  --profile-job <str>             Real AWS Glue job to inspect for an AWS CLI
                                  profile, as PROFILE=GLUE_JOB_NAME. May be
                                  repeated.
  --help                          Show this message and exit.

  **Examples**

  - `gluekit init`

gluekit build --help

Usage: gluekit build [OPTIONS]

  Build local artifacts; only touches AWS when --push is used.

Options:
  --job-name <str>                Active or explicit local Glue job to update
                                  when using --package-whl or --push.
  --dry-run                       Show what would be built and zipped without
                                  running build commands.
  --bump-version, --bump <str>    Bump pyproject.toml [project].version before
                                  building: major, minor, patch, or X.Y.Z.
  --build-tool <str>              Build frontend to use: auto, uv, or build.
                                  [default: auto]
  --out-dir <path>                The output directory to which distributions
                                  should be written.
  --no-version-suffix             Remove the version suffix from generated zip
                                  artifact names.
  -v, --verbose                   Print build commands before execution.
  --package-whl                   After building, replace the tracked dist/*.whl
                                  in the selected Glue config.
  --push                          Push the selected Glue config and artifacts
                                  after building.
  --update-config / --no-update-config
                                  For --push, update Glue job configuration
                                  after uploading files.  [default: update-
                                  config]
  --config-dir <path>             Directory containing Glue job config files.
                                  [default: glue/configs]
  -p, --profile <str>             AWS CLI credential profile for profile-scoped
                                  config params and, with --push, real AWS
                                  Glue/S3 API calls.
  --auto-login / --no-auto-login  For --push with a real AWS profile,
                                  automatically run 'aws sso login' when
                                  credentials are missing or expired.  [default:
                                  auto-login]
  --help                          Show this message and exit.

  **Examples**

  - `gluekit build`

  - `gluekit build --build-tool uv`

  - `gluekit build --out-dir build/artifacts --no-version-suffix`

  - `gluekit build --build-tool build --dry-run`

gluekit local --help

Usage: gluekit local [OPTIONS] COMMAND [ARGS]...

  Local Glue fixtures and setup.

Options:
  --help  Show this message and exit.

Commands:
  setup   Create or update and activate the local-only development setup.
  status  Show the active local development setup.
  s3      Manage local mocked S3 fixture mappings; does not call AWS S3.
  ssm     Manage local mocked SSM Parameter Store values; does not call AWS...

gluekit run --help

Usage: gluekit run [OPTIONS] {script_path}

  Run a Glue script locally with emulated Glue libraries and mocked AWS
  services.

Arguments:
  script_path  Path to the Glue Python script to execute locally.  [required]

Options:
  --glue-arg <str>       Glue script argument in KEY=VALUE form; passed through
                         as --KEY VALUE.
  --job-name <str>       Default JOB_NAME used when not provided in script args.
                         [default: local-glue-job]
  --create-bucket <str>  Create a mocked S3 bucket only inside the local run.
                         Repeat as needed.
  --ssm-parameter <str>  Seed a mocked SSM parameter only inside the local run
                         as NAME=VALUE. Repeat as needed.
  --aws-region <str>     Mock AWS region used for local boto3 clients. Defaults
                         to us-east-1.
  --config-file <file>   Local fixture config file for mocked S3/SSM. Defaults
                         to .gluekit/local.json when local settings are active.
  --runtime <str>        Local DataFrame runtime: sqlframe (fast default) or
                         pyspark (real local Spark).  [default: sqlframe]
  --spark-master <str>   Spark master URL used when --runtime=pyspark.
                         [default: local[2]]
  --help                 Show this message and exit.

  **Examples**

  - `gluekit run ../ppsc-survey/glue/scripts/job_a.py --create-bucket my-input-
  bucket`

  - `gluekit run glue/scripts/job-a.py --ssm-parameter /app/env=dev --glue-arg
  report_date=2026-06-18`

  - `gluekit run glue/scripts/job-a.py -- --JOB_NAME custom-local-job
  --report_date 2026-06-18`

Package releases

Install the repository task runner with uv tool install poethepoet==0.48.0, then run poe release-check for validation and poe build for a local wheel and source distribution. One-off usage is uvx --from poethepoet==0.48.0 poe release-check.

The publish-to-pypi.yml workflow has three jobs: prepare, publish, and release. Pushes to dev start or increment a patch development version (1.0.0 → 1.0.1.dev1 → 1.0.1.dev2). Pushes to main promote a prerelease to stable, or increment the patch when the source already has a stable version. After a stable tag exists, dev starts the next patch series. Normal releases should not edit the version manually. Merge updated main back into dev when needed to keep the branches' version baselines aligned.

poe release --branch dev --source <full-commit-sha> runs scripts/release.py in a clean checkout of that source. It uses uv version --no-sync to update pyproject.toml and uv.lock, invokes poe build, and creates a local release commit and annotated tag. This is the CI preparation command: it changes the local checkout but does not push or publish. The workflow first saves the built distributions, then atomically pushes the release commit and tag. Both branches share one release concurrency group; a stale run fails rather than overwriting newer work. Rapid pushes may supersede pending runs; the latest source should be released. Workflow pushes use GITHUB_TOKEN and do not recursively trigger another push workflow.

PyPI publication uses the pypi GitHub environment and Trusted Publishing bound to this repository and publish-to-pypi.yml. No long-lived PyPI token is needed. Only stable main versions get a GitHub Release, after PyPI succeeds. The branch rules must permit the workflow's version commit; rejected pushes leave both remote refs unchanged.

For a failed publish, use Re-run failed jobs on the original Actions run. The publish job downloads the saved wheel and source distribution without rebuilding. Identical PyPI uploads can be retried, including a partially completed upload. A full rerun recognizes the tagged source/branch before calculating a version and reuses its original artifact. Existing GitHub Releases are left intact. If preparation failed before the atomic push, a fresh attempt can rebuild and replace that run's unpublished artifact. If the original artifact has expired or been deleted after the push, stop and recover those exact files; the workflow deliberately does not rebuild a published version.

poe release-test exercises the release helper with temporary local Git remotes and real uv version changes; distribution builds are mocked. No test publishes packages or contacts cloud providers.

Gluekit's poe docs-check creates .venv-docs and installs the package with pip before checking generated CLI help. This preserves the existing docs resolution behavior separately from uv.lock and the heavier development extras. poe release-check runs that check and the release tests. After the stacked CLI and release PRs land, create dev from the updated, released main; do not start it from the older pre-refactor branch.