vibedata-dlt-redshift-studio

The dlt runtime execution unit for an Amazon Redshift Domain, installed in the Studio-local intent venv.

import dlt
from vibedata.dlt.redshift import setup_environment, finalize

setup_environment()
pipeline = dlt.pipeline(
    pipeline_name="my_pipeline", destination="redshift", dataset_name="analytics"
)
try:
    pipeline.run(source)
except Exception as e:
    finalize(pipeline, error_message=str(e))
    raise
finalize(pipeline)

setup_environment() registers the Domain's Secret Store provider and mints the Ephemeral database's password into this process, under dlt's own DESTINATION__REDSHIFT__CREDENTIALS__PASSWORD. Studio publishes the other four coordinates — HOST, PORT, DATABASE, USERNAME — into the Intent environment under the matching names, and dlt reads those natively, so the pipeline names no VD_* variable itself.

The password is separate because Redshift authenticates with a credential minted per invocation, scoped to one database and valid for at most an hour.

The Broker's database_password answer also names the database user the password was minted for, and the password is valid for that user only. That username replaces the projected USERNAME, verbatim: a composite such as IAMA:<user>:<group> is the whole login name and is not parsed. An older Broker answers with access_token alone, and the projected username stays.

The credential is minted once per run and is not refreshed. A connection opened after it expires fails to authenticate, so a single dlt command is bounded by the credential's lifetime.

Without one, dlt refuses to configure the destination and names DESTINATION__REDSHIFT__CREDENTIALS__PASSWORD — which no authored pipeline sets, so this unit fails at the mint and names the Broker instead. A blank password is the case worth guarding: dlt resolves it cleanly and the load then reaches Redshift as a real login attempt, so an empty Broker answer is refused rather than set.

The minted password lives in os.environ for the life of the process, as the MotherDuck unit's token does — inherited by any child it spawns.

finalize() persists the audit tables. A Redshift load writes directly, so there is nothing to sync afterwards.