DBT Copilot Python

A set of utility functions for running Django & Flask apps in AWS ECS via AWS Copilot.

Using dbt-copilot-python

Supported Python versions

3.9, 3.10, 3.11, 3.12 and 3.13.

Installation

pip install dbt-copilot-python

With celery

If you are using celery, you should also install an appropriate version of celery using your project's package manager, or our built-in extra.

# Poetry
poetry add celery

# Pip
pip install celery
pip freeze | grep "^celery==" >> requirements.txt

# Pipenv
pipenv install celery

OR

# Poetry
poetry add dbt-copilot-python[celery]

# Pip
pip install dbt-copilot-python[celery]
pip freeze | grep "^celery==" >> requirements.txt

# Pipenv
pipenv install dbt-copilot-python[celery]

With error tracking

If you intend on using the error tracking filter, you should also install an appropriate version of ddtrace using your project's package manager, or our built-in extra.

# Poetry
poetry add ddtrace

# Pip
pip install ddtrace
pip freeze | grep "^ddtrace==" >> requirements.txt

# Pipenv
pipenv install ddtrace

OR

# Poetry
poetry add dbt-copilot-python[error-tracking]

# Pip
pip install dbt-copilot-python[error-tracking]
pip freeze | grep "^ddtrace==" >> requirements.txt

# Pipenv
pipenv install dbt-copilot-python[error-tracking]

Usage

ALLOWED_HOSTS setting

Add the ECS container IP to ALLOWED_HOSTS setting so that the Application Load Balancer (ALB) healthcheck will succeed:

from dbt_copilot_python.network import setup_allowed_hosts

ALLOWED_HOSTS = [...]

ALLOWED_HOSTS = setup_allowed_hosts(ALLOWED_HOSTS)

Celery health check

Install the library with celery.

Add the health check in your application's Celery config file...

from dbt_copilot_python.celery_health_check import healthcheck

celery_app = Celery("application_name")
...

celery_app = healthcheck.setup(celery_app)

Add the health check to the Celery worker service in docker-compose.yml...

healthcheck:
  test: [ "CMD-SHELL", "python -m dbt_copilot_python.celery_health_check.healthcheck" ]
  interval: 10s
  timeout: 5s
  retries: 2
  start_period: 5s

In your *-deploy codebase, add the health check to the Celery worker service in copilot/celery-worker/manifest.yml...

healthcheck:
  command: [ "CMD-SHELL", "launcher bash -c 'python -m dbt_copilot_python.celery_health_check.healthcheck'" ]
  interval: 10s
  timeout: 5s
  retries: 2
  start_period: 10s

DATABASES setting

To configure the DATABASES setting from an RDS JSON object stored in AWS Secrets Manager, there are two options.

1. Configure the DATABASES setting to use a database URL (recommended):

   Note: This is dependent on the dj-database-url package which can be installed via pip install dj-database-url.

   import dj_database_url
   
   from dbt_copilot_python.database import database_url_from_env
   
   DATABASES = {
       "default": dj_database_url.config(
           default=database_url_from_env("DATABASE_ENV_VAR_KEY")
       )
   }
   

2. Configure the DATABASES setting to use a dictionary containing the settings:

   from dbt-copilot-python.database import database_from_env
   
   DATABASES = database_from_env("DATABASE_ENV_VAR_KEY")
   

Error tracking

To enable error tracking for logs you must attach the DatadogErrorTrackingFilter to any of the loggers which you'd like to track logs emitted at a level >=logging.ERROR.

Generally speaking you could do something as follows

import logging

from dbt_copilot_python import DatadogErrorTrackingFilter

logger = logging.getLogger()
logger.addFilter(DatadogErrorTrackingFilter())

Contributing to dbt-copilot-python

Requirements

• Poetry; pip install poetry

Install dependencies & pre-commit hooks

poetry install && poetry run pre-commit install

Run the tests

poetry run pytest

Publishing

To publish the Python package dbt-copilot-python, you will need an API token.

1. Acquire API token from Passman.
   • Request access from the SRE team.
   • Note: You will need access to the platform group in Passman.
2. Run poetry config pypi-token.pypi <token> to add the token to your Poetry configuration.

Update the version, as the same version cannot be published to PyPi.

poetry version patch

More options for the version command can be found in the Poetry documentation. For example, for a minor version bump: poetry version minor.

Update the CHANGELOG.md with details of the changes you've made.

Build the Python package.

poetry build

Publish the Python package.

Note: Make sure your Pull Request (PR) is approved and contains the version upgrade in pyproject.toml before publishing the package.

poetry publish

Check the PyPi Release history to make sure the package has been updated.

For an optional manual check, install the package locally and test everything works as expected.