Skip to main content

An AWS Aurora Serverless Data API dialect for SQLAlchemy

Project description

This package provides a SQLAlchemy dialect for accessing PostgreSQL and MySQL databases via the AWS Aurora Data API.

Installation

pip install sqlalchemy-aurora-data-api

Prerequisites

  • Set up an AWS Aurora Serverless cluster and enable Data API access for it. If you have previously set up an Aurora Serverless cluster, you can enable Data API with the following AWS CLI command:

    aws rds modify-db-cluster --db-cluster-identifier DB_CLUSTER_NAME --enable-http-endpoint --apply-immediately
    
  • Save the database credentials in AWS Secrets Manager using a format expected by the Data API (a JSON object with the keys username and password):

    aws secretsmanager put-secret-value --secret-id MY_DB_CREDENTIALS --secret-string "$(jq -n '.username=env.PGUSER | .password=env.PGPASSWORD')"
    
  • Configure your AWS command line credentials using standard AWS conventions. You can verify that everything works correctly by running a test query via the AWS CLI:

    aws rds-data execute-statement --resource-arn RESOURCE_ARN --secret-arn SECRET_ARN --sql "select * from pg_catalog.pg_tables"
    

Usage

The package registers two SQLAlchemy dialects, mysql+auroradataapi:// and postgresql+auroradataapi://. Two sqlalchemy.create_engine() connect_args keyword arguments are required to connect to the database:

  • aurora_cluster_arn (also referred to as resourceArn in the Data API documentation)
    • If not given as a keyword argument, this can also be specified using the AURORA_CLUSTER_ARN environment variable
  • secret_arn (the database credentials secret)
    • If not given as a keyword argument, this can also be specified using the AURORA_SECRET_ARN environment variable

All connection string contents other than the protocol (dialect) and the database name (path component, my_db_name in the example below) are ignored.

from sqlalchemy import create_engine

cluster_arn = "arn:aws:rds:us-east-1:123456789012:cluster:my-aurora-serverless-cluster"
secret_arn = "arn:aws:secretsmanager:us-east-1:123456789012:secret:MY_DB_CREDENTIALS"

engine = create_engine('postgresql+auroradataapi://:@/my_db_name',
                       echo=True,
                       connect_args=dict(aurora_cluster_arn=cluster_arn, secret_arn=secret_arn))

with engine.connect() as conn:
    for result in conn.execute("select * from pg_catalog.pg_tables"):
        print(result)

Motivation

The RDS Data API is the link between the AWS Lambda serverless environment and the sophisticated features provided by PostgreSQL and MySQL. The Data API tunnels SQL over HTTP, which has advantages in the context of AWS Lambda:

  • It eliminates the need to open database ports to the AWS Lambda public IP address pool
  • It uses stateless HTTP connections instead of stateful internal TCP connection pools used by most database drivers (the stateful pools become invalid after going through AWS Lambda freeze-thaw cycles, causing connection errors and burdening the database server with abandoned invalid connections)
  • It uses AWS role-based authentication, eliminating the need for the Lambda to handle database credentials directly

Debugging

This package uses standard Python logging conventions. To enable debug output, set the package log level to DEBUG:

logging.basicConfig()

logging.getLogger("aurora_data_api").setLevel(logging.DEBUG)

License

Licensed under the terms of the Apache License, Version 2.0.

https://travis-ci.org/chanzuckerberg/sqlalchemy-aurora-data-api.png https://codecov.io/github/chanzuckerberg/sqlalchemy-aurora-data-api/coverage.svg?branch=master https://img.shields.io/pypi/v/sqlalchemy-aurora-data-api.svg https://img.shields.io/pypi/l/sqlalchemy-aurora-data-api.svg https://readthedocs.org/projects/sqlalchemy-aurora-data-api/badge/?version=latest

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for sqlalchemy-aurora-data-api, version 0.2.0
Filename, size File type Python version Upload date Hashes
Filename, size sqlalchemy_aurora_data_api-0.2.0-py2.py3-none-any.whl (9.5 kB) File type Wheel Python version py2.py3 Upload date Hashes View hashes
Filename, size sqlalchemy-aurora-data-api-0.2.0.tar.gz (7.2 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page