aurora-to-rs 0.1.0

Lean Aurora PostgreSQL to Redshift loader via copy_expert + S3 COPY

aurora_to_rs

Lean Aurora PostgreSQL to Redshift loader using psycopg2.copy_expert + S3 + Redshift COPY.

~10x faster than DataFrame-based approaches for large tables (bypasses pandas entirely).

Install

pip install aurora_to_rs

Usage

from aurora_to_rs import aurora_to_rs

loader = aurora_to_rs(
    region_name='ap-south-1',
    s3_bucket='my-bucket',
    redshift_c=redshift_conn,          # psycopg2 connection
    postgres_engine=pg_engine,          # SQLAlchemy engine
    iam_role_arn='arn:aws:iam::123:role/MY_ROLE'
)

# Full table refresh (TRUNCATE + COPY)
loader.upsert("SELECT * FROM master.items", "master.items", ['item_id'], clear_dest_table=True)

# Incremental upsert (staging-based)
loader.upsert("SELECT * FROM tran.orders WHERE dt>=current_date-1", "tran.orders", ['order_id'])

# Delete and insert
loader.delete_and_insert(
    "SELECT * FROM tran.shipment WHERE dt>=current_date-3",
    "tran.shipment", "dt>=current_date-3",
    min_timestamp='2026-02-19', timestamp_col='dt'
)

# Direct copy (no delete)
loader.copy_expert_upload("SELECT * FROM staging.temp", "staging.temp")

Methods

Method	Use Case
copy_expert_upload	Direct Aurora -> S3 -> Redshift COPY
delete_and_insert	DELETE by filter, then COPY new data
upsert	TRUNCATE+COPY (clear_dest_table=True) or staging-based upsert