mkpipe-extractor-redshift

Amazon Redshift extractor plugin for MkPipe. Reads Redshift tables via JDBC.

Documentation

For more detailed documentation, please visit the GitHub repository.

License

This project is licensed under the Apache 2.0 License - see the LICENSE file for details.

Connection Configuration

connections:
  redshift_source:
    variant: redshift
    host: my-cluster.abc123.us-east-1.redshift.amazonaws.com
    port: 5439
    database: mydb
    schema: public
    user: myuser
    password: mypassword

Table Configuration

pipelines:
  - name: redshift_to_pg
    source: redshift_source
    destination: pg_target
    tables:
      - name: public.events
        target_name: stg_events
        replication_method: full
        fetchsize: 100000

Incremental Replication

      - name: public.events
        target_name: stg_events
        replication_method: incremental
        iterate_column: updated_at
        iterate_column_type: datetime
        partitions_column: id
        partitions_count: 8
        fetchsize: 50000

Custom SQL

      - name: public.events
        target_name: stg_events
        replication_method: full
        custom_query: "SELECT id, user_id, event_type FROM public.events WHERE {query_filter}"

Read Parallelism

Set partitions_column and partitions_count to read in parallel via multiple JDBC connections. Partitioning only applies to incremental replication.

Performance Notes

• For large Redshift tables, using UNLOAD to S3 + Spark S3 read is significantly faster than JDBC. JDBC is suitable for moderate-sized extracts.
• Keep fetchsize at 50,000–200,000 to balance memory usage and round-trips.

All Table Parameters