mkpipe-extractor-mysql

MySQL extractor plugin for MkPipe. Reads MySQL tables via JDBC.

Documentation

For more detailed documentation, please visit the GitHub repository.

License

This project is licensed under the Apache 2.0 License - see the LICENSE file for details.

Connection Configuration

connections:
  mysql_source:
    variant: mysql
    host: localhost
    port: 3306
    database: mydb
    user: myuser
    password: mypassword

Table Configuration

pipelines:
  - name: mysql_to_pg
    source: mysql_source
    destination: pg_target
    tables:
      - name: orders
        target_name: stg_orders
        replication_method: full
        fetchsize: 100000

Incremental Replication

      - name: orders
        target_name: stg_orders
        replication_method: incremental
        iterate_column: updated_at
        iterate_column_type: datetime
        partitions_column: id
        partitions_count: 4
        fetchsize: 50000

Custom SQL

      - name: orders
        target_name: stg_orders
        replication_method: incremental
        iterate_column: updated_at
        iterate_column_type: datetime
        custom_query: "SELECT id, user_id, total FROM orders WHERE {query_filter}"

Read Parallelism

Set partitions_column and partitions_count to read in parallel via multiple JDBC connections:

      - name: events
        target_name: stg_events
        replication_method: incremental
        iterate_column: created_at
        iterate_column_type: datetime
        partitions_column: id
        partitions_count: 8
        fetchsize: 50000

Performance Notes

• partitions_column should be a numeric column (e.g. primary key).
• MySQL JDBC streaming requires fetchsize: Integer.MIN_VALUE to avoid loading all rows into memory — mkpipe uses a positive fetchsize which works with standard cursors. For very large tables, keep fetchsize moderate (50,000–100,000).
• Partitioning only applies to incremental replication.

All Table Parameters