mkpipe-extractor-oracledb

Oracle Database extractor plugin for MkPipe. Reads Oracle tables via JDBC.

Documentation

For more detailed documentation, please visit the GitHub repository.

License

This project is licensed under the Apache 2.0 License - see the LICENSE file for details.

Connection Configuration

connections:
  oracle_source:
    variant: oracledb
    host: localhost
    port: 1521
    database: ORCL
    user: myuser
    password: mypassword

Table Configuration

pipelines:
  - name: oracle_to_pg
    source: oracle_source
    destination: pg_target
    tables:
      - name: MYSCHEMA.EVENTS
        target_name: stg_events
        replication_method: full
        fetchsize: 100000

Incremental Replication

      - name: MYSCHEMA.EVENTS
        target_name: stg_events
        replication_method: incremental
        iterate_column: UPDATED_AT
        iterate_column_type: datetime
        partitions_column: ID
        partitions_count: 8
        fetchsize: 50000

Custom SQL

      - name: MYSCHEMA.EVENTS
        target_name: stg_events
        replication_method: full
        custom_query: "SELECT ID, USER_ID, EVENT_TYPE FROM MYSCHEMA.EVENTS WHERE {query_filter}"

Read Parallelism

Set partitions_column and partitions_count for parallel JDBC reads. Partitioning only applies to incremental replication.

Performance Notes

• Oracle's ROWID range partitioning is the most efficient for parallel scans, but mkpipe uses column-range partitioning for simplicity. Use a numeric PK column for best distribution.
• Oracle JDBC fetchsize defaults are often low — explicitly set fetchsize: 50000 or higher.

All Table Parameters