mkpipe-loader-mysql 0.7.0

MySQL loader for mkpipe.

mkpipe-loader-mysql

MySQL loader plugin for MkPipe. Writes Spark DataFrames into MySQL tables via JDBC.

Documentation

For more detailed documentation, please visit the GitHub repository.

License

This project is licensed under the Apache 2.0 License - see the LICENSE file for details.

---

Connection Configuration

connections:
  mysql_target:
    variant: mysql
    host: localhost
    port: 3306
    database: mydb
    user: myuser
    password: mypassword

---

Table Configuration

pipelines:
  - name: pg_to_mysql
    source: pg_source
    destination: mysql_target
    tables:
      - name: public.orders
        target_name: stg_orders
        replication_method: full
        batchsize: 10000

---

Write Strategy

Control how data is written to MySQL:

      - name: public.orders
        target_name: stg_orders
        write_strategy: upsert       # append | replace | upsert | merge
        write_key: [id]              # required for upsert/merge

Strategy	MySQL Behavior
append	Plain INSERT via JDBC (default for incremental)
replace	Drop and recreate table, then insert (default for full). With if_exists: append: truncate + insert (preserves schema/indexes)
upsert	INSERT ... ON DUPLICATE KEY UPDATE via temp table
merge	Same as upsert for MySQL

Note: upsert/merge requires write_key. The target table must have a unique index/primary key on the write_key columns.

---

Write Parallelism & Throughput

      - name: public.orders
        target_name: stg_orders
        replication_method: full
        batchsize: 10000
        write_partitions: 4

• batchsize: rows per JDBC batch INSERT. MySQL handles 5,000–20,000 well.
• write_partitions: reduces concurrent JDBC connections via coalesce(N).

---

All Table Parameters

Parameter	Type	Default	Description
name	string	required	Source table name
target_name	string	required	MySQL destination table name
replication_method	full / incremental	full	Replication strategy
batchsize	int	10000	Rows per JDBC batch insert
write_partitions	int	—	Coalesce DataFrame to N partitions before writing
write_strategy	string	—	append, replace, upsert, merge
write_key	list	—	Key columns for upsert/merge (required)
if_exists	string	—	replace (drop+create) or append (preserve table, truncate+insert). Inherits from settings
dedup_columns	list	—	Columns used for mkpipe_id hash deduplication
tags	list	[]	Tags for selective pipeline execution
pass_on_error	bool	false	Skip table on error instead of failing