mkpipe-loader-mysql

MySQL loader plugin for MkPipe. Writes Spark DataFrames into MySQL tables via JDBC.

Documentation

For more detailed documentation, please visit the GitHub repository.

License

This project is licensed under the Apache 2.0 License - see the LICENSE file for details.

Connection Configuration

connections:
  mysql_target:
    variant: mysql
    host: localhost
    port: 3306
    database: mydb
    user: myuser
    password: mypassword

Table Configuration

pipelines:
  - name: pg_to_mysql
    source: pg_source
    destination: mysql_target
    tables:
      - name: public.orders
        target_name: stg_orders
        replication_method: full
        batchsize: 10000

Write Strategy

Control how data is written to MySQL:

      - name: public.orders
        target_name: stg_orders
        write_strategy: upsert       # append | replace | upsert | merge
        write_key: [id]              # required for upsert/merge

Strategy	MySQL Behavior
`append`	Plain `INSERT` via JDBC (default for incremental)
`replace`	Drop and recreate table, then insert (default for full). With `if_exists: append`: truncate + insert (preserves schema/indexes)
`upsert`	`INSERT ... ON DUPLICATE KEY UPDATE` via temp table
`merge`	Same as upsert for MySQL

Note: upsert/merge requires write_key. The target table must have a unique index/primary key on the write_key columns.

Write Parallelism & Throughput

      - name: public.orders
        target_name: stg_orders
        replication_method: full
        batchsize: 10000
        write_partitions: 4

batchsize: rows per JDBC batch INSERT. MySQL handles 5,000–20,000 well.
write_partitions: reduces concurrent JDBC connections via coalesce(N).

All Table Parameters

Parameter	Type	Default	Description
`name`	string	required	Source table name
`target_name`	string	required	MySQL destination table name
`replication_method`	`full` / `incremental`	`full`	Replication strategy
`batchsize`	int	`10000`	Rows per JDBC batch insert
`write_partitions`	int	—	Coalesce DataFrame to N partitions before writing
`write_strategy`	string	—	`append`, `replace`, `upsert`, `merge`
`write_key`	list	—	Key columns for upsert/merge (required)
`if_exists`	string	—	`replace` (drop+create) or `append` (preserve table, truncate+insert). Inherits from settings
`dedup_columns`	list	—	Columns used for `mkpipe_id` hash deduplication
`tags`	list	`[]`	Tags for selective pipeline execution
`pass_on_error`	bool	`false`	Skip table on error instead of failing

Metadata

Release files for mkpipe-loader-mysql 0.7.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for mkpipe-loader-mysql 0.7.0
File	Size	Uploaded	Provenance
mkpipe_loader_mysql-0.7.0.tar.gz	7.0 kB	May 14, 2026	Without provenance	Details

Built distribution (wheel)

Table of built distributions (wheels) for mkpipe-loader-mysql 0.7.0
File	Interpreter	ABI	Platform	Reset
mkpipe_loader_mysql-0.7.0-py3-none-any.whl 7.6 kB May 14, 2026 Without provenance	Python 3	none	any	Details

Total release size: 14.6 kB

Release files / mkpipe_loader_mysql-0.7.0.tar.gz

Download URL	mkpipe_loader_mysql-0.7.0.tar.gz
Size	7.0 kB
Tags	Source
SHA-256 checksum How to use checksums	`45933ca73f72dc2dfc2e1419fb0c9f8642664dd46d91c0890c1e68308a6ce1c7`
BLAKE2b-256 checksum How to use checksums	`8367a96e90df1ca2f7bbc51ed638ec89b8d861165ad41c965d394ca89d8d9ea3`
Upload date	May 14, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.2.0 CPython/3.11.15`

Release files / mkpipe_loader_mysql-0.7.0-py3-none-any.whl

Download URL	mkpipe_loader_mysql-0.7.0-py3-none-any.whl
Size	7.6 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`91407a50621e28d242c53a852e8f42be72c6a38071b38fe33a07b96515cf39b0`
BLAKE2b-256 checksum How to use checksums	`ddabf4f0baa1cf433d201077e8ca4c70d2e3b594cadd13069a35a2687223d116`
Upload date	May 14, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.2.0 CPython/3.11.15`

Security for mkpipe-loader-mysql 0.7.0

Report project as malware

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.