mkpipe-loader-clickhouse

ClickHouse loader plugin for MkPipe. Writes Spark DataFrames into ClickHouse tables using the native clickhouse-spark connector, which uses ClickHouse's binary HTTP protocol for efficient columnar inserts.

Documentation

For more detailed documentation, please visit the GitHub repository.

License

This project is licensed under the Apache 2.0 License - see the LICENSE file for details.

Connection Configuration

connections:
  clickhouse_target:
    variant: clickhouse
    host: localhost
    port: 8123
    database: target_db
    user: default
    password: mypassword

Table Configuration

pipelines:
  - name: pg_to_clickhouse
    source: pg_source
    destination: clickhouse_target
    tables:
      - name: public.events
        target_name: stg_events
        replication_method: full
        batchsize: 50000

Write Strategy

Control how data is written to ClickHouse:

      - name: public.events
        target_name: stg_events
        write_strategy: upsert       # append | replace | upsert
        write_key: [id]              # required for upsert

Strategy	ClickHouse Behavior
`append`	Insert via ClickHouse Spark connector (default for incremental)
`replace`	Drop and recreate table, then insert (default for full). With `if_exists: append`: truncate + insert (preserves table structure)
`upsert`	Creates table with `ReplacingMergeTree` engine using `write_key` as `ORDER BY`. ClickHouse deduplicates rows with the same key on background merges.

Note: ClickHouse does not support SQL MERGE statements. Upsert semantics are achieved via ReplacingMergeTree, which deduplicates asynchronously during compaction. Use FINAL in queries to get deduplicated results.

Write Parallelism & Throughput

ClickHouse loader inherits from JdbcLoader. Two parameters control write performance:

      - name: public.events
        target_name: stg_events
        replication_method: full
        batchsize: 50000        # rows per JDBC batch insert (default: 10000)
        write_partitions: 4     # coalesce DataFrame to N partitions before writing

How they work

batchsize: number of rows buffered before sending a single INSERT to ClickHouse. ClickHouse benefits greatly from large batches — use 50,000–500,000 for best throughput.
write_partitions: calls coalesce(N) on the DataFrame before writing, reducing the number of concurrent JDBC connections. Useful when you have many Spark partitions and want to limit load on ClickHouse.

Performance Notes

ClickHouse is optimized for large bulk inserts. batchsize is the most impactful parameter — increase it as much as your driver memory allows.
Avoid many small write_partitions (e.g. 1) as it reduces parallelism. A value of 4–8 balances load and throughput.
ClickHouse's MergeTree engine merges parts in the background; very frequent small inserts create many parts and degrade query performance. Prefer fewer large batches.

All Table Parameters

Parameter	Type	Default	Description
`name`	string	required	Source table name
`target_name`	string	required	ClickHouse destination table name
`replication_method`	`full` / `incremental`	`full`	Replication strategy
`batchsize`	int	`10000`	Rows per JDBC batch insert
`write_partitions`	int	—	Coalesce DataFrame to N partitions before writing
`write_strategy`	string	—	`append`, `replace`, `upsert`
`write_key`	list	—	Key columns for upsert/merge, used as `ORDER BY` in ReplacingMergeTree
`if_exists`	string	—	`replace` (drop+create) or `append` (preserve table, truncate+insert). Inherits from settings
`dedup_columns`	list	—	Columns used for `mkpipe_id` hash deduplication
`tags`	list	`[]`	Tags for selective pipeline execution
`pass_on_error`	bool	`false`	Skip table on error instead of failing

Metadata

Release files for mkpipe-loader-clickhouse 0.12.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for mkpipe-loader-clickhouse 0.12.0
File	Size	Uploaded	Provenance
mkpipe_loader_clickhouse-0.12.0.tar.gz	10.3 kB	May 14, 2026	Without provenance	Details

Built distribution (wheel)

Table of built distributions (wheels) for mkpipe-loader-clickhouse 0.12.0
File	Interpreter	ABI	Platform	Reset
mkpipe_loader_clickhouse-0.12.0-py3-none-any.whl 11.1 kB May 14, 2026 Without provenance	Python 3	none	any	Details

Total release size: 21.4 kB

Release files / mkpipe_loader_clickhouse-0.12.0.tar.gz

Download URL	mkpipe_loader_clickhouse-0.12.0.tar.gz
Size	10.3 kB
Tags	Source
SHA-256 checksum How to use checksums	`50bfe8c5e886a6e597da94ecd11fe21c0c318465acb2bf9d9435ee3a491e036a`
BLAKE2b-256 checksum How to use checksums	`894a6cf5d38c19864e2427329da38735b4d5abd632cb950b0d876874f4ffe924`
Upload date	May 14, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.2.0 CPython/3.11.15`

Release files / mkpipe_loader_clickhouse-0.12.0-py3-none-any.whl

Download URL	mkpipe_loader_clickhouse-0.12.0-py3-none-any.whl
Size	11.1 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`39626b3de337c4e85b4fd1a1105fc304eb353ca6251d6c457052c45ff84779cb`
BLAKE2b-256 checksum How to use checksums	`deced652721ad3f85c41abd870e6973ede983aaa74f2a0abd3a9dc24c2ffb9c3`
Upload date	May 14, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.2.0 CPython/3.11.15`

Security for mkpipe-loader-clickhouse 0.12.0

Report project as malware

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.