ray-clickhouse

Ray Data read and append connector for ClickHouse physical tables.

This release provides:

• structured physical-table reads returning ray.data.Dataset;
• single-query reads from direct tables, Views, and Distributed tables;
• Arrow streaming with bounded rows and bytes per emitted block;
• optional partition and integer range planning for direct MergeTree-family tables;
• append writes plus explicit safe create/overwrite table-management modes;
• structured ordering and Ray task resource passthrough;
• fail-closed schema, type, permission, timeout, and ambiguous-write errors.

The connector does not provide arbitrary SQL execution, arbitrary DDL, delete, update, upsert, connector-side Distributed shard routing, cross-task transactions, or exactly-once guarantees. Distributed reads use one ClickHouse query; ClickHouse owns shard routing.

Automatic range planning is opt-in. For a direct MergeTree-family table with a simple sorting key starting with an integer column, use split="auto" to select that column without supplying range_column:

from ray_clickhouse import read_clickhouse

dataset = read_clickhouse(
    host="clickhouse.example",
    database="analytics",
    table="events",
    split="auto",
    target_tasks=4,
)

The default remains split="single". Automatic planning reuses the existing filtered range aggregate and disjoint range planner, not ordered pagination. Unsuitable sorting keys fall back to one query; use discovery_policy="error" to require an eligible range plan. Authentication, permission, and transport failures still raise. See Architecture for supported keys and consistency limits.

The read API follows the Ray Data Datasource/ReadTask contract. The write API uses Dataset.write_datasink() and does not monkey-patch Ray Dataset methods. The package-root public API consists of read_clickhouse(), write_clickhouse(), WriteReceipt, and the documented error hierarchy. Datasource and Datasink implementation classes are internal and are not independent compatibility promises.

Documentation

• Documentation index
• Compatibility
• Architecture
• Security model
• Troubleshooting
• Contributing
• Security policy
• Release notes

Query result reads

The read API also accepts a trusted query source. Query reads require exactly one of table or query, use one data-read task, and infer the result schema with DESCRIBE and a zero-row Arrow probe.

dataset = read_clickhouse(
    host="clickhouse.example",
    database="analytics",
    query=(
        "SELECT tenant_id, count() AS event_count FROM events "
        "WHERE tenant_id = %(tenant)s GROUP BY tenant_id"
    ),
    query_parameters={"tenant": 42},
)

Query mode supports the documented SELECT/WITH subset, client-side named value bindings, and unique simple result aliases. Table projection, filter, ordering, range columns, and split modes are supplied in the SQL rather than connector options. The connector enforces readonly=1 and rejects multiple statements, DDL/DML, output formats, output files, and inline settings. Metadata probes and the data read are independent requests and do not share a database snapshot. A zero-row probe can still incur server work. override_num_blocks changes Ray output blocks and does not multiply the query.

Usage

from ray_clickhouse import read_clickhouse, write_clickhouse

dataset = read_clickhouse(
    host="clickhouse.example",
    database="analytics",
    table="events",
    columns=("event_id", "tenant_id", "payload"),
    filter="tenant_id = %(tenant)s",
    query_parameters={"tenant": 42},
)

write_clickhouse(
    dataset,
    host="clickhouse.example",
    database="analytics",
    table="events_copy",
    insert_mode="sync",
)

# Explicit table management is opt-in and restricted to safe generated DDL.
write_clickhouse(
    dataset,
    host="clickhouse.example",
    database="analytics",
    table="events_new",
    write_mode="create",
    order_by=("event_id",),
)

When diagnosing an Arrow or transport failure, set diagnostic_flush_logs=True on read_clickhouse() if the ClickHouse account may run SYSTEM FLUSH LOGS:

dataset = read_clickhouse(
    host="clickhouse.example",
    database="analytics",
    table="events",
    diagnostic_flush_logs=True,
)

Read failures carry a generated query ID and a bounded client_diagnostic with the underlying exception type, operation, and safe message context. The connector performs a bounded, best-effort lookup in the discovered ClickHouse query-log table and appends the server exception code and message when available. It never replays the failed SQL; if query logging is unavailable, the client diagnostic and query ID remain available. Raw exception chains remain suppressed to protect credentials. The diagnostic text is server-provided and may include query context, so do not put credentials in query predicates or parameters.

filter is a trusted predicate fragment, not an arbitrary SQL query. Use tenant_id = {tenant:Nullable(UInt32)} with query_parameters={"tenant": 42} for native server-side binding. Existing %(tenant)s client binding remains supported. Mixing binding styles or using identifier parameters is rejected. Typed parameter-looking text inside quotes/comments is rejected when bindings are active. Supported hints are String, FixedString, Bool, Int8/16/32/64, UInt8/16/32/64, Float32/64, Date/Date32, DateTime/DateTime64, Decimal/32/64/128, and one Nullable wrapper. DateTime64 precision must be 0 through 9; timezone hints use a quoted timezone name. Collection and identifier hints are unsupported. Values must be passed through query_parameters; the connector rejects unrestricted statement execution and does not provide delete, update, upsert, connector-side Distributed shard routing, or exactly-once semantics. write_mode="overwrite" is destructive and must be selected explicitly. For create/overwrite, nullable columns must be listed explicitly through nullable_columns; timestamp precision is preserved and unsupported binary types are rejected.

Writing an empty Dataset is a confirmed no-op: it returns a zero-valued WriteReceipt without performing ClickHouse discovery, table management, or INSERT. If Ray returns without invoking the normal sink completion callback, write_clickhouse() raises WriteError instead of reporting a zero-valued success. MaterializedView reads are not part of the supported capability profile.

Compatibility

ray-clickhouse supports final Ray releases >=2.55,<2.59 on Python 3.10–3.13. Prerelease, development, and post-release Ray builds are rejected. PEP 440 local build suffixes are allowed. Pandas is resolved through ray[data]; it is not an independent ray-clickhouse compatibility promise.

Development

uv sync --extra dev
uv sync --extra dev --group docs
uv run ruff format --check .
uv run ruff check .
uv run mypy
uv run python tools/check_compatibility.py
uv run python tools/check_release.py check
uv run pytest tests/unit tests/contract
uv build
uv run twine check dist/*.whl dist/*.tar.gz
uv run python tools/check_docs.py
make -C doc html
make -C doc spelling
make -C doc linkcheck
./scripts/run_clickhouse_it.sh
./scripts/run_ray_cluster_it.sh

ClickHouse and multi-node Ray integration commands remain available in the repository, but they are required only when the affected production or infrastructure behavior changes. The pinned infrastructure baselines are ClickHouse 26.8.1.2041 and a Ray 2.58.0/Python 3.12 three-node runtime.