ray-hive

ray-hive is an independent HiveServer2 datasource for Ray Data. HiveServer2 owns table-format, storage, authorization, and query execution. Ray workers receive Arrow batches only through the HS2 endpoint; they do not read HDFS, object storage, table manifests, or the Hive Metastore.

The project is currently Alpha. The compatibility target is Python 3.12/3.13, final Ray releases >=2.55.1,<2.59, PyArrow 19.0.1, and Apache Hive 4.2.0. Production Core status will not be claimed until the real Hive, security, network-purity, cancellation, and Ray cluster conformance suites pass. See SUPPORT.md for evidence levels, API stability, maintenance ownership, and the external community support boundary.

Install

pip install ray-hive

The current native profile supports the binary Thrift transport with NOSASL. HTTP, SASL, Kerberos, LDAP, CUSTOM authentication, and alternative drivers fail closed until their dedicated extras and real infrastructure matrices are implemented and verified.

Read a table

from ray_hive import (
    HiveConnectionOptions,
    HiveReadOptions,
    HiveTableIdentifier,
    read_hive,
)

dataset = read_hive(
    connection=HiveConnectionOptions(
        host="hiveserver2.example.net",
        port=10000,
        database="analytics",
        auth="NOSASL",
        username="ray-reader",
    ),
    read=HiveReadOptions(
        table=HiveTableIdentifier(database="analytics", table="events"),
        columns=("event_id", "event_time", "score"),
    ),
)

The Alpha native NOSASL profile neither resolves nor sends a password. SecretRef and custom credential providers are available for authenticated profiles under development: references are bound to the configured endpoint and resolved only by a trusted process immediately before it opens an HS2 session. Table reads perform schema planning on the Ray driver, so authenticated table reads require the provider to be resolvable on both the driver and workers. Raw SQL with an explicit schema skips driver-side HS2 planning and can keep credential resolution worker-only.

Structured source predicates are available from ray_hive.sql:

from ray_hive.sql import col

read = HiveReadOptions(
    table=HiveTableIdentifier("analytics", "events"),
    filter=col("score").ge(0) & col("event_type").isin(["open", "close"]),
)

The built-in ray_hive.sql builder is part of the stable public API. Only its immutable predicate tree is accepted for source pushdown; custom Column or Predicate subclasses are not supported SQL extension points.

Raw SQL accepts one conservatively validated top-level SELECT; WITH, EXPLAIN, DDL, DML, comments, unquoted semicolons, Hive variable substitution, and ambiguous quoting or parentheses fail during immutable option construction before credential resolution or network access. unsafe_where_sql is a trusted-code predicate escape hatch, but it is confined to one balanced fragment with the same delimiter, comment, substitution, and quoting checks. Both inputs are redacted from object representations and default diagnostics. The client-side checks prevent submission of DDL, DML, and multiple statements; they do not replace HiveServer2 authorization or prove that every UDF or server hook is free of external side effects.

Consistency and parallelism

The default strict mode executes exactly one Hive query in one Ray read task and streams multiple Arrow blocks from that operation. This preserves the single Hive query snapshot, including ACID and Iceberg semantics supplied by the server. The public facade forces Ray task max_retries=0; a failure after partial output fails the Dataset instead of replaying and appending a second query attempt.

The stable public surface consists of the top-level facade, configuration and error types, plus the built-in ray_hive.sql predicate builder. Only read_hive() carries the strict retry guarantee; lower-level extension points are DeveloperAPIs. See SUPPORT.md for the compatibility boundary.

HS2 exposes one sequential result stream, so initial source parallelism is one. Blocks can be processed in parallel after they enter Ray. Experimental independent_queries requires explicit typed split predicates and uses multiple independent Hive snapshots; it is not equivalent to strict mode and rejects a global source limit.

Ray Dataset.filter() remains a Ray-side filter in V1. The connector intentionally declines Ray predicate pushdown because the supported Ray releases have no partial-residual contract. Explicit HiveReadOptions.filter predicates are validated as a complete expression and pushed into Hive.

Security boundaries

• TLS certificate verification is enabled whenever TLS is selected, unless a caller explicitly disables it.
• Session configuration is allowlisted.
• Query-bearing identifiers, literals, raw SQL, and unsafe predicates reject Hive variable substitution because substitution occurs before server parsing.
• Credentials, full SQL, operation secrets, and row values are excluded from default repr and diagnostics.
• The client does not accept filesystem paths, Hadoop configuration, HMS endpoints, storage credentials, or table locations.
• Client-side query validation does not replace HiveServer2 authorization.

See docs/security.md, docs/consistency.md, and docs/compatibility.md before evaluating the connector for a production environment.

Development

uv sync --extra dev --extra telemetry
.venv/bin/ruff format --check .
.venv/bin/ruff check .
.venv/bin/mypy
.venv/bin/python -m pytest tests/unit tests/contract --cov=ray_hive

SQL, schema, protocol, transport, authentication, lifecycle, cancellation, or distributed Ray changes must also pass the real ray-hive-it infrastructure. Never substitute mocks for that gate or use broad Docker cleanup commands.

The generated Hive 4.2.0 TCLIService code is reproducible with:

.venv/bin/python scripts/generate_thrift_stubs.py --build-image --check

License

Apache License 2.0. The generated TCLIService bindings are derived from Apache Hive's Apache-2.0 IDL; exact provenance and checksums are recorded beside the generated package.