h2o-connector-service 0.1.0.dev11001

Python client SDK for the H2O Connector Service — create connectors, open connections, and stream extracted data

h2o-connector-service

• Python client: https://pypi.org/project/h2o-connector-service/
• Source: https://github.com/h2oai/connector-service

Python client SDK for the H2O Connector Service. Provides a high-level API to create connectors, open connections, and stream extracted data from supported data sources (PostgreSQL, Snowflake, Hive, Delta Lake, Blob Storage, and more).

pip install h2o-connector-service

Quick Start (H2O Cloud Discovery)

The recommended way to connect when running on H2O AI Cloud:

from h2o_connector_service import ConnectorService

with ConnectorService.from_discovery("https://cloud.h2o.ai", "my-workspace") as svc:
    with svc.open_session("postgresql", {
        "host": "db.example.com",
        "port": "5432",
        "database": "mydb",
        "username": "user",
        "password": "pass",
    }, worker_name="pg-worker") as session:
        # Stream rows one-by-one (constant memory)
        for row in session.stream_records():
            print(row)

Quick Start (Manual / Legacy)

For direct connections without H2O Cloud Discovery (deprecated):

from h2o_connector_service import ConnectorService

with ConnectorService("http://localhost:8080", "<your-oidc-token>", "my-workspace") as svc:
    with svc.open_session("postgresql", {
        "host": "db.example.com",
        "port": "5432",
        "database": "mydb",
        "username": "user",
        "password": "pass",
    }, worker_name="pg-worker") as session:
        for row in session.stream_records():
            print(row)

Output Formats

Once you have a session, stream data into various formats:

# CSV file (memory-safe — rows written as they arrive)
session.stream_to_csv("output.csv")

# pandas DataFrame (requires: pip install h2o-connector-service[pandas])
df = session.stream_to_pandas()

# Parquet file (memory-safe, chunked row groups)
# requires: pip install h2o-connector-service[parquet]
session.stream_to_parquet("output.parquet")

# datatable Frame (memory-safe, chunked rbind)
# requires: pip install h2o-connector-service[datatable]
frame = session.stream_to_data_table()

# H2O Frame (requires running H2O cluster + h2o.init())
# requires: pip install h2o-connector-service[h2o]
h2o_frame = session.stream_to_h2o_frame()

# Collect all rows into a list of dicts
records = session.stream_to_records()

Advanced Usage

For full control over the connector lifecycle, use the individual service clients:

from h2o_connector_service import (
    Client,
    ConnectorServiceClient,
    ConnectionServiceClient,
    ConnectorSession,
)

with Client.from_discovery("https://cloud.h2o.ai", "my-workspace") as client:
    connector_svc = ConnectorServiceClient(client)
    conn_svc = ConnectionServiceClient(client)

    # 1. Create a connector
    connector_svc.create_connector("my-workspace", {
        "metadata": {"name": "my-pg"},
        "data_source_type": "postgresql",
        "data_source_config": {"host": "db.example.com", "port": "5432", "database": "mydb"},
    })

    # 2. Create a connection (worker must be pre-provisioned by an admin)
    connection = conn_svc.create_connection("my-workspace", {
        "connector": "workspaces/my-workspace/connectors/my-pg",
        "worker": "workspaces/my-workspace/workers/pg-worker",
        "extraction": {"query": "SELECT * FROM my_table"},
    })

    # 3. Wait for the worker pod and stream data
    session = ConnectorSession(client, "my-workspace", connection["connection_id"])
    session.wait_for_worker_ready(timeout=300)
    session.stream_to_csv("output.csv")

Optional Dependencies

Install extras for additional output format support:

pip install h2o-connector-service[pandas]       # pandas DataFrames
pip install h2o-connector-service[parquet]      # Parquet files (pyarrow)
pip install h2o-connector-service[datatable]    # datatable Frames
pip install h2o-connector-service[h2o]          # H2O Frames (pandas + pyarrow + h2o)

Supported Data Source Types

data_source_type	Display Name	Category	Worker Language
postgresql	PostgreSQL	Tabular	Go
snowflake	Snowflake	Tabular	Go
hive	Apache Hive	Tabular	Java
delta-lake	Delta Lake	Tabular	Rust
s3	Amazon S3	Blob	Go
gcs	Google Cloud Storage	Blob	Go
azure-blob	Azure Blob Storage	Blob	Go
minio	MinIO	Blob	Go