Hologres DataFrame API (Python)

A lazy, pandas-style DataFrame API for Hologres. Chain transformations in Python; the library compiles them into a single PostgreSQL statement and pushes it down to Hologres, so computation stays in the engine and only results come back.

Naming and execution model follow Snowpark for Python, with snake_case only (group_by, never groupBy) and PostgreSQL semantics.

Install

pip install --pre hologres-dataframe

The 0.1.0.dev0 pre-release requires hologres-client>=0.1.2.dev0, so the matching client pre-release must be available from the same package index.

For development from this repository:

pip install -e ".[dev]"

pandas and pyarrow are optional; they are only needed for to_pandas(), from_pandas() and local read_files().

Function Compute deployment is also optional:

pip install "hologres-dataframe[fc]"

Quickstart

import hologres.dataframe as hg

session = hg.connect(
    host="xxx.hologres.aliyuncs.com",
    port=80,
    dbname="my_db",
    user="<access_id>",
    password="<access_key>",
)

(
    session.table("public.orders")
    .filter(hg.col("amount") > 100)  # WHERE amount > 100
    .with_column("tax", hg.col("amount") * 0.06)  # amount * 0.06 AS tax
    .select("order_id", "region", "amount", "tax")
    .sort(hg.col("amount"), ascending=False)  # ORDER BY amount DESC
    .show(10)
)  # <- the only step that runs SQL

Everything before show() is lazy: transformations just build a logical plan. Actions (show, collect, to_pandas, count, first, take, iter_rows, write, write_files, as_dynamic_table) are what trigger execution. session.refresh_table() also executes immediately.

Like Snowpark, in-memory DataFrames should normally be created from a Session so the returned DataFrame already knows where actions and writes should execute:

df = session.from_dict({"id": [1, 2], "name": ["a", "b"]})
df.write("public.target")

The top-level hg.from_dict and hg.from_records functions remain available for unbound SQL construction and compatibility. The legacy top-level hg.from_pandas entry still accepts session=...; new code should use session.from_pandas(pdf). An unbound DataFrame can be compiled, but it cannot execute an action or write until it is created through a Session-bound entry point.

Materialize a lazy query as a Dynamic Table with safe create-if-absent behavior by default:

summary = session.table("public.orders").select("order_id", "region", "amount")
summary.as_dynamic_table(
    "public.dt_orders",
    freshness="10 minutes",
    refresh_mode="incremental",
    options={"cdc_format": "binlog"},
)
session.refresh_table("public.dt_orders")

Use mode="errorifexists" to reject an existing target or mode="replace" to drop and recreate it in one transaction. Query the current contents through session.table("public.dt_orders").

Request Serverless Computing for a query by marking the DataFrame before its action:

rows = (
    session.table("public.orders")
    .filter(hg.col("amount") > 100)
    .serverless(priority=4, required_cores=64, max_cores=128)
    .collect()
)

The marker is lazy and immutable. At execution time the client applies transaction-scoped SET LOCAL settings on the same connection as the query, so they do not leak through the connection pool. Hologres still decides whether the statement is eligible for Serverless execution; unsupported Serverless settings raise an error instead of silently falling back.

Function Compute UDFs

FC account/region settings belong to the Session. Alibaba Cloud credentials come from the standard credential chain; do not put AK secrets in decorators:

session = hg.connect(
    host="xxx.hologres.aliyuncs.com",
    port=80,
    dbname="my_db",
    user="<access_id>",
    password="<access_key>",
    fc_config={
        "endpoint": "123.cn-hangzhou-internal.fc.aliyuncs.com",
        "region": "cn-hangzhou",
        "role_arn": "acs:ram::123:role/fc-execution-role",
    },
)

Decorators only validate and retain Python definitions. The first action using the function packages its source/imports, creates or updates the FC function, and registers LANGUAGE function_compute in Hologres. Later actions in the same Session reuse the registration.

@hg.udf(packages=["numpy"])
def grade(n: int) -> str:
    return "high" if n >= 2 else "low"


scored = session.table("public.scores").select("id", grade("score").alias("grade"))

UDTFs are called directly in select; no lateral-join API is required:

@hg.udtf(
    output_schema=hg.StructType(
        [
            hg.StructField("word", hg.StringType()),
        ]
    )
)
class SplitWords:
    def process(self, text: str):
        for word in text.split():
            yield (word,)


words = session.table("public.docs").select("id", SplitWords("text"))

Hologres Remote UDX cannot return PostgreSQL record, so output_schema must contain exactly one field. FC endpoints must be internal endpoints in the same region as Hologres. Third-party packages are installed as Linux wheels for the selected runtime; package names without compatible wheels fail during deployment instead of producing an incompatible ZIP.

Runnable Examples

The examples directory contains scripts that connect directly to Hologres and exercise the implemented APIs:

• basic_query.py: filter, derive, project, and sort.
• join_and_aggregate.py: join and grouped aggregation inside Hologres.
• write_rows.py: append or UPSERT in-memory records into an existing table.
• oss_files.py: read and export OSS files through EXTERNAL_FILES.
• dynamic_table.py: create and refresh a Dynamic Table.

Connection values and example table names come from environment variables; see examples/README.md for the required schemas and commands.

Data types

Types are named after Snowpark/Spark and each knows the PostgreSQL type it compiles to:

StructType is a client-side row descriptor, not a column type. It is what df.schema returns and what @hg.udtf(output_schema=...) and session.read_files(schema=...) accept.

Packaging

This distribution is named hologres-dataframe and installs into the hologres namespace as hologres.dataframe, alongside hologres-client (../holo-client-py), which it depends on for high-throughput write channels.

Implementation status

Tests

python3 -m pytest        # line coverage gate of 80% is enforced via pytest.ini

Tests do not need a live Hologres instance: the compiler is a pure plan-to-SQL function and is verified by asserting on generated SQL, while the execution layer is covered with fake connections.