Skip to main content

PyPaimon Rust

This project builds the Rust-powered core for PyPaimon while also providing DataFusion integration for querying Paimon tables.

Install via PyPI:

pip install pypaimon-rust

If you want to use the native Python DataFusion SessionContext, install datafusion as well.

Query Paimon Tables with DataFusion

The recommended way to query Paimon tables is through SQLContext, which supports multi-catalog registration, DDL, DML, and all Paimon-specific SQL extensions:

from pypaimon_rust.datafusion import SQLContext

ctx = SQLContext()
ctx.register_catalog("paimon", {
    "warehouse": "/path/to/warehouse",
})

# DDL
ctx.sql("CREATE SCHEMA paimon.my_db")
ctx.sql("CREATE TABLE paimon.my_db.users (id INT, name STRING, PRIMARY KEY (id))")

# DML
ctx.sql("INSERT INTO paimon.my_db.users VALUES (1, 'alice'), (2, 'bob')")

# Query tables via SQL (catalog.database.table)
batches = ctx.sql("SELECT * FROM paimon.my_db.users")

Blob Source Streams and Commit Identity

The write and update APIs accept a UriReaderFactory for custom Blob reference sources. Both stream protocols are public runtime imports and support structural typing; implementations do not need to inherit from them.

from io import BytesIO
from pypaimon_rust.datafusion import UriReader, UriReaderFactory

payloads = {"custom://example": b"content"}

class Reader:
    def new_input_stream(self, uri: str):
        return BytesIO(payloads[uri])

class Factory:
    def create(self, uri: str) -> UriReader:
        return Reader()

factory: UriReaderFactory = Factory()
builder = table.new_batch_write_builder().with_commit_user("my-job")
writer = builder.new_write().with_blob_uri_reader_factory(factory)
update = builder.new_update().with_blob_uri_reader_factory(factory)

Rust copies the requested Blob ranges and closes the source streams. Pass None to clear the factory. Configure a writer before its first write. Batch and stream updates, including incremental row-ID updaters, expose the same chainable setter. with_commit_user is available on batch, stream, and postpone fixed bucket write builders, and the identity applies to both writers and committers from that builder.

BucketRouter(table.schema()).route(batch) assigns fixed buckets in Rust and returns an Arrow array without creating a writer. POSTPONE engines can use PostponeBucketPlanner to collect mergeable per-partition statistics, infer a shared plan, and configure that router with with_bucket_plan(plan). Collect statistics across all input batches before planning; the planner stores schema and counts only. Existing layouts are resolved separately by the write builder using write authorization, without requiring SELECT.

Typed MERGE Actions

Batch and stream merge_into accept ordered WhenMatched and WhenNotMatched objects. Assignments use MergeAssignment.source_column, target_column, literal, or function. A function receives the selected clause's Arrow table and returns an Array or ChunkedArray with the same number of rows.

Execution engines that already joined target and source data can reuse these types through the batch execution factories:

import pyarrow as pa
from pypaimon_rust.datafusion import MergeAssignment, MergeBatchExecution, WhenMatched

joined = pa.table({
    "t._ROW_ID": [10, 11],
    "t.value": pa.array([20, 30], type=pa.int32()),
    "s.value": pa.array([40, 10], type=pa.int32()),
})
matched = [
    WhenMatched.update(
        {"value": MergeAssignment.source_column("value")},
        condition='"s.value" > "t.value"',
    ),
    WhenMatched.delete(),
]
update_schema = pa.schema([
    pa.field("_ROW_ID", pa.int64(), nullable=False),
    pa.field("value", pa.int32()),
])
updates = MergeBatchExecution.for_updates(joined.schema, update_schema, matched)
deletes = MergeBatchExecution.for_deletes(joined.schema, matched)
assert updates.execute(joined).to_pylist() == [{"_ROW_ID": 10, "value": 40}]
assert deletes.execute(joined).to_pylist() == [{"_ROW_ID": 11}]

Both matched factories consume the complete ordered branch list. The delete factory skips UPDATE assignments and callbacks while preserving their conditions. For unmatched source batches, use for_inserts(source_schema, target_schema, when_not_matched) with WhenNotMatched objects. The factories compile conditions once and return an executor with execute(Table | RecordBatch) -> Table. The calling engine owns joins, authorization and table writes.

Hybrid Search Phases

Hybrid queries can execute routes separately and then fuse their results:

hybrid = (table.new_hybrid_search_builder()
          .add_vector_route("embedding", [1.0, 0.0], 20)
          .with_limit(10))
routes = hybrid.route_builders()
results = [hybrid.to_route_result(route, route.search_builder().execute())
           for route in routes]
ranked = hybrid.rank(results)

route_builders() checks read authorization and captures one snapshot for all routes, including an empty view. Each route retains its search builder: repeated search_builder() calls return the same object, and configuring it changes later route.execute() calls. Returned builders retain their snapshot after later commits. to_route_result() accepts scored global results and normalizes empty unscored selections; nonempty unscored or physical primary-key selections cannot participate in global row-ID fusion.

For externally evaluated scores, HybridSearchWeightedResult and HybridSearchRanker.RRF, .WEIGHTED_SCORE, or .MRR provide the same fusion without a table or query:

from pypaimon_rust.datafusion import (
    GlobalIndexResult, HybridSearchRanker, HybridSearchWeightedResult,
)

hits = GlobalIndexResult.from_scored_row_ids({7: 0.9, 8: 0.5})
ranked = HybridSearchRanker.MRR.rank([HybridSearchWeightedResult(hits, 2.0)], 1)
assert ranked.row_ids() == {7: 2.0}

Rankers use Java's float32 scores, weights and arithmetic. A shared captured read context is retained only when every nonempty input belongs to the same table view. Fusing external scores or different views returns scores without attaching an unrelated table or snapshot.

Multimodal SQL Helpers

SQLContext also registers Python scalar UDFs for common BLOB media and vector workflows. Install the optional media dependencies when you need image or video decoding:

pip install "pypaimon-rust[video]"
batches = ctx.sql("""
    SELECT
        id,
        media_info(content) AS info_json,
        media_thumbnail(content, 160, 90) AS preview_png,
        video_snapshot(content, 1000) AS frame_png
    FROM paimon.my_db.assets
""")

For vector search, vector_from_json converts JSON-encoded embeddings into Arrow List<Float32> values that can be used from a temporary table or subquery:

batches = ctx.sql("""
    WITH queries AS (
        SELECT id, vector_from_json(embedding_json) AS embedding
        FROM paimon.my_db.query_embeddings
    )
    SELECT q.id AS query_id, r.id AS result_id
    FROM queries q
    CROSS JOIN LATERAL vector_search(
        'paimon.my_db.items',
        'embedding',
        q.embedding,
        10
    ) AS r
""")

Temporary Tables

You can register temporary in-memory tables programmatically. Names support the same resolution rules as SQL: bare names use the current catalog and database, partially qualified names use the current catalog, and fully qualified names specify catalog.database.table.

Register a single PyArrow RecordBatch as a temporary table:

import pyarrow as pa

batch = pa.record_batch([[1, 2], ["alice", "bob"]], names=["id", "name"])

ctx.register_batch("paimon.default.my_temp", batch)

batches = ctx.sql("SELECT * FROM paimon.default.my_temp")

# Drop it via SQL when no longer needed
ctx.sql("DROP TEMPORARY TABLE paimon.default.my_temp")

Alternatively, if you want to use the native Python DataFusion SessionContext, install datafusion and register a PaimonCatalog:

from datafusion import SessionContext
from pypaimon_rust.datafusion import PaimonCatalog

catalog = PaimonCatalog({
    "warehouse": "/path/to/warehouse",
})

ctx = SessionContext()
ctx.register_catalog_provider("paimon", catalog)

# Query tables via SQL (catalog.database.table)
df = ctx.sql("SELECT * FROM paimon.default.my_table LIMIT 10")
df.show()

REST Catalog

from datafusion import SessionContext
from pypaimon_rust.datafusion import PaimonCatalog

catalog = PaimonCatalog({
    "metastore": "rest",
    "uri": "http://localhost:8080",
    "warehouse": "my_warehouse",
})

ctx = SessionContext()
ctx.register_catalog_provider("paimon", catalog)

REST catalogs also support branch management through Rust core:

identifier = ("db", "table")
catalog.create_branch(identifier, "empty-branch")
catalog.create_branch(identifier, "release-branch", from_tag="release-tag")
branches = catalog.list_branches(identifier)
release = catalog.get_table(("db", "table$branch_release-branch"))
catalog.fast_forward(identifier, "release-branch")
catalog.drop_branch(identifier, "empty-branch")

The server validates branch names and publishes metadata changes. Fast-forward updates the identifier's current branch (main by default); it does not change the source branch. These catalog methods currently require a REST catalog. Other catalog implementations raise NotImplementedError.

Time travel queries are not supported in the Python binding at this time.

Metadata

Release files for pypaimon-rust 0.5.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pypaimon-rust 0.5.0
File Size Uploaded Provenance Actions
pypaimon_rust-0.5.0.tar.gz 4.2 MB Without provenance Details for pypaimon_rust-0.5.0.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for pypaimon-rust 0.5.0
File Actions
pypaimon_rust-0.5.0-cp310-abi3-win_amd64.whl CPython 3.10 abi3 Windows x86-64 Details for pypaimon_rust-0.5.0-cp310-abi3-win_amd64.whl
pypaimon_rust-0.5.0-cp310-abi3-manylinux_2_28_aarch64.whl CPython 3.10 abi3 Linux glibc 2.28+ ARM64 Details for pypaimon_rust-0.5.0-cp310-abi3-manylinux_2_28_aarch64.whl
pypaimon_rust-0.5.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl CPython 3.10 abi3 Linux glibc 2.17+ x86-64 Details for pypaimon_rust-0.5.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
pypaimon_rust-0.5.0-cp310-abi3-macosx_11_0_arm64.whl CPython 3.10 abi3 macOS 11.0+ ARM64 Details for pypaimon_rust-0.5.0-cp310-abi3-macosx_11_0_arm64.whl
pypaimon_rust-0.5.0-cp310-abi3-macosx_10_12_x86_64.whl CPython 3.10 abi3 macOS 10.12+ x86-64 Details for pypaimon_rust-0.5.0-cp310-abi3-macosx_10_12_x86_64.whl

Total release size: 478.0 MB

Release files / pypaimon_rust-0.5.0.tar.gz

Download URL pypaimon_rust-0.5.0.tar.gz
Size 4.2 MB
Tags Source
SHA-256 checksum
How to use checksums
0f379a97982f0c18c1d41d78fef911f86c25473fd226c655970007a6bfb5318c
BLAKE2b-256 checksum
How to use checksums
0b81e4bef5a9ca6cd6b848ddebc682384837bef9c48b5119c41b9e61b851beae
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.16

Release files / pypaimon_rust-0.5.0-cp310-abi3-win_amd64.whl

Download URL pypaimon_rust-0.5.0-cp310-abi3-win_amd64.whl
Size 89.6 MB
Tags CPython 3.10 Windows x86-64 abi3
SHA-256 checksum
How to use checksums
fd1a1a14c8f212516bd1d1a79d12fb4c85c35ebd95a57cb768e8ead8afbcb027
BLAKE2b-256 checksum
How to use checksums
7935384e2678932dedfd99696280829e767c676848611b48b8cee749a776dc8d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.16

Release files / pypaimon_rust-0.5.0-cp310-abi3-manylinux_2_28_aarch64.whl

Download URL pypaimon_rust-0.5.0-cp310-abi3-manylinux_2_28_aarch64.whl
Size 103.3 MB
Tags CPython 3.10 Linux glibc 2.28+ ARM64 abi3
SHA-256 checksum
How to use checksums
36ae66a482459d2edb5ad7f97726f80add925018ad41e520c4f6912dd596db07
BLAKE2b-256 checksum
How to use checksums
35a02e0c7afdd4c744e8a78644395855a5107619ed1bc9ac83a7484f593ffedb
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.16

Release files / pypaimon_rust-0.5.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Download URL pypaimon_rust-0.5.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Size 99.6 MB
Tags CPython 3.10 Linux glibc 2.17+ x86-64 abi3
SHA-256 checksum
How to use checksums
f31d8053ad199f02df5bed55a5252b1aa9bd0b3dd2b6d563160a4a34c91098ab
BLAKE2b-256 checksum
How to use checksums
19a990e3524c6f7cb14a7a23d58e92f3cc3fd23bcea6e9fa6e032a1a1d642ffb
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.16

Release files / pypaimon_rust-0.5.0-cp310-abi3-macosx_11_0_arm64.whl

Download URL pypaimon_rust-0.5.0-cp310-abi3-macosx_11_0_arm64.whl
Size 88.6 MB
Tags CPython 3.10 abi3 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
644f1a32029639583508fd53cb76d914ae0e49c2060ea24206b26c234567163d
BLAKE2b-256 checksum
How to use checksums
fbfbef17448ccc37400bc3654d82b7ddf092ff182dede5a29feb8f0895bc5aaa
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.16

Release files / pypaimon_rust-0.5.0-cp310-abi3-macosx_10_12_x86_64.whl

Download URL pypaimon_rust-0.5.0-cp310-abi3-macosx_10_12_x86_64.whl
Size 92.7 MB
Tags CPython 3.10 abi3 macOS 10.12+ x86-64
SHA-256 checksum
How to use checksums
e32aa9c4a50b1a97bd54771b3bdef3a48e480036c9a8c490f9a0c2d19ad57c96
BLAKE2b-256 checksum
How to use checksums
95218da599c6a3cf3485021783f21c447566956946b1c2a928e9bd89295521bd
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.16

Security for pypaimon-rust 0.5.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page