PyPaimon Rust
This project builds the Rust-powered core for PyPaimon while also providing DataFusion integration for querying Paimon tables.
Install via PyPI:
pip install pypaimon-rust
If you want to use the native Python DataFusion SessionContext, install datafusion as well.
Query Paimon Tables with DataFusion
The recommended way to query Paimon tables is through SQLContext, which supports
multi-catalog registration, DDL, DML, and all Paimon-specific SQL extensions:
from pypaimon_rust.datafusion import SQLContext
ctx = SQLContext()
ctx.register_catalog("paimon", {
"warehouse": "/path/to/warehouse",
})
# DDL
ctx.sql("CREATE SCHEMA paimon.my_db")
ctx.sql("CREATE TABLE paimon.my_db.users (id INT, name STRING, PRIMARY KEY (id))")
# DML
ctx.sql("INSERT INTO paimon.my_db.users VALUES (1, 'alice'), (2, 'bob')")
# Query tables via SQL (catalog.database.table)
batches = ctx.sql("SELECT * FROM paimon.my_db.users")
Blob Source Streams and Commit Identity
The write and update APIs accept a UriReaderFactory for custom Blob reference
sources. Both stream protocols are public runtime imports and support structural
typing; implementations do not need to inherit from them.
from io import BytesIO
from pypaimon_rust.datafusion import UriReader, UriReaderFactory
payloads = {"custom://example": b"content"}
class Reader:
def new_input_stream(self, uri: str):
return BytesIO(payloads[uri])
class Factory:
def create(self, uri: str) -> UriReader:
return Reader()
factory: UriReaderFactory = Factory()
builder = table.new_batch_write_builder().with_commit_user("my-job")
writer = builder.new_write().with_blob_uri_reader_factory(factory)
update = builder.new_update().with_blob_uri_reader_factory(factory)
Rust copies the requested Blob ranges and closes the source streams. Pass None
to clear the factory. Configure a writer before its first write. Batch and stream
updates, including incremental row-ID updaters, expose the same chainable setter.
with_commit_user is available on batch, stream, and postpone fixed bucket write
builders, and the identity applies to both writers and committers from that builder.
BucketRouter(table.schema()).route(batch) assigns fixed buckets in Rust and
returns an Arrow array without creating a writer. POSTPONE engines can use
PostponeBucketPlanner to collect mergeable per-partition statistics, infer a
shared plan, and configure that router with with_bucket_plan(plan). Collect
statistics across all input batches before planning; the planner stores schema
and counts only. Existing layouts are resolved separately by the write builder
using write authorization, without requiring SELECT.
Typed MERGE Actions
Batch and stream merge_into accept ordered WhenMatched and WhenNotMatched
objects. Assignments use MergeAssignment.source_column, target_column,
literal, or function. A function receives the selected clause's Arrow table
and returns an Array or ChunkedArray with the same number of rows.
Execution engines that already joined target and source data can reuse these types through the batch execution factories:
import pyarrow as pa
from pypaimon_rust.datafusion import MergeAssignment, MergeBatchExecution, WhenMatched
joined = pa.table({
"t._ROW_ID": [10, 11],
"t.value": pa.array([20, 30], type=pa.int32()),
"s.value": pa.array([40, 10], type=pa.int32()),
})
matched = [
WhenMatched.update(
{"value": MergeAssignment.source_column("value")},
condition='"s.value" > "t.value"',
),
WhenMatched.delete(),
]
update_schema = pa.schema([
pa.field("_ROW_ID", pa.int64(), nullable=False),
pa.field("value", pa.int32()),
])
updates = MergeBatchExecution.for_updates(joined.schema, update_schema, matched)
deletes = MergeBatchExecution.for_deletes(joined.schema, matched)
assert updates.execute(joined).to_pylist() == [{"_ROW_ID": 10, "value": 40}]
assert deletes.execute(joined).to_pylist() == [{"_ROW_ID": 11}]
Both matched factories consume the complete ordered branch list. The delete
factory skips UPDATE assignments and callbacks while preserving their conditions.
For unmatched source batches, use for_inserts(source_schema, target_schema, when_not_matched) with WhenNotMatched objects. The factories compile conditions
once and return an executor with execute(Table | RecordBatch) -> Table.
The calling engine owns joins, authorization and table writes.
Hybrid Search Phases
Hybrid queries can execute routes separately and then fuse their results:
hybrid = (table.new_hybrid_search_builder()
.add_vector_route("embedding", [1.0, 0.0], 20)
.with_limit(10))
routes = hybrid.route_builders()
results = [hybrid.to_route_result(route, route.search_builder().execute())
for route in routes]
ranked = hybrid.rank(results)
route_builders() checks read authorization and captures one snapshot for all
routes, including an empty view. Each route retains its search builder: repeated
search_builder() calls return the same object, and configuring it changes later
route.execute() calls. Returned builders retain their snapshot after later
commits. to_route_result() accepts scored
global results and normalizes empty unscored selections; nonempty unscored or
physical primary-key selections cannot participate in global row-ID fusion.
For externally evaluated scores, HybridSearchWeightedResult and
HybridSearchRanker.RRF, .WEIGHTED_SCORE, or .MRR provide the same fusion
without a table or query:
from pypaimon_rust.datafusion import (
GlobalIndexResult, HybridSearchRanker, HybridSearchWeightedResult,
)
hits = GlobalIndexResult.from_scored_row_ids({7: 0.9, 8: 0.5})
ranked = HybridSearchRanker.MRR.rank([HybridSearchWeightedResult(hits, 2.0)], 1)
assert ranked.row_ids() == {7: 2.0}
Rankers use Java's float32 scores, weights and arithmetic. A shared captured read context is retained only when every nonempty input belongs to the same table view. Fusing external scores or different views returns scores without attaching an unrelated table or snapshot.
Multimodal SQL Helpers
SQLContext also registers Python scalar UDFs for common BLOB media and vector
workflows. Install the optional media dependencies when you need image or video
decoding:
pip install "pypaimon-rust[video]"
batches = ctx.sql("""
SELECT
id,
media_info(content) AS info_json,
media_thumbnail(content, 160, 90) AS preview_png,
video_snapshot(content, 1000) AS frame_png
FROM paimon.my_db.assets
""")
For vector search, vector_from_json converts JSON-encoded embeddings into
Arrow List<Float32> values that can be used from a temporary table or subquery:
batches = ctx.sql("""
WITH queries AS (
SELECT id, vector_from_json(embedding_json) AS embedding
FROM paimon.my_db.query_embeddings
)
SELECT q.id AS query_id, r.id AS result_id
FROM queries q
CROSS JOIN LATERAL vector_search(
'paimon.my_db.items',
'embedding',
q.embedding,
10
) AS r
""")
Temporary Tables
You can register temporary in-memory tables programmatically. Names support the same resolution rules as SQL: bare names use the current catalog and database, partially qualified names use the current catalog, and fully qualified names specify catalog.database.table.
Register a single PyArrow RecordBatch as a temporary table:
import pyarrow as pa
batch = pa.record_batch([[1, 2], ["alice", "bob"]], names=["id", "name"])
ctx.register_batch("paimon.default.my_temp", batch)
batches = ctx.sql("SELECT * FROM paimon.default.my_temp")
# Drop it via SQL when no longer needed
ctx.sql("DROP TEMPORARY TABLE paimon.default.my_temp")
Alternatively, if you want to use the native Python DataFusion SessionContext,
install datafusion and register a PaimonCatalog:
from datafusion import SessionContext
from pypaimon_rust.datafusion import PaimonCatalog
catalog = PaimonCatalog({
"warehouse": "/path/to/warehouse",
})
ctx = SessionContext()
ctx.register_catalog_provider("paimon", catalog)
# Query tables via SQL (catalog.database.table)
df = ctx.sql("SELECT * FROM paimon.default.my_table LIMIT 10")
df.show()
REST Catalog
from datafusion import SessionContext
from pypaimon_rust.datafusion import PaimonCatalog
catalog = PaimonCatalog({
"metastore": "rest",
"uri": "http://localhost:8080",
"warehouse": "my_warehouse",
})
ctx = SessionContext()
ctx.register_catalog_provider("paimon", catalog)
REST catalogs also support branch management through Rust core:
identifier = ("db", "table")
catalog.create_branch(identifier, "empty-branch")
catalog.create_branch(identifier, "release-branch", from_tag="release-tag")
branches = catalog.list_branches(identifier)
release = catalog.get_table(("db", "table$branch_release-branch"))
catalog.fast_forward(identifier, "release-branch")
catalog.drop_branch(identifier, "empty-branch")
The server validates branch names and publishes metadata changes. Fast-forward
updates the identifier's current branch (main by default); it does not change
the source branch. These catalog methods currently require a REST catalog.
Other catalog implementations raise NotImplementedError.
Time travel queries are not supported in the Python binding at this time.
Metadata
Release files for pypaimon-rust 0.5.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| pypaimon_rust-0.5.0.tar.gz | 4.2 MB | Without provenance | Details for pypaimon_rust-0.5.0.tar.gz |
Built distributions (wheels)
| File | Actions Reset | |||
|---|---|---|---|---|
| pypaimon_rust-0.5.0-cp310-abi3-win_amd64.whl | CPython 3.10 | abi3 | Windows x86-64 | Details for pypaimon_rust-0.5.0-cp310-abi3-win_amd64.whl |
| pypaimon_rust-0.5.0-cp310-abi3-manylinux_2_28_aarch64.whl | CPython 3.10 | abi3 | Linux glibc 2.28+ ARM64 | Details for pypaimon_rust-0.5.0-cp310-abi3-manylinux_2_28_aarch64.whl |
| pypaimon_rust-0.5.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl | CPython 3.10 | abi3 | Linux glibc 2.17+ x86-64 | Details for pypaimon_rust-0.5.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
| pypaimon_rust-0.5.0-cp310-abi3-macosx_11_0_arm64.whl | CPython 3.10 | abi3 | macOS 11.0+ ARM64 | Details for pypaimon_rust-0.5.0-cp310-abi3-macosx_11_0_arm64.whl |
| pypaimon_rust-0.5.0-cp310-abi3-macosx_10_12_x86_64.whl | CPython 3.10 | abi3 | macOS 10.12+ x86-64 | Details for pypaimon_rust-0.5.0-cp310-abi3-macosx_10_12_x86_64.whl |
Total release size: 478.0 MB
Release files / pypaimon_rust-0.5.0.tar.gz
| Download URL | pypaimon_rust-0.5.0.tar.gz |
|---|---|
| Size | 4.2 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
0f379a97982f0c18c1d41d78fef911f86c25473fd226c655970007a6bfb5318c
|
|
BLAKE2b-256 checksum How to use checksums |
0b81e4bef5a9ca6cd6b848ddebc682384837bef9c48b5119c41b9e61b851beae
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Release files / pypaimon_rust-0.5.0-cp310-abi3-win_amd64.whl
| Download URL | pypaimon_rust-0.5.0-cp310-abi3-win_amd64.whl |
|---|---|
| Size | 89.6 MB |
| Tags | CPython 3.10 Windows x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
fd1a1a14c8f212516bd1d1a79d12fb4c85c35ebd95a57cb768e8ead8afbcb027
|
|
BLAKE2b-256 checksum How to use checksums |
7935384e2678932dedfd99696280829e767c676848611b48b8cee749a776dc8d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Release files / pypaimon_rust-0.5.0-cp310-abi3-manylinux_2_28_aarch64.whl
| Download URL | pypaimon_rust-0.5.0-cp310-abi3-manylinux_2_28_aarch64.whl |
|---|---|
| Size | 103.3 MB |
| Tags | CPython 3.10 Linux glibc 2.28+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
36ae66a482459d2edb5ad7f97726f80add925018ad41e520c4f6912dd596db07
|
|
BLAKE2b-256 checksum How to use checksums |
35a02e0c7afdd4c744e8a78644395855a5107619ed1bc9ac83a7484f593ffedb
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Release files / pypaimon_rust-0.5.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | pypaimon_rust-0.5.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 99.6 MB |
| Tags | CPython 3.10 Linux glibc 2.17+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
f31d8053ad199f02df5bed55a5252b1aa9bd0b3dd2b6d563160a4a34c91098ab
|
|
BLAKE2b-256 checksum How to use checksums |
19a990e3524c6f7cb14a7a23d58e92f3cc3fd23bcea6e9fa6e032a1a1d642ffb
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Release files / pypaimon_rust-0.5.0-cp310-abi3-macosx_11_0_arm64.whl
| Download URL | pypaimon_rust-0.5.0-cp310-abi3-macosx_11_0_arm64.whl |
|---|---|
| Size | 88.6 MB |
| Tags | CPython 3.10 abi3 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
644f1a32029639583508fd53cb76d914ae0e49c2060ea24206b26c234567163d
|
|
BLAKE2b-256 checksum How to use checksums |
fbfbef17448ccc37400bc3654d82b7ddf092ff182dede5a29feb8f0895bc5aaa
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Release files / pypaimon_rust-0.5.0-cp310-abi3-macosx_10_12_x86_64.whl
| Download URL | pypaimon_rust-0.5.0-cp310-abi3-macosx_10_12_x86_64.whl |
|---|---|
| Size | 92.7 MB |
| Tags | CPython 3.10 abi3 macOS 10.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
e32aa9c4a50b1a97bd54771b3bdef3a48e480036c9a8c490f9a0c2d19ad57c96
|
|
BLAKE2b-256 checksum How to use checksums |
95218da599c6a3cf3485021783f21c447566956946b1c2a928e9bd89295521bd
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Security for pypaimon-rust 0.5.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.