deltacat-io-core

deltacat-io-core is the shared local execution layer for DeltaCAT reads and writes.

It is used by both:

• deltacat-client for direct thin-plan execution
• deltacat for shared local execution and compatibility wrappers

Naming

• distribution/package name: deltacat-io-core
• Python import module: deltacat_io_core

The distribution uses dashes for consistency with deltacat-client. The import module keeps underscores because Python module names cannot contain -.

Scope

deltacat-io-core owns the code that should behave the same regardless of whether the caller is using the thin client or the thick DeltaCAT package.

Today that includes:

• direct execution of thin Plan objects
• MOR execution for thin and thick paths
• local file materialization and manifest building
• bounded, authority-bound recovery-frontier index construction and scanning
• schema alignment and table conversion helpers
• sort-aware file ordering and manifest handling
• shared compaction/MOR helper layers and model types
• format-specific local readers/writers

Non-Goals

deltacat-io-core does not own:

• server routes or REST/MCP request handling
• authoritative catalog/storage mutations
• native Ray job orchestration surfaces
• public end-user API shape for deltacat or deltacat-client

It is a shared implementation layer, not the top-level user product.

Architecture

The current read architecture is:

1. The server resolves a thin Plan.
2. client.catalog.read(plan=...) executes that plan directly through deltacat-io-core.
3. dc.read_table(plan=...) for thin plans also executes through the same shared path.

There is no longer a runtime bridge back into thick DeltaCAT for thin plan execution. The plan contract is expected to carry the metadata required for direct execution.

The current write architecture is:

1. The client stages local files or materializes local data through shared helpers.
2. The authoritative commit still happens through DeltaCAT server/native boundaries.
3. Shared write-preparation and manifest logic lives in deltacat-io-core.

Installation

Base install:

uv pip install deltacat-io-core

Optional extras:

• deltacat-io-core[io] for local file readers/writers (pyarrow, fastavro)
• deltacat-io-core[pandas] for Pandas conversions
• deltacat-io-core[polars] for Polars conversions and lazy scan helpers
• deltacat-io-core[daft] for Daft conversions and lazy scan helpers
• deltacat-io-core[lance] for Lance dataset support
• deltacat-io-core[umi] for aligned-Zarr UMI input and LeRobotV2 video output
• deltacat-io-core[all] for the full local IO stack

Read Capabilities

The shared read executor currently handles:

• schema-table reads
• schemaless manifest-table reads
• MOR reads
• direct pyarrow, pandas, polars, numpy, daft, and ray_dataset outputs where supported
• lazy pyarrow_parquet
• lazy lance

It also enforces direct validation for unsupported combinations, for example:

• schemaless + pyarrow_parquet
• schemaless + lance
• mixed-content lazy plans for format-specific readers
• unknown content types in the shared path

Polars / Daft Capability Matrix

The shared executor applies the same capability decision in thin execute_read_plan(...) execution and in thick reads that delegate into that shared path.

Notes:

• Mixed-schema lazy eligibility on the shared path requires per-file schema_id lookups plus top-level schema information with resolvable field types, whether that comes from schema_serialized or a typed top-level schema summary.
• JSON schema summaries and requests share the canonical datatype codec in schema_types.py. Readable scalar names follow PyArrow (float64, not PyArrow's double display text); parameter-rich types use a lossless one-field Arrow IPC token. Request parsing is strict, while response/plan decoding narrowly tolerates the five scalar spellings emitted by older DeltaCAT servers so rolling upgrades and durable plans remain readable.
• On the shared Daft path, non-identity Parquet content encodings (for example .parquet.gz) stay on the eager PyArrow path.
• When the process is pinned to DAFT_RUNNER=ray, the shared local Daft lazy path declines and falls back to the eager shared path instead of spawning a Ray-backed local lazy scan.

Write Capabilities

The shared write layer currently covers:

• write input normalization
• local data materialization
• manifest construction for existing files and datasets
• schema/read compatibility helpers
• standard catalog write orchestration slices

Authoritative catalog mutation, commit, retention, and compaction boundaries still remain on the native/server side where they belong.

Long-running Lance writes

write_lance_batch_stream accepts an optional refresh_access_ref callback from an existing scoped credential owner. The shared credential factory checks the original binding/scope/routing and provider intent on each refresh and emits an engine refresh deadline before actual expiry. The owner's existing lease cache keeps vending independent of fragment count. A failed refresh is retained for that write attempt so the engine cannot amplify it into repeated credential requests; job-level retry supplies a fresh stream and output path.

The writer uses Lance's public namespace interface only as a fixed-path credential adapter (namespace on pylance3, namespace_client on pylance11). It does not create catalog namespaces, implement a second writer, or retry/delete partially consumed streams. Unsupported writers refuse before stream consumption. Nonrenewing and local callers retain the ordinary write path.

open_prepared_lance_dataset accepts the same scoped access_ref and optional refresh_access_ref pair for native in-flight reads. It supplies initial routing options as well as the native provider: the callback alone is insufficient for Lance's bootstrap. Pylance11 uses the same fixed-path namespace adapter for reads instead of3's direct provider argument. Both generated-model bootstrap requests and mapping-shaped native refresh requests validate the fixed ID/path. Unsupported reader interfaces refuse before opening storage. Earlier source qualification described the3.0.0 environment as11.0.0 incorrectly; actual engine versions and installed wheel qualification must be recorded independently.

storage_read_support.EntryReadAuthority reuses RenewingStorageAuthority for one job's manifest reads. Path/revision validation precedes lease reuse; at most 128 scoped bindings are retained. The owner keeps an independent copy of each original binding policy and refuses conflicting policies for that same scope. Credential refresh uses the original policy, not a mutable current revision. Fresh reads share one acquisition per binding/scope; a failed acquisition is retained for the attempt, including concurrent native callbacks. No catalog RPC or global credential cache is introduced. Training inventory, native payload and media copies share that owner. Local HTTP controls cover 8/32 independent datasets, in-flight fragment rotation, and zero extra storage requests while a lease remains fresh. This is not a claim of in-flight renewal for every engine or already-open Parquet/stream handles, nor live AWS/Alpha qualification. An already-open Parquet handle retains its original credentials. Real HTTP tests over 8/32 row groups deliberately expire those credentials after the footer GET: the scan fails, and reopening the retained file with a fresh read owner recovers the exact rows/schema using the original binding revision. A changed current revision is not substituted. This is read-attempt recovery, not transparent mid-handle renewal or proof of whole-job recovery for every consumer.

Recovery-frontier indexes

AuthoritativeFrontierIndexWriter consumes exact, authenticated state-census Arrow batches and produces one immutable identity index. The index is bound to the census checkpoint URI, VersionId, payload digest/size/content type, manifest digest, catalog root, protocol, and hold fence. It rejects marker filenames, malformed or stale state identities, duplicate/out-of-order census rows, and mismatched cardinality.

ExactVersionFrontierIndexScan implements the FrontierScan contract over that index. It requires a binding-scoped StorageAccessRef, reads the exact S3 VersionId once, verifies the byte count and SHA-256 before exposing rows, and streams Arrow batches from bounded local disk. Its logical remote-call count is one regardless of census or frontier cardinality. Callers must continue to run LeaseBatchPrecondition before every remote mutation/publication batch.

Retirement-cohort authorities

write_authoritative_retirement_cohort streams a sorted exact membership set and a complete frozen transaction-success census into one immutable Arrow file. RetirementCohortIndex memory-maps that file and performs exact binary search over bounded record batches; a lookup is O(log N) and never scans or materializes the full cohort as Python rows. Its monotonic-successor proof streams both sorted sections in fixed Arrow chunks, proving member and success set inclusion in O(M + S) time and bounded memory.

RetirementStateEvidence carries only state already authenticated by the caller: member identity, normalized status, explicit commit scope, and at most one direct transaction parent. Resolution is pure and depth-one. Unbound sessions resolve to reversible abort; applied state with complete success identity resolves commit-forward. Unknown, contradictory, self-referential, or unresolved state fails closed.

Relationship To Other Packages

Use deltacat-client when you want the public thin client.

Use deltacat when you want the thick/native package.

Use deltacat-io-core directly only if you are intentionally building against the shared execution layer itself.