deltacat-io-core

deltacat-io-core is the shared local execution layer for DeltaCAT reads and writes.

It is used by both:

• deltacat-client for direct thin-plan execution
• deltacat for shared local execution and compatibility wrappers

Naming

• distribution/package name: deltacat-io-core
• Python import module: deltacat_io_core

The distribution uses dashes for consistency with deltacat-client. The import module keeps underscores because Python module names cannot contain -.

Scope

deltacat-io-core owns the code that should behave the same regardless of whether the caller is using the thin client or the thick DeltaCAT package.

Today that includes:

• direct execution of thin Plan objects
• MOR execution for thin and thick paths
• local file materialization and manifest building
• bounded, authority-bound recovery-frontier index construction and scanning
• schema alignment and table conversion helpers
• sort-aware file ordering and manifest handling
• shared compaction/MOR helper layers and model types
• format-specific local readers/writers

Non-Goals

deltacat-io-core does not own:

• server routes or REST/MCP request handling
• authoritative catalog/storage mutations
• native Ray job orchestration surfaces
• public end-user API shape for deltacat or deltacat-client

It is a shared implementation layer, not the top-level user product.

Architecture

The current read architecture is:

1. The server resolves a thin Plan.
2. client.catalog.read(plan=...) executes that plan directly through deltacat-io-core.
3. dc.read_table(plan=...) for thin plans also executes through the same shared path.

There is no longer a runtime bridge back into thick DeltaCAT for thin plan execution. The plan contract is expected to carry the metadata required for direct execution.

The current write architecture is:

1. The client stages local files or materializes local data through shared helpers.
2. The authoritative commit still happens through DeltaCAT server/native boundaries.
3. Shared write-preparation and manifest logic lives in deltacat-io-core.

Installation

Base install:

uv pip install deltacat-io-core

Optional extras:

• deltacat-io-core[io] for local file readers/writers (pyarrow, fastavro)
• deltacat-io-core[pandas] for Pandas conversions
• deltacat-io-core[polars] for Polars conversions and lazy scan helpers
• deltacat-io-core[daft] for Daft conversions and lazy scan helpers
• deltacat-io-core[lance] for Lance dataset support
• deltacat-io-core[all] for the full local IO stack

Read Capabilities

The shared read executor currently handles:

• schema-table reads
• schemaless manifest-table reads
• MOR reads
• direct pyarrow, pandas, polars, numpy, daft, and ray_dataset outputs where supported
• lazy pyarrow_parquet
• lazy lance

It also enforces direct validation for unsupported combinations, for example:

• schemaless + pyarrow_parquet
• schemaless + lance
• mixed-content lazy plans for format-specific readers
• unknown content types in the shared path

Polars / Daft Capability Matrix

The shared executor applies the same capability decision in thin execute_read_plan(...) execution and in thick reads that delegate into that shared path.

Notes:

• Mixed-schema lazy eligibility on the shared path requires per-file schema_id lookups plus top-level schema information with resolvable field types, whether that comes from schema_serialized or a typed top-level schema summary.
• On the shared Daft path, non-identity Parquet content encodings (for example .parquet.gz) stay on the eager PyArrow path.
• When the process is pinned to DAFT_RUNNER=ray, the shared local Daft lazy path declines and falls back to the eager shared path instead of spawning a Ray-backed local lazy scan.

Write Capabilities

The shared write layer currently covers:

• write input normalization
• local data materialization
• manifest construction for existing files and datasets
• schema/read compatibility helpers
• standard catalog write orchestration slices

Authoritative catalog mutation, commit, retention, and compaction boundaries still remain on the native/server side where they belong.

Recovery-frontier indexes

AuthoritativeFrontierIndexWriter consumes exact, authenticated state-census Arrow batches and produces one immutable identity index. The index is bound to the census checkpoint URI, VersionId, payload digest/size/content type, manifest digest, catalog root, protocol, and hold fence. It rejects marker filenames, malformed or stale state identities, duplicate/out-of-order census rows, and mismatched cardinality.

ExactVersionFrontierIndexScan implements the FrontierScan contract over that index. It requires a binding-scoped StorageAccessRef, reads the exact S3 VersionId once, verifies the byte count and SHA-256 before exposing rows, and streams Arrow batches from bounded local disk. Its logical remote-call count is one regardless of census or frontier cardinality. Callers must continue to run LeaseBatchPrecondition before every remote mutation/publication batch.

Retirement-cohort authorities

write_authoritative_retirement_cohort streams a sorted exact membership set and a complete frozen transaction-success census into one immutable Arrow file. RetirementCohortIndex memory-maps that file and performs exact binary search over bounded record batches; a lookup is O(log N) and never scans or materializes the full cohort as Python rows. Its monotonic-successor proof streams both sorted sections in fixed Arrow chunks, proving member and success set inclusion in O(M + S) time and bounded memory.

RetirementStateEvidence carries only state already authenticated by the caller: member identity, normalized status, explicit commit scope, and at most one direct transaction parent. Resolution is pure and depth-one. Unbound sessions resolve to reversible abort; applied state with complete success identity resolves commit-forward. Unknown, contradictory, self-referential, or unresolved state fails closed.

Relationship To Other Packages

Use deltacat-client when you want the public thin client.

Use deltacat when you want the thick/native package.

Use deltacat-io-core directly only if you are intentionally building against the shared execution layer itself.