deltacat-io-core

deltacat-io-core is the shared local execution layer for DeltaCAT reads and writes.

It is used by both:

• deltacat-client for direct thin-plan execution
• deltacat for shared local execution and compatibility wrappers

Naming

• distribution/package name: deltacat-io-core
• Python import module: deltacat_io_core

The distribution uses dashes for consistency with deltacat-client. The import module keeps underscores because Python module names cannot contain -.

Scope

deltacat-io-core owns the code that should behave the same regardless of whether the caller is using the thin client or the thick DeltaCAT package.

Today that includes:

• direct execution of thin Plan objects
• MOR execution for thin and thick paths
• local file materialization and manifest building
• bounded, authority-bound recovery-frontier index construction and scanning
• schema alignment and table conversion helpers
• sort-aware file ordering and manifest handling
• shared compaction/MOR helper layers and model types
• format-specific local readers/writers

Non-Goals

deltacat-io-core does not own:

• server routes or REST/MCP request handling
• authoritative catalog/storage mutations
• native Ray job orchestration surfaces
• public end-user API shape for deltacat or deltacat-client

It is a shared implementation layer, not the top-level user product.

For bounded nested-reference enumeration shared by admission callers, see source dependency planning. Its PackDS adapter reuses the native reference decoder and returns exact object/binding metadata without reading media or acquiring storage capabilities.

Architecture

The current read architecture is:

1. The server resolves a thin Plan.
2. client.catalog.read(plan=...) executes that plan directly through deltacat-io-core.
3. dc.read_table(plan=...) for thin plans also executes through the same shared path.

There is no longer a runtime bridge back into thick DeltaCAT for thin plan execution. The plan contract is expected to carry the metadata required for direct execution.

The current write architecture is:

1. The client stages local files or materializes local data through shared helpers.
2. The authoritative commit still happens through DeltaCAT server/native boundaries.
3. Shared write-preparation and manifest logic lives in deltacat-io-core.

Installation

Base install:

uv pip install deltacat-io-core

Optional extras:

• deltacat-io-core[io] for local file readers/writers (pyarrow, fastavro)
• deltacat-io-core[pandas] for Pandas conversions
• deltacat-io-core[polars] for Polars conversions and lazy scan helpers
• deltacat-io-core[daft] for Daft conversions and lazy scan helpers
• deltacat-io-core[lance] for Lance dataset support
• deltacat-io-core[umi] for aligned-Zarr UMI input and LeRobotV2 video output
• deltacat-io-core[all] for the full local IO stack

Request-owned S3 clients

Storage clients must be created from a complete authority-vetted StorageAccessRef, through storage_credentials. The default factory reuses its authority-scoped cache. Callers that need isolated request hooks can opt in to create_boto3_s3_client_for_access_with_retirement_token(..., request_private=True), which bypasses the shared client cache and returns an owned client and retirement token. The bounded authority snapshot retains all fields, including credential-source/profile intent; no ambient fallback is added.

borrow_private_boto3_s3_client(client, token) yields the concrete SDK client and frozen lease facts. It requires the exact live private token in its creator process and captured explicit credentials with sufficient expiry headroom. The additional borrow prevents parent-token retirement from closing the client mid-request. Install hooks before use, retain the scope through response-body drain, and release the original retirement token when the request owner is done. Facts are process-local, not a transferable grant. This helper supplies no request budgets, locality proof, auth-body decoder or automatic replay.

Changed/expired authority holds that capability: leave and drain the scope, then obtain fresh exact authority outside it within the original deadline. Transient SDK errors propagate to the caller's existing recovery policy; neither credentials nor clients refresh inside the concrete borrow. Production callers remain on their existing route until their request accounting and authority integration are qualified separately.

Observing an existing concrete SDK receiver

run_boto3_s3_read_once and run_boto3_s3_mutation_once accept the optional client_observer_scope(actual_client) context factory. Its default is None. For renewable proxies, it enters after native acquisition and explicit-lease headroom checks, before model/dispatch guards, and remains active through the read consumer and body drains. Each acquisition observes its actual current SDK client, including a replacement generation; attaching to proxy.meta before acquisition is not equivalent. The scope spans existing bounded read attempts; mutation calls still never replay.

For a caller that already owns a retained HEAD/MPU/readback lifetime, borrow_boto3_s3_observed_client(client, operation=..., client_observer_scope=...) yields that actual receiver under the proxy's native active-borrow accounting. Keep it in the existing ExitStack until all bodies have drained. Observer exit precedes release; it cannot suppress or replace a primary SDK/body error. There is no refresh inside this retained borrow and no new private token or replacement client. A directly supplied SDK client remains owned by its original owner and is not closed on scope exit.

This is an observation seam, not authority, private event-hook ownership or a new retry controller. Callers retain their exact access-ref expiry, deadline, scope, attempt-budget and concurrency-isolation checks. In particular an observer on a shared client must isolate its own concurrent operation hooks. On headroom refusal, leave/drain and obtain fresh authority through the owner outside the scope. No writer/relay consumer is activated by this helper alone.

Private read attempt and XML observation

The unused s3_read_observation.observe_private_s3_read helper composes the factory's genuine private-token borrow with temporary SDK hooks. Required admit_attempt(operation, prepared_request) and check() callbacks belong to the request owner: retain exact key/range/IfMatch/version/auth scope, shared attempt accounting and the original claim/deadline. No callback or policy default admits requests. Hooks observe each native GetObject/CreateSession before-send, including retries, redirects and hidden Express authentication. These are admissions, not proof that network bytes were sent. The helper does not change SDK retry configuration or activate any read/writer consumer. An owner callback failure is kept outside the SDK retry classifier and its identical original exception is restored when leaving the scope. Catch errors outside that scope; swallowing its internal refusal cannot permit another admission or turn scope exit into success. Real SDK transport failures still use the SDK's configured retry policy. An owner refusal after a response arrives uses that same latch and closes any unreturned streaming body before releasing decoder workspace. A secondary close failure adds only a static note to the original refusal; successful bodies are left untouched for the caller to drain.

An explicit S3XmlLimits bounds arrived body bytes, elements, depth, attributes and UTF-8 text before the real REST-XML parser runs. A SAX Expat guard produces no credential mapping or DOM and rejects DTD/entity input. Headers-only errors remain available to native SDK redirect/retry handling. The raw body has already arrived; this is not a pre-wire allocation or hard RSS bound. The owner supplies decoder_workspace() covering both guard and SDK parsing for the whole scope. Guard messages contain no body values; this does not redact independent SDK debug logging or authorize logging credentials.

Keep the scope alive until all response bodies are consumed/closed, with the same pending-byte/FD owner. It pins the concrete client without refreshing or replaying mutations. Concurrent observer scopes on that receiver refuse; a hook-removal failure preserves a primary exception and prevents observer reuse of that capability. Drain/discard it and acquire fresh exact authority through the owner within the original deadline. Shared clients are never stamped as private, and production policy, scoped range transport and caller activation remain separate integration work.

Read Capabilities

Selected companion reads can explicitly declare one predecessor physical schema and a current schema that appends nullable fields. Existing fields, including their order, nullability, nested types and metadata, must match exactly. Each Parquet or Lance footer is checked against one of those complete schemas before projection. Existing arrays are retained unchanged; newly introduced fields are null for predecessor files. This is separate from catalog schema evolution and does not authorize a write or rewrite existing files.

The current schema is bound before creating the selected contract and its input coverage. The same executed plan carries the predecessor reference to downstream validators. Historical contract hashes may match only the current or declared predecessor schema; all creator, snapshot and exact input-revision checks remain. Finite episode filters still run at the file source. Predicates on newly added columns run over the bounded projected result, where absent fields are null. Metadata, rows and retained Arrow buffers keep the caller's explicit limits; these post-decode checks do not promise a bound on the file decoder's peak memory.

The shared read executor currently handles:

• schema-table reads
• schemaless manifest-table reads
• MOR reads
• direct pyarrow, pandas, polars, numpy, daft, and ray_dataset outputs where supported
• lazy pyarrow_parquet
• lazy lance

It also enforces direct validation for unsupported combinations, for example:

• schemaless + pyarrow_parquet
• schemaless + lance
• mixed-content lazy plans for format-specific readers
• unknown content types in the shared path

Polars / Daft Capability Matrix

The shared executor applies the same capability decision in thin execute_read_plan(...) execution and in thick reads that delegate into that shared path.

Notes:

• Mixed-schema lazy eligibility on the shared path requires per-file schema_id lookups plus top-level schema information with resolvable field types, whether that comes from schema_serialized or a typed top-level schema summary.
• JSON schema summaries and requests share the canonical datatype codec in schema_types.py. Readable scalar names follow PyArrow (float64, not PyArrow's double display text); parameter-rich types use a lossless one-field Arrow IPC token. Request parsing is strict, while response/plan decoding narrowly tolerates the five scalar spellings emitted by older DeltaCAT servers so rolling upgrades and durable plans remain readable.
• On the shared Daft path, non-identity Parquet content encodings (for example .parquet.gz) stay on the eager PyArrow path.
• When the process is pinned to DAFT_RUNNER=ray, the shared local Daft lazy path declines and falls back to the eager shared path instead of spawning a Ray-backed local lazy scan.

Write Capabilities

The shared write layer currently covers:

• write input normalization
• local data materialization
• manifest construction for existing files and datasets
• schema/read compatibility helpers
• standard catalog write orchestration slices

Authoritative catalog mutation, commit, retention, and compaction boundaries still remain on the native/server side where they belong.

Long-running Lance writes

write_lance_batch_stream accepts an optional refresh_access_ref callback from an existing scoped credential owner. The shared credential factory checks the original binding/scope/routing and provider intent on each refresh and emits an engine refresh deadline before actual expiry. The owner's existing lease cache keeps vending independent of fragment count. A failed refresh is retained for that write attempt so the engine cannot amplify it into repeated credential requests; job-level retry supplies a fresh stream and output path.

The writer uses Lance's public namespace interface only as a fixed-path credential adapter (namespace on pylance3, namespace_client on pylance11). It does not create catalog namespaces, implement a second writer, or retry/delete partially consumed streams. Unsupported writers refuse before stream consumption. Nonrenewing and local callers retain the ordinary write path.

open_prepared_lance_dataset accepts the same scoped access_ref and optional refresh_access_ref pair for native in-flight reads. It supplies initial routing options as well as the native provider: the callback alone is insufficient for Lance's bootstrap. Pylance11 uses the same fixed-path namespace adapter for reads instead of3's direct provider argument. Both generated-model bootstrap requests and mapping-shaped native refresh requests validate the fixed ID/path. Unsupported reader interfaces refuse before opening storage. Earlier source qualification described the3.0.0 environment as11.0.0 incorrectly; actual engine versions and installed wheel qualification must be recorded independently.

storage_read_support.EntryReadAuthority reuses RenewingStorageAuthority for one job's manifest reads. Path/revision validation precedes lease reuse; at most 128 scoped bindings are retained. The owner keeps an independent copy of each original binding policy and refuses conflicting policies for that same scope. Credential refresh uses the original policy, not a mutable current revision. Fresh reads share one acquisition per binding/scope; a failed acquisition is retained for the attempt, including concurrent native callbacks. No catalog RPC or global credential cache is introduced. Training inventory, native payload and media copies share that owner. Local HTTP controls cover 8/32 independent datasets, in-flight fragment rotation, and zero extra storage requests while a lease remains fresh. This is not a claim of in-flight renewal for every engine or already-open Parquet/stream handles, nor live AWS/Alpha qualification. An already-open Parquet handle retains its original credentials. Real HTTP tests over 8/32 row groups deliberately expire those credentials after the footer GET: the scan fails, and reopening the retained file with a fresh read owner recovers the exact rows/schema using the original binding revision. A changed current revision is not substituted. This is read-attempt recovery, not transparent mid-handle renewal or proof of whole-job recovery for every consumer.

Recovery-frontier indexes

AuthoritativeFrontierIndexWriter consumes exact, authenticated state-census Arrow batches and produces one immutable identity index. The index is bound to the census checkpoint URI, VersionId, payload digest/size/content type, manifest digest, catalog root, protocol, and hold fence. It rejects marker filenames, malformed or stale state identities, duplicate/out-of-order census rows, and mismatched cardinality.

ExactVersionFrontierIndexScan implements the FrontierScan contract over that index. It requires a binding-scoped StorageAccessRef, reads the exact S3 VersionId once, verifies the byte count and SHA-256 before exposing rows, and streams Arrow batches from bounded local disk. Its logical remote-call count is one regardless of census or frontier cardinality. Callers must continue to run LeaseBatchPrecondition before every remote mutation/publication batch.

Retirement-cohort authorities

write_authoritative_retirement_cohort streams a sorted exact membership set and a complete frozen transaction-success census into one immutable Arrow file. RetirementCohortIndex memory-maps that file and performs exact binary search over bounded record batches; a lookup is O(log N) and never scans or materializes the full cohort as Python rows. Its monotonic-successor proof streams both sorted sections in fixed Arrow chunks, proving member and success set inclusion in O(M + S) time and bounded memory.

RetirementStateEvidence carries only state already authenticated by the caller: member identity, normalized status, explicit commit scope, and at most one direct transaction parent. Resolution is pure and depth-one. Unbound sessions resolve to reversible abort; applied state with complete success identity resolves commit-forward. Unknown, contradictory, self-referential, or unresolved state fails closed.

Relationship To Other Packages

Use deltacat-client when you want the public thin client.

Use deltacat when you want the thick/native package.

Use deltacat-io-core directly only if you are intentionally building against the shared execution layer itself.