deltacat-io-core 0.1.1

Shared local IO execution layer for DeltaCAT read/write clients.

deltacat-io-core

deltacat-io-core is the shared local execution layer for DeltaCAT reads and writes.

It is used by both:

• deltacat-client for direct thin-plan execution
• deltacat for shared local execution and compatibility wrappers

Naming

• distribution/package name: deltacat-io-core
• Python import module: deltacat_io_core

The distribution uses dashes for consistency with deltacat-client. The import module keeps underscores because Python module names cannot contain -.

Scope

deltacat-io-core owns the code that should behave the same regardless of whether the caller is using the thin client or the thick DeltaCAT package.

Today that includes:

• direct execution of thin Plan objects
• MOR execution for thin and thick paths
• local file materialization and manifest building
• schema alignment and table conversion helpers
• sort-aware file ordering and manifest handling
• shared compaction/MOR helper layers and model types
• format-specific local readers/writers

Non-Goals

deltacat-io-core does not own:

• server routes or REST/MCP request handling
• authoritative catalog/storage mutations
• native Ray job orchestration surfaces
• public end-user API shape for deltacat or deltacat-client

It is a shared implementation layer, not the top-level user product.

Architecture

The current read architecture is:

1. The server resolves a thin Plan.
2. client.catalog.read(plan=...) executes that plan directly through deltacat-io-core.
3. dc.read_table(plan=...) for thin plans also executes through the same shared path.

There is no longer a runtime bridge back into thick DeltaCAT for thin plan execution. The plan contract is expected to carry the metadata required for direct execution.

The current write architecture is:

1. The client stages local files or materializes local data through shared helpers.
2. The authoritative commit still happens through DeltaCAT server/native boundaries.
3. Shared write-preparation and manifest logic lives in deltacat-io-core.

Installation

Base install:

uv pip install deltacat-io-core

Optional extras:

• deltacat-io-core[io] for local file readers/writers (pyarrow, fastavro)
• deltacat-io-core[pandas] for Pandas conversions
• deltacat-io-core[polars] for Polars conversions and lazy scan helpers
• deltacat-io-core[lance] for Lance dataset support
• deltacat-io-core[all] for the full local IO stack

Read Capabilities

The shared read executor currently handles:

• schema-table reads
• schemaless manifest-table reads
• MOR reads
• direct pyarrow, pandas, polars, numpy, daft, and ray_dataset outputs where supported
• lazy pyarrow_parquet
• lazy lance

It also enforces direct validation for unsupported combinations, for example:

• schemaless + pyarrow_parquet
• schemaless + lance
• mixed-content lazy plans for format-specific readers
• unknown content types in the shared path

Write Capabilities

The shared write layer currently covers:

• write input normalization
• local data materialization
• manifest construction for existing files and datasets
• schema/read compatibility helpers
• standard catalog write orchestration slices

Authoritative catalog mutation, commit, retention, and compaction boundaries still remain on the native/server side where they belong.

Relationship To Other Packages

Use deltacat-client when you want the public thin client.

Use deltacat when you want the thick/native package.

Use deltacat-io-core directly only if you are intentionally building against the shared execution layer itself.