deltalite

Streaming, partition-level upsert for Delta Lake tables that replaces delta-rs's SQL MERGE with a bounded-memory merge engine. Memory is bounded by the size of the incoming batch and a few concurrency knobs — never by the size of the target table.

delta-rs stays the storage and protocol layer (transaction log, checkpoints, Parquet writing, Add-action statistics, S3 conditional-put commits, conflict resolution). deltalite replaces only the merge execution.

import deltalite

table = deltalite.DeltaLiteTable.open("s3://bucket/my_table")
stats = table.upsert(record_batch, primary_keys=["id"], partition_key="day")
print(f"v{stats.version}: +{stats.rows_inserted} / ~{stats.rows_updated}")

Why

delta-rs MERGE executes a DataFusion hash join whose memory scales with the scanned target, and it can deadlock under a bounded memory pool (delta-io/delta-rs#4614). For a large, slowly-changing table merged against a comparatively small batch — the typical incremental-sync shape — that means either OOM risk or a hang.

deltalite takes a different route:

Build a primary-key hash set over the (small) source batch.
Stream the (large) target one Parquet row group at a time, dropping rows whose key is in the source set.
Write survivors plus the source rows into new files.
Commit every touched partition in one atomic Delta commit.

Peak memory is bounded by the source batch and the concurrency knobs, not by the table. In validation, resident memory stayed flat from 62k- to 1M-row partitions (~4× below MERGE) at matching write volume, via exact content-based file selection.

Installation

pip install deltalite
# or
uv add deltalite

Prebuilt cp312-abi3 wheels are published for manylinux (2_28) and musllinux on x86_64/aarch64, and macOS on arm64/x86_64. A single wheel works on any CPython 3.12 or newer. No Rust toolchain is needed to install.

Usage

upsert accepts anything with the pyarrow C-stream interface — a pyarrow.Table, a RecordBatch, or a RecordBatchReader:

import pyarrow as pa
import deltalite

table = deltalite.DeltaLiteTable.open(
    "s3://bucket/events",
    storage_options={
        "AWS_REGION": "us-east-1",
        "AWS_ACCESS_KEY_ID": "...",
        "AWS_SECRET_ACCESS_KEY": "...",
    },
)

batch = pa.table({
    "id":  [1, 2, 3],
    "day": ["2026-01-01", "2026-01-01", "2026-01-02"],
    "val": ["a", "b", "c"],
})

stats = table.upsert(
    batch,
    primary_keys=["id"],
    partition_key="day",          # omit for an unpartitioned table
    commit_metadata={"source": "my-sync"},
)

print(stats)
# UpsertStats(version=42, partitions_touched=2, files_added=2, files_removed=1, ...)

Rows in the batch whose primary key already exists are replaced; new keys are inserted. Deletes are not expressed through upsert — it is an insert-or-replace by key. Duplicate primary keys within a single batch are rejected (raising DeltaLiteError) rather than silently double-inserted.

API

`DeltaLiteTable`

Method	Description
`DeltaLiteTable.open(uri, storage_options=None)`	Open an existing Delta table. `storage_options` is the usual object-store dict (S3/GCS/Azure/local).
`DeltaLiteTable.is_deltatable(uri, storage_options=None)`	`True` if a Delta table exists at `uri`.
`.upsert(data, primary_keys, partition_key=None, **opts)`	Insert-or-replace `data` by key. Returns `UpsertStats`. See knobs below.
`.version()`	Current table version (`int`).
`.reload()`	Reload the table state from the log.
`.schema_arrow()`	Table schema as a pyarrow `Schema`.
`.partition_columns()`	Partition column names (`list[str]`).
`.file_uris()`	URIs of the table's active data files.
`.history(limit)`	Recent commit history entries.

`UpsertStats`

Returned by upsert. Counts: version, partitions_touched, files_added, files_removed, files_carried_over, files_probed, rows_updated, rows_inserted, rows_copied, source_rows, null_pk_rows. Per-phase wall-clock timings (milliseconds): plan_ms (listing + pruning files), rewrite_ms (reading + rewriting the touched partitions), commit_ms (committing to the Delta log).

Exceptions

All inherit from DeltaLiteError, so you can catch the base or branch on kind:

Exception	Raised when
`DeltaLiteError`	Base class / generic failure.
`DeltaLiteCommitConflictError`	Concurrent commit won the conditional-put race (retry-exhausted).
`DeltaLiteSchemaMismatchError`	Batch schema is incompatible with the table.
`DeltaLiteTableNotFoundError`	No Delta table at the URI.
`DeltaLiteUnsupportedTableError`	Table uses a feature deltalite can't handle (e.g. deletion vectors, column mapping).
`DeltaLiteSourceTooLargeError`	Batch exceeds `max_source_bytes` (see below).

Operational knobs

Per-call — keyword arguments to upsert (defaults in parentheses):

Argument	Default	Purpose
`max_parallel_partitions`	`2`	Partitions merged concurrently.
`max_parallel_files`	`4`	Files read concurrently within a partition.
`max_buffered_bytes`	`64 MiB`	Output buffered in memory before flushing.
`prune_strategy`	`"probe"`	`"probe"` skips files that can't contain a source key; `"none"` scans all.
`skip_unmatched_files`	`True`	Convenience toggle: `False` ≡ `prune_strategy="none"`.
`probe_concurrency`	`8`	Concurrent statistics/probe reads.
`read_batch_size`	`8192`	Row-group read batch size.
`target_file_size`	table setting, else 100 MiB	Output file size target.
`max_source_bytes`	`2 GiB`	Oversized-batch guard (`0` disables).
`multipart_threshold` / `multipart_part_size`	`64 MiB` / `16 MiB`	Multipart upload thresholds (`0` threshold disables).
`commit_max_retries`	`15`	Conditional-put commit retry budget.
`commit_metadata`	`None`	Extra key/values recorded in the Delta commit.

Process-global — environment variables, enforced on top of the per-call knobs so that many concurrent upsert threads in one process cannot multiply the budgets:

DELTALITE_PROCESS_MAX_PARALLEL_PARTITIONS (8), DELTALITE_PROCESS_MAX_PARALLEL_FILES (16), DELTALITE_PROCESS_MAX_BUFFERED_BYTES (256 MiB), DELTALITE_MAX_SOURCE_BYTES, DELTALITE_MULTIPART_THRESHOLD_BYTES, DELTALITE_MULTIPART_PART_SIZE_BYTES.

Metrics

deltalite emits via the Rust metrics facade (static labels only): deltalite_upserts_total (outcome, prune_strategy, error_kind), deltalite_upsert_duration_seconds, deltalite_files_{added,removed,carried_over,probed}_total, and deltalite_rows_{updated,inserted,copied}_total.

Compatibility & status

Built against deltalake (delta-rs) 0.32.x as the storage/protocol layer. Correctness is guaranteed by a differential parity suite that runs the same batch sequences through real delta-rs MERGE and through deltalite.upsert and asserts identical logical content — not by version equality.
Not supported: tables with deletion vectors or column mapping (detected and raised as DeltaLiteUnsupportedTableError), and SCD2 merges.
deltalite rejects duplicate source primary keys that MERGE silently double-inserts — check pre-existing data if you migrate an existing pipeline.

This package is developed in the PostHog monorepo under rust/deltalite/. Issues and source live there.

Development Status
- 4 - Beta
Operating System
- OS Independent
Programming Language

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

deltalite-0.1.6.tar.gz (175.5 kB view details)

Uploaded Aug 10, 2026 Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

deltalite-0.1.6-cp312-abi3-musllinux_1_2_x86_64.whl (63.8 MB view details)

Uploaded Aug 10, 2026 CPython 3.12+musllinux: musl 1.2+ x86-64

deltalite-0.1.6-cp312-abi3-musllinux_1_2_aarch64.whl (62.8 MB view details)

Uploaded Aug 10, 2026 CPython 3.12+musllinux: musl 1.2+ ARM64

deltalite-0.1.6-cp312-abi3-manylinux_2_28_x86_64.whl (65.5 MB view details)

Uploaded Aug 10, 2026 CPython 3.12+manylinux: glibc 2.28+ x86-64

deltalite-0.1.6-cp312-abi3-manylinux_2_28_aarch64.whl (63.2 MB view details)

Uploaded Aug 10, 2026 CPython 3.12+manylinux: glibc 2.28+ ARM64

deltalite-0.1.6-cp312-abi3-macosx_11_0_arm64.whl (17.9 MB view details)

Uploaded Aug 10, 2026 CPython 3.12+macOS 11.0+ ARM64

deltalite-0.1.6-cp312-abi3-macosx_10_12_x86_64.whl (18.0 MB view details)

Uploaded Aug 10, 2026 CPython 3.12+macOS 10.12+ x86-64

File details

Details for the file deltalite-0.1.6.tar.gz.

File metadata

Download URL: deltalite-0.1.6.tar.gz
Upload date: Aug 10, 2026
Size: 175.5 kB
Tags: Source
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/6.1.0 CPython/3.13.7

File hashes

Hashes for deltalite-0.1.6.tar.gz
Algorithm	Hash digest
SHA256	`b5f8f728ad12be62875e01e7959e761d0fb350b3279571d32727aa959e78fb4c`
MD5	`42ef5042fe770973963cde6b2aacb783`
BLAKE2b-256	`e60751964c17d24dce50a90e617e959dac50bd6ee961ec1d30149a11221435da`

See more details on using hashes here.

Provenance

The following attestation bundles were made for deltalite-0.1.6.tar.gz:

Publisher: build-deltalite.yml on PostHog/posthog

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: deltalite-0.1.6.tar.gz
- Subject digest: b5f8f728ad12be62875e01e7959e761d0fb350b3279571d32727aa959e78fb4c
- Sigstore transparency entry: 2407845414
- Sigstore integration time: Aug 10, 2026
Source repository:
- Permalink: PostHog/posthog@7739b9343e57618085b03b66c76f2938453a63fe
- Branch / Tag: refs/pull/80498/merge
- Owner: https://github.com/PostHog
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: build-deltalite.yml@7739b9343e57618085b03b66c76f2938453a63fe
- Trigger Event: pull_request

File details

Details for the file deltalite-0.1.6-cp312-abi3-musllinux_1_2_x86_64.whl.

File metadata

Download URL: deltalite-0.1.6-cp312-abi3-musllinux_1_2_x86_64.whl
Upload date: Aug 10, 2026
Size: 63.8 MB
Tags: CPython 3.12+, musllinux: musl 1.2+ x86-64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/6.1.0 CPython/3.13.7

File hashes

Hashes for deltalite-0.1.6-cp312-abi3-musllinux_1_2_x86_64.whl
Algorithm	Hash digest
SHA256	`b267a1adbc22ed802475dbf9e6ea629275c0dc99a77b6c119bd9909d5965ca54`
MD5	`3daf31e7cb0cc6ad1cc4b200f1c2f603`
BLAKE2b-256	`d17a0039dcf77fdfe54962311e5f61491f4155e761f7059f1c331ae0c594e399`

See more details on using hashes here.

Provenance

The following attestation bundles were made for deltalite-0.1.6-cp312-abi3-musllinux_1_2_x86_64.whl:

Publisher: build-deltalite.yml on PostHog/posthog

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: deltalite-0.1.6-cp312-abi3-musllinux_1_2_x86_64.whl
- Subject digest: b267a1adbc22ed802475dbf9e6ea629275c0dc99a77b6c119bd9909d5965ca54
- Sigstore transparency entry: 2407845500
- Sigstore integration time: Aug 10, 2026
Source repository:
- Permalink: PostHog/posthog@7739b9343e57618085b03b66c76f2938453a63fe
- Branch / Tag: refs/pull/80498/merge
- Owner: https://github.com/PostHog
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: build-deltalite.yml@7739b9343e57618085b03b66c76f2938453a63fe
- Trigger Event: pull_request

File details

Details for the file deltalite-0.1.6-cp312-abi3-musllinux_1_2_aarch64.whl.

File metadata

Download URL: deltalite-0.1.6-cp312-abi3-musllinux_1_2_aarch64.whl
Upload date: Aug 10, 2026
Size: 62.8 MB
Tags: CPython 3.12+, musllinux: musl 1.2+ ARM64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/6.1.0 CPython/3.13.7

File hashes

Hashes for deltalite-0.1.6-cp312-abi3-musllinux_1_2_aarch64.whl
Algorithm	Hash digest
SHA256	`701e7093ff60ed2e6764caf1c8846430f71714b469ac0dcc052740c00fca5f80`
MD5	`7ee1c0de420061653429dbdb34eb3813`
BLAKE2b-256	`4f3ca05a0addbf7b9ca333ded94b360782afd6b9bdd163b665bdf13143cb5d12`

See more details on using hashes here.

Provenance

The following attestation bundles were made for deltalite-0.1.6-cp312-abi3-musllinux_1_2_aarch64.whl:

Publisher: build-deltalite.yml on PostHog/posthog

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: deltalite-0.1.6-cp312-abi3-musllinux_1_2_aarch64.whl
- Subject digest: 701e7093ff60ed2e6764caf1c8846430f71714b469ac0dcc052740c00fca5f80
- Sigstore transparency entry: 2407845464
- Sigstore integration time: Aug 10, 2026
Source repository:
- Permalink: PostHog/posthog@7739b9343e57618085b03b66c76f2938453a63fe
- Branch / Tag: refs/pull/80498/merge
- Owner: https://github.com/PostHog
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: build-deltalite.yml@7739b9343e57618085b03b66c76f2938453a63fe
- Trigger Event: pull_request

File details

Details for the file deltalite-0.1.6-cp312-abi3-manylinux_2_28_x86_64.whl.

File metadata

Download URL: deltalite-0.1.6-cp312-abi3-manylinux_2_28_x86_64.whl
Upload date: Aug 10, 2026
Size: 65.5 MB
Tags: CPython 3.12+, manylinux: glibc 2.28+ x86-64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/6.1.0 CPython/3.13.7

File hashes

Hashes for deltalite-0.1.6-cp312-abi3-manylinux_2_28_x86_64.whl
Algorithm	Hash digest
SHA256	`dfbccd799364847e262004acb5dc61d6e373ed79ac9e0e0bf5fae8c4aacfce2d`
MD5	`a7dc3e9eb8430e0e70dafd7c7c9cafb4`
BLAKE2b-256	`cffa07d484ef2f8db20ad8841b91084edd0e885fe2c4a7f9e7e7f44aa6210a4d`

See more details on using hashes here.

Provenance

The following attestation bundles were made for deltalite-0.1.6-cp312-abi3-manylinux_2_28_x86_64.whl:

Publisher: build-deltalite.yml on PostHog/posthog

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: deltalite-0.1.6-cp312-abi3-manylinux_2_28_x86_64.whl
- Subject digest: dfbccd799364847e262004acb5dc61d6e373ed79ac9e0e0bf5fae8c4aacfce2d
- Sigstore transparency entry: 2407845440
- Sigstore integration time: Aug 10, 2026
Source repository:
- Permalink: PostHog/posthog@7739b9343e57618085b03b66c76f2938453a63fe
- Branch / Tag: refs/pull/80498/merge
- Owner: https://github.com/PostHog
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: build-deltalite.yml@7739b9343e57618085b03b66c76f2938453a63fe
- Trigger Event: pull_request

File details

Details for the file deltalite-0.1.6-cp312-abi3-manylinux_2_28_aarch64.whl.

File metadata

Download URL: deltalite-0.1.6-cp312-abi3-manylinux_2_28_aarch64.whl
Upload date: Aug 10, 2026
Size: 63.2 MB
Tags: CPython 3.12+, manylinux: glibc 2.28+ ARM64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/6.1.0 CPython/3.13.7

File hashes

Hashes for deltalite-0.1.6-cp312-abi3-manylinux_2_28_aarch64.whl
Algorithm	Hash digest
SHA256	`5308c408502aca987f1a315621fbe5937301263e4a236971525db2d5bc98df89`
MD5	`491fcacde7ef423b5b1a4cc19aa4ce95`
BLAKE2b-256	`168127ca3a8da82ae61c556dbf648bb9ff9eeaf400b03d32b8a7429f2dac5fce`

See more details on using hashes here.

Provenance

The following attestation bundles were made for deltalite-0.1.6-cp312-abi3-manylinux_2_28_aarch64.whl:

Publisher: build-deltalite.yml on PostHog/posthog

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: deltalite-0.1.6-cp312-abi3-manylinux_2_28_aarch64.whl
- Subject digest: 5308c408502aca987f1a315621fbe5937301263e4a236971525db2d5bc98df89
- Sigstore transparency entry: 2407845569
- Sigstore integration time: Aug 10, 2026
Source repository:
- Permalink: PostHog/posthog@7739b9343e57618085b03b66c76f2938453a63fe
- Branch / Tag: refs/pull/80498/merge
- Owner: https://github.com/PostHog
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: build-deltalite.yml@7739b9343e57618085b03b66c76f2938453a63fe
- Trigger Event: pull_request

File details

Details for the file deltalite-0.1.6-cp312-abi3-macosx_11_0_arm64.whl.

File metadata

Download URL: deltalite-0.1.6-cp312-abi3-macosx_11_0_arm64.whl
Upload date: Aug 10, 2026
Size: 17.9 MB
Tags: CPython 3.12+, macOS 11.0+ ARM64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/6.1.0 CPython/3.13.7

File hashes

Hashes for deltalite-0.1.6-cp312-abi3-macosx_11_0_arm64.whl
Algorithm	Hash digest
SHA256	`b4ad22983bfbc8335c7880b7d9a2bc8d575f2225b05437f426ec86ceb060891b`
MD5	`e26e2d852ca7bc07185a84d21c387042`
BLAKE2b-256	`dba2917ca2215a7c54ce42f7b3669c7917d1b5470b236c5d680d8ec83f065cc1`

See more details on using hashes here.

Provenance

The following attestation bundles were made for deltalite-0.1.6-cp312-abi3-macosx_11_0_arm64.whl:

Publisher: build-deltalite.yml on PostHog/posthog

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: deltalite-0.1.6-cp312-abi3-macosx_11_0_arm64.whl
- Subject digest: b4ad22983bfbc8335c7880b7d9a2bc8d575f2225b05437f426ec86ceb060891b
- Sigstore transparency entry: 2407845529
- Sigstore integration time: Aug 10, 2026
Source repository:
- Permalink: PostHog/posthog@7739b9343e57618085b03b66c76f2938453a63fe
- Branch / Tag: refs/pull/80498/merge
- Owner: https://github.com/PostHog
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: build-deltalite.yml@7739b9343e57618085b03b66c76f2938453a63fe
- Trigger Event: pull_request

File details

Details for the file deltalite-0.1.6-cp312-abi3-macosx_10_12_x86_64.whl.

File metadata

Download URL: deltalite-0.1.6-cp312-abi3-macosx_10_12_x86_64.whl
Upload date: Aug 10, 2026
Size: 18.0 MB
Tags: CPython 3.12+, macOS 10.12+ x86-64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/6.1.0 CPython/3.13.7

File hashes

Hashes for deltalite-0.1.6-cp312-abi3-macosx_10_12_x86_64.whl
Algorithm	Hash digest
SHA256	`6fbdbe6e110c738f567b2bd01e79897479f11989b437735daab7b144c5267503`
MD5	`2f1adfacbcd093c9c5c511071327deef`
BLAKE2b-256	`8c3bc28f2fcba6ebbbab14a7781108956a288e5e275d6ae342c3641d56ff410b`

See more details on using hashes here.

Provenance

The following attestation bundles were made for deltalite-0.1.6-cp312-abi3-macosx_10_12_x86_64.whl:

Publisher: build-deltalite.yml on PostHog/posthog

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: deltalite-0.1.6-cp312-abi3-macosx_10_12_x86_64.whl
- Subject digest: 6fbdbe6e110c738f567b2bd01e79897479f11989b437735daab7b144c5267503
- Sigstore transparency entry: 2407845548
- Sigstore integration time: Aug 10, 2026
Source repository:
- Permalink: PostHog/posthog@7739b9343e57618085b03b66c76f2938453a63fe
- Branch / Tag: refs/pull/80498/merge
- Owner: https://github.com/PostHog
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: build-deltalite.yml@7739b9343e57618085b03b66c76f2938453a63fe
- Trigger Event: pull_request

Release history Release notifications | RSS feed

0.1.7

Aug 17, 2026 7 files

This release

0.1.6 This release

Aug 10, 2026 7 files

Aug 6, 2026 7 files

Aug 6, 2026 7 files

Jul 31, 2026 7 files

Jul 29, 2026 7 files

Jul 28, 2026 7 files

deltalite 0.1.6

deltalite

Why

Installation

Usage

API

DeltaLiteTable

UpsertStats

Exceptions

Operational knobs

Metrics

Compatibility & status

Project links

Key dates

Owner

Credits

GitHub Statistics

License expression

Requires

Classifiers

Download files

Source Distribution

Built Distributions

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

Release history Release notifications | RSS feed

`DeltaLiteTable`

`UpsertStats`