chunkstore

License: MIT CI PyPI Python Rust Bindings

Embeddable content-addressed chunk storage (CAS) with byte-level deduplication, reference-count GC, and bindings for Rust, Python, and Go.

Drop a dedup layer into your app — not a backup CLI. When uploads, versions, and templates share bytes, store each unique chunk once and cut disk/S3 cost by up to ~90% on real workloads (see benchmarks).

Table of contents

Quick start

30 seconds — Python (PyPI, Linux / macOS / Windows wheels):

pip install chunkstore

from chunkstore import ChunkStore, FilesystemBackend

store = ChunkStore.open(FilesystemBackend("/data/chunks"))
store.ingest("doc_v1", b"hello world")
assert store.read("doc_v1") == b"hello world"
print(store.stats())  # savings_pct grows as you deduplicate

From source (all platforms, or hacking on the wrapper):

cd python && maturin develop --release

Rust:

cargo build --release && cargo test -p chunkstore-core

use chunkstore::{ChunkStore, FsBackend};

let store = ChunkStore::open(FsBackend::new("/data/chunks")?)?;
store.ingest("doc_v1", b"hello world")?;
println!("{:?}", store.stats()?);

Go (build core first):

./scripts/build-core.sh
cd go/chunkstore && go test -v

import "github.com/MuratovER/chunkstore/go/chunkstore"

store, _ := chunkstore.OpenFilesystem("/data/chunks")
defer store.Close()
store.Ingest("doc_v1", []byte("hello world"))

S3 (Python or Go — see docs/S3.md):

from chunkstore import ChunkStore, S3Backend
store = ChunkStore.open(S3Backend("my-bucket", prefix="chunks"))
store.ingest("doc_v1", b"hello world")

Cross-language smoke test: pytest -m cross_lang (Python write → Go read/delete → Python stats).

Streaming read (v0.3 — no full-file buffer in memory):

with open("out.bin", "wb") as f:
    store.read_to_writer("doc_v1", f)

for chunk in store.iter_chunks("doc_v1"):
    process(chunk)

var buf bytes.Buffer
store.ReadTo(&buf, "doc_v1")

Async Python (FastAPI / asyncio — Rust work off the event loop):

from chunkstore import AsyncChunkStore, AsyncChunkClient, FilesystemBackend

async with await AsyncChunkStore.open(FilesystemBackend("/data/chunks")) as store:
    client = AsyncChunkClient(store)
    await client.upload_file("doc_v1", b"hello world")
    data = await client.download_file("doc_v1")

Async S3: AsyncS3Backend + pip install "chunkstore[s3]" (includes aiobotocore). See python/README.md.

Design principles

Patterns borrowed from Restic, RocksDB, and zstd — applied to an embeddable dedup layer:

Principle What it means for you
Embed Library in your process. No daemon, no S3 proxy, no separate backup agent.
Content-addressed Chunk key = full SHA-256 hex (64 chars). Same bytes → same key → automatic dedup.
Verifiable Every read checks sha256(data) == digest. Bit rot and tampering surface immediately.
Efficient Refcount + GC: delete a file only drops chunks nothing else references.
Portable One on-disk format across Rust, Python, and Go (_manifest/, _refcount/, chunk blobs).
Honest scope Byte-identical dedup only. No perceptual hashing, no distributed metadata.

Public API boundary: language wrappers call the Rust core (PyO3 / cgo / direct crate). Backend layout and metadata keys are stable; internal Rust modules may change between minor versions until v1.0.

Should I use chunkstore?

flowchart TD
    start[Need to store files in my app] --> q1{Same bytes across uploads or versions?}
    q1 -->|No - unique photos/video| no[Skip dedup - use raw S3/FS]
    q1 -->|Yes - duplicates, templates, versions| q2{Want a backup CLI?}
    q2 -->|Yes| backup[Use Restic/Borg/Kopia]
    q2 -->|No - embed in upload service| q3{Edits at file start?}
    q3 -->|Often| cdc[chunkstore + CDC chunking]
    q3 -->|Rare - bulk immutable blobs| fixed[chunkstore + fixed 4 MiB]
Your situation Recommendation
Document CMS, PDF versions, rescans chunkstore + CDC
Idempotent re-upload of same file chunkstore (near 100% savings on re-upload)
Template / boilerplate libraries chunkstore (shared chunks across files)
Unique camera rolls, encodes, ZIPs Raw blob storage — dedup ~0%
Disaster recovery, off-site backups Restic / Borg — not chunkstore
S3-compatible object store for everything SeaweedFS / MinIO — not chunkstore

The problem

Most apps store files naively: every upload writes a full copy to disk or S3. With document versions, rescans, shared templates, and duplicate uploads, the same bytes are stored many times. Blob storage has no dedup API — you either accept the cost or build your own layer.

Naive storage What you pay
User uploads the same PDF twice 2× disk / S3 bytes
100 document versions with 90% shared content ~100× tail, ~100× shared prefix
1000 files built from 200 shared 4 MiB blocks Full logical size on disk

chunkstore splits files into chunks, addresses each chunk by SHA-256, stores each unique chunk once, and tracks references. Identical bytes → one physical object, refcount++. Delete a file → refcount-- → garbage-collect chunks at zero.

What you get

Benefit How
Lower storage bill Unique chunks stored once; shared prefixes and duplicates reused
Safe reads sha256(chunk_bytes) == digest on every read
Embeddable library Call from upload/versioning code — like embedding RocksDB, not running a DB server
Cross-language format Same directory from Python and Go
Pluggable backend FS and S3 in wrappers; core uses callbacks or built-in FS
Two chunking modes Fixed (fast) and CDC (edit-friendly)

Storage impact (measured)

Scenario: 1000 files from a pool of 200 × 4 MiB chunks (~7.8 GiB logical).

Naive storage vs chunkstore dedup

Approach Bytes on disk Savings vs naive
Naive per-file copy ~7.8 GiB —
chunkstore dedup ~0.78 GiB ~90%

Savings by workload

Bar chart of dedup savings percent by workload type

Savings band Meaning
< 10% Noise — dedup overhead may not pay off
20–30% Noticeable in disk/S3 billing
50%+ Strong fit: duplicates, clones, versions
80%+ Shared templates, chunk pools

Who it's for

Use case Why chunkstore
Document management PDF versions, scans, templates — large shared prefixes
File upload services Same file uploaded twice; resumable re-uploads
File versioning v2 = v1 + small edit; CDC keeps chunk boundaries
Multi-service stacks Python API writes, Go worker reads — same chunk directory

Dogfood target: document storage / docs_service — versioned PDFs, scans, boilerplate templates.

What this is / is not

chunkstore is chunkstore is not
Embeddable CAS/dedup library Backup CLI (Restic / Borg / Kopia)
Dedup layer over FS or S3 S3 gateway or reverse proxy
Byte-identical dedup (SHA-256) Perceptual / similarity dedup
Manifest + refcount + GC Distributed multi-node store
Shared format across Python / Go / Rust Encryption or compliance product

Current limits: single-process store lock; refcount and metadata on the backend — fine for one app node, not for concurrent multi-writer clusters without external coordination.

Compared to alternatives

vs raw blob storage (S3 / local FS)

Raw S3/FS chunkstore
Dedup None SHA-256 chunk-level
Integration Native SDK Library in your app
Versioning Your DB/metadata file_id → manifest
Best for Unique blobs, media Repeated bytes across files

vs backup tools (Restic, Borg, Kopia)

Backup tool chunkstore
Model init → backup → restore ingest / read / delete in-process
Encryption / retention Built-in Not included
Dedup + CDC Yes (Restic uses Rabin CDC) Yes — similar ideas, different format
Fit Ops / DR Application embed

Restic (~35k GitHub stars) solves backup. chunkstore solves "my upload service stores too many duplicate bytes" — complementary, not competing.

vs object stores (SeaweedFS, MinIO)

Object store chunkstore
API S3 HTTP In-process library
Scope Cluster, buckets, ACLs Single-store dedup layer
When to use both — chunkstore under your app; object store as backend via S3 wrapper

How it works

Architecture flow

file → chunks → SHA-256 (64-char hex) → backend
                ↓
         manifest (file_id → [digests])
         refcount (digest → count) → GC when count == 0
Layer Role
core/ Rust: chunking, hashing, manifests, refcount, C-API
python/ PyO3/maturin wrapper + FS/S3 backends
go/ cgo wrapper + FS/S3 backends

Metadata keys (shared across languages):

Key Content
_manifest/{file_id} JSON: ordered digests + file_bytes
_refcount/{digest} JSON: reference count
{digest} (64 hex) Raw chunk bytes

When dedup works — and when it doesn't

Works well

Workload Typical savings Notes
Pool 200×4 MiB chunks, 1000 files ~90% Benchmark scenario
Full duplicate file (2 copies) ~50% Second copy reuses all chunks
Partial overlap (shared prefix) 30–45% Only tail chunks are new
Document versions (90/10) ~40% 20 MiB files
Prefix insert 1 byte + CDC ~45% 4/5 CDC chunks reused
Re-upload of identical file ~100% All chunks exist

Poor or zero savings

Workload Typical savings Why
Prefix insert 1 byte + fixed 4 MiB ~0% All boundaries shift
Random unique binaries ~0% No overlap
Unique photos / video ~0% High entropy
JPEG, MP4, ZIP ~0% cross-file Already compressed

Fixed vs CDC on prefix insert

Chunking: fixed vs CDC

Fixed (default 4 MiB) CDC (FastCDC v2020)
Speed Fastest Moderate
Chunk sizes Uniform min 256 KiB, avg 4 MiB, max 8 MiB
Best for Bulk uploads Versioned docs, edits at start
Prefix insert (+1 byte) ~0% reuse ~45% reuse
API ingest / Ingest ingest_cdc / IngestCDC

Multi-language

Language Open Upload Read Delete Stream read
Rust ChunkStore::open(FsBackend::new(path)?) ingest read delete read_to_writer
Python ChunkStore.open(FilesystemBackend(path)) ingest read delete read_to_writer, iter_chunks
Python async AsyncChunkStore.open(backend) await ingest await read await delete await read_to_writer, async iter_chunks
Go chunkstore.OpenFilesystem(path) Ingest Read Delete ReadTo(io.Writer)

CI verifies: Python write → Go read/delete → Python stats (pytest -m cross_lang).

Installation

Language Requirements Command
Python Python 3.10+ pip install chunkstore — Linux, macOS, Windows wheels (PyPI)
Python (dev) Rust (maturin) cd python && maturin develop --release
Rust Rust 1.70+ cargo build --release -p chunkstore-core
Go Go 1.24+, built libchunkstore.a go get github.com/MuratovER/chunkstore/go@v0.3.0 — see go/README.md
Extras — pip install "chunkstore[s3,fastapi,dev]" ([s3] = boto3 + aiobotocore)

Docs: S3 backend · Chunking guide · Rust crate

# Full local verify (from repo root)
./scripts/build-core.sh
cd python && maturin develop --release && pytest -q
cd ../go/chunkstore && go test -v

Examples

Example Description
examples/fastapi/ Upload / download / delete HTTP API
examples/fastapi-backup/ Gzip backup dumps + SQLite date catalog
examples/go-http/ Go HTTP service (upload / download / delete / stats)

FastAPI (basic):

cd python && maturin develop --release && pip install ".[fastapi]"
PYTHONPATH=../examples/fastapi uvicorn main:app --reload

FastAPI (backup storage pattern):

cd python && maturin develop --release && pip install ".[fastapi]"
PYTHONPATH=examples/fastapi-backup uvicorn main:app --host 0.0.0.0 --port 8081 --reload

Go HTTP:

CARGO_TARGET_DIR=target cargo build --release -p chunkstore-core
cd examples/go-http && go run .

Endpoints: POST /files/{id}, GET /files/{id}, DELETE /files/{id}, GET /stats.

Benchmarks

Reproducible workloads via workload_analysis — run locally:

cargo run -p chunkstore-core --example workload_analysis --release
cargo bench -p chunkstore-core
Workload Savings
Dedup pool 200×4 MiB, 1000 files 90.0%
Versions 90/10 40.0%
Prefix insert + CDC 45.1%
Full duplicate (2 copies) 50.0%

On document-style workloads above, chunkstore averages ~45% savings on typical cases; the shared chunk-pool scenario reaches 90% — roughly 2× better.

xychart-beta
    title "Dedup savings % (document workloads)"
    x-axis ["Pool 1k", "CDC +1B", "Versions", "Dup 2x"]
    y-axis "Savings %" 0 --> 100
    bar [90, 45, 40, 50]

Contributing

Want to fix a bug, add a feature, or improve bindings? See CONTRIBUTING.md for:

  • local setup and full CI commands
  • repository layout and where to put changes
  • test requirements (including cross-language)
  • Rust / Python / Go style rules
  • on-disk format stability rules

PyPI releases: docs/PYPI.md

Roadmap: docs/ROADMAP.md

License

MIT — see LICENSE.

Project links

Data verified by PyPI on Jun 25, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for MuratovER from gravatar.com MuratovER

Credits

Author: Eldar Muratov

GitHub Statistics

Data verified by PyPI on Jun 25, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

MIT
View SPDX License List

Requires

Python >=3.10

Provides Extra

dev fastapi s3

Tags

cas deduplication chunking storage

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

chunkstore-0.3.0.tar.gz (50.6 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

chunkstore-0.3.0-cp310-abi3-win_amd64.whl (312.1 kB view details)

Uploaded CPython 3.10+Windows x86-64

chunkstore-0.3.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (472.7 kB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ x86-64

chunkstore-0.3.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (451.8 kB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ ARM64

chunkstore-0.3.0-cp310-abi3-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl (823.1 kB view details)

Uploaded CPython 3.10+macOS 10.12+ universal2 (ARM64, x86-64)macOS 10.12+ x86-64macOS 11.0+ ARM64

File details

Details for the file chunkstore-0.3.0.tar.gz.

File metadata

  • Download URL: chunkstore-0.3.0.tar.gz
  • Upload date:
  • Size: 50.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for chunkstore-0.3.0.tar.gz
Algorithm Hash digest
SHA256 a96562f2b6390c05d76c45c61aadb410ef4b4f7fb5f2350c3abe26f2c994e0e8
MD5 456b977fe4f8208a42da5650cc448297
BLAKE2b-256 81493f749c923f504ce91089180db644752e392e363cde8b43a1741ed5e1a81e

See more details on using hashes here.

Provenance

The following attestation bundles were made for chunkstore-0.3.0.tar.gz:

Publisher: pypi.yml on MuratovER/chunkstore

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file chunkstore-0.3.0-cp310-abi3-win_amd64.whl.

File metadata

  • Download URL: chunkstore-0.3.0-cp310-abi3-win_amd64.whl
  • Upload date:
  • Size: 312.1 kB
  • Tags: CPython 3.10+, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for chunkstore-0.3.0-cp310-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 5d84f629cd8f205322866a14ac0a044e3e3a97d2cafa7dc56f5ef00de05f5468
MD5 4f69a2851be93cd763b3f4d3bcd48d94
BLAKE2b-256 b2725b0e47a994d24b013e13956837250ffdc0e5e40236495f4c367138ed126b

See more details on using hashes here.

Provenance

The following attestation bundles were made for chunkstore-0.3.0-cp310-abi3-win_amd64.whl:

Publisher: pypi.yml on MuratovER/chunkstore

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file chunkstore-0.3.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for chunkstore-0.3.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 929f0ea52c02eb27c8ce8b47176c3d24ab781003ecb927ed2fbe046492d9a515
MD5 9856d0bae5bd20199ad3dd07992b5ec8
BLAKE2b-256 183b4701e02bc60cef66cc2fa3653edb73d052ac227057d73905ba1fe53d13ba

See more details on using hashes here.

Provenance

The following attestation bundles were made for chunkstore-0.3.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: pypi.yml on MuratovER/chunkstore

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file chunkstore-0.3.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for chunkstore-0.3.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 7e48ac58fe4451ce4d5114de321c2c04074fe58da5a5abef4f696b7587a82536
MD5 4e1fc6273af6ea036e9e89f404ac819b
BLAKE2b-256 6da5f0f06e80ec642c5b584bb818632dbd8311ed8d1d8cd366cd57c307e2819e

See more details on using hashes here.

Provenance

The following attestation bundles were made for chunkstore-0.3.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: pypi.yml on MuratovER/chunkstore

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file chunkstore-0.3.0-cp310-abi3-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl.

File metadata

File hashes

Hashes for chunkstore-0.3.0-cp310-abi3-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl
Algorithm Hash digest
SHA256 36a0c7020192f6b88d2104e360600c7b2136a3473e161345455af68c379031cb
MD5 177782c71248ffd5b4792ba92554a2ac
BLAKE2b-256 aad5bf46a8f8434ff1886c029550ee0afb65a59236ba0735de043dc5ba625669

See more details on using hashes here.

Provenance

The following attestation bundles were made for chunkstore-0.3.0-cp310-abi3-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl:

Publisher: pypi.yml on MuratovER/chunkstore

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

0.3.0 This release

5 files

0.2.1

5 files

0.1.0

3 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page