zpe-multimodal 3.0.1

ZPE Integrated Modality Codec with text/emoji/diagram/image/music/voice/mental/touch/smell+BPE

ZPE-IMC

Quick Start

# Install from PyPI
pip install zpe-multimodal

Or install from source (development):

# 1. Clone and enter
git clone https://github.com/zer0pa/zpe-imc
cd zpe-imc

# 2. Create venv and install Python package
python -m venv .venv
source .venv/bin/activate
pip install -e ".[full,diagram,music,dev]"

# 3. Build and install the Rust kernel (REQUIRED — requires Rust toolchain)
#    Without this step, native-kernel tests will fail.
#    This compiles the native acceleration layer via maturin.
bash code/rust/imc_kernel/build_install.sh

# 4. Verify
pytest

# 5. Run the demo
python executable/demo.py

Expected output: the demo runs all 10 modalities and prints the canonical mixed-stream word count (total_words: 844). This is the authority proof path.

What This Is

Ten modalities. One deterministic pipeline. One 20-bit word envelope. 277K words/sec on the Rust kernel. 172/174 tests with determinism hash match.

ZPE-IMC contains reference implementations of 10 modality codecs sharing a common 20-bit word envelope, delivered as one Python package: zpe-multimodal (available on PyPI: pip install zpe-multimodal). IMC demonstrates the encoding framework that the Zer0pa family is built on. It does not import from the standalone ZPE-* lane repos; each modality codec within IMC is an independent reference implementation. Not a text compressor -- a representation layer. Domain repos (IoT, Robotics, Geo) are the buying surfaces. IMC is the platform core they run on.

All current evidence is bounded to synthetic and reference inputs. No real-world production workload validation exists.

Readiness: staged, synthetic evidence only. Public repository. No production workload validation.

Not claimed: Production deployment readiness, specialist-encoder parity, CLI/demo equivalence, audio support beyond Python 3.11/3.12.

Part of the Zer0pa family. Sibling codec repos: ZPE-Bio, ZPE-FT, ZPE-Geo, ZPE-Ink, ZPE-IoT, ZPE-Mocap, ZPE-Neuro, ZPE-Prosody, ZPE-Robotics, ZPE-XR.

Field	Value
Architecture	MULTIMODAL_DISPATCH
Encoding	UNIFIED_20BIT_WORD

Commercial Readiness

Field	Value
Verdict	STAGED
Commit SHA	933adca9
Confidence	85%
Source	proofs/artifacts/modality_benchmarks.json

Evaluators: Domain repos are the entry points. IMC is the platform core they run on. Contact [e-mail] for portfolio evaluation.

Key Metrics

Metric	Value	Baseline
MODALITIES	10	--
THROUGHPUT	276,799	words/sec
DETERMINISM	11/11	--
DEMO_TESTS	172/174	--

Image modality note: The image codec achieves CR 2.64x (the encoded representation before gzip is larger than raw, but gzipped output is 2.64x smaller than input: raw 3,072 bytes / gzipped encoded 1,162 bytes). Image roundtrip fidelity (PSNR 99 dB) is verified. See proofs/artifacts/modality_benchmarks.json for per-modality ratios.

Voice modality note: Voice CR 373.2x reflects prosodic contour extraction from a 38 KB WAV (pitch, energy, and timing mapped to a 64-command sequence), not sample-level audio compression.

What We Prove

Auditable guarantees backed by committed proof artifacts. Start at AUDITOR_PLAYBOOK.md.

• Unified 20-bit word envelope dispatches across all 10 modalities through a single API
• Deterministic roundtrip encoding and decoding verified for every modality
• Mixed-stream demo anchored to a canonical 844-word frozen contract (total_words=844 in the Wave-1 compatibility vector); live demo runs may produce additional demonstration words beyond the canonical set
• Per-lane regression suite (62/62 PASS) maintained independently from sibling codec repos
• ONNX export parity achieved for the tokenizer operator

What We Don't Claim

• Production deployment readiness
• Performance parity with single-modality specialist encoders
• Validation on real-world production workloads
• CLI surface equivalence to demo path (tracked 780 vs 844 word split)
• Audio toolchain support beyond Python 3.11/3.12
• Integration with sibling ZPE-* repos -- IMC contains independent reference implementations, not imports from the standalone lane repos
• Universal compression across all modalities -- text (80 B), emoji (12 B), and diagram (105 B) inputs expand through the 20-bit word envelope due to serialization overhead on sub-200-byte inputs. Compression emerges on inputs above ~500 bytes (music 32.9x, voice 373.2x, taste 18.4x, mental 5.9x, touch 4.8x, smell 3.7x, image 2.6x)

Tests and Verification

Code	Check	Verdict
V_01	Wave-1 runtime test suite (172/174)	PASS
V_02	Modality roundtrip (10/10)	PASS
V_03	Regression battery (62/62)	PASS
V_04	Determinism hash match	PASS
V_05	ONNX export parity	PASS
V_06	Mixed-stream canonical count (844)	PASS
V_07	Taste regression (2 legacy path tests)	FAIL
V_08	CLI/demo parity (780 vs 844)	FAIL
V_09	Path portability cleanup	INC

Proof Anchors

Path	State
proofs/artifacts/modality_benchmarks.json	VERIFIED
proofs/artifacts/2026-02-24_program_maximal/A6/metrics/onnx_parity.json	MISSING -- A6 directory not present in public snapshot
proofs/artifacts/2026-02-24_program_maximal/A6/TEST_RESULTS.md	MISSING -- A6 directory not present in public snapshot
proofs/artifacts/2026-02-24_program_maximal/A6/CHECKSUMS.sha256	MISSING -- A6 directory not present in public snapshot
proofs/artifacts/2026-02-24_program_maximal/A6/DELIVERY.md	MISSING -- A6 directory not present in public snapshot

Note: The A6 proof artifacts are operator-only and intentionally excluded from the public snapshot. Only modality_benchmarks.json is publicly verifiable. The public audit path uses the rerun bundle and logs; see AUDITOR_PLAYBOOK.md.

Repo Shape

Field	Value
Proof Anchors	1 verified, 4 missing (A6 operator-only)
Modality Lanes	10
Authority Source	proofs/artifacts/modality_benchmarks.json

Ecosystem

Workstream	Route	Notes
ZPE-Bio	github.com/Zer0pa/ZPE-Bio	Biology codec workstream
ZPE-FT	github.com/Zer0pa/ZPE-FT	Finance codec workstream
ZPE-Geo	github.com/Zer0pa/ZPE-Geo	Geospatial codec workstream
ZPE-Ink	github.com/Zer0pa/ZPE-Ink	Handwriting codec workstream
ZPE-IoT	github.com/Zer0pa/ZPE-IoT	IoT telemetry codec workstream
ZPE-Mocap	github.com/Zer0pa/ZPE-Mocap	Motion capture codec workstream
ZPE-Neuro	github.com/Zer0pa/ZPE-Neuro	Neural signal codec workstream
ZPE-Prosody	github.com/Zer0pa/ZPE-Prosody	Speech prosody codec workstream
ZPE-Robotics	github.com/Zer0pa/ZPE-Robotics	Robotics codec workstream
ZPE-XR	github.com/Zer0pa/ZPE-XR	XR spatial codec workstream
Package	code/README.md	Installable zpe-multimodal package
Proof corpus	proofs/	Evidence and benchmark artifacts

Who This Is For

IMC is not the first buying surface. Domain repos are the entry points:

Domain need	Start here
Industrial sensor compression	ZPE-IoT
Robot motion telemetry	ZPE-Robotics
Trajectory / fleet / AIS	ZPE-Geo
XR hand-pose transport	ZPE-XR
Financial time-series	ZPE-FT

IMC matters when you need multi-modal dispatch — a single pipeline encoding heterogeneous signal types with deterministic semantics across domains.

---

Lane Status

Workstream-level status for the IMC platform and related codec workstreams.

Workstream	Status
IMC Wave-1	GO (7/7 phase gates PASS; 52/52 regression PASS)
IoT Wave-1	READY_FOR_USER_RATIFICATION (27/27 strict DT PASS)
Bio Wave-1	GO (RC rehearsal: 38 tests passed)
Sector board	GO_QUALIFIED=6, INCONCLUSIVE=1, NO_GO/FAIL=3
Tokenizer	INCONCLUSIVE_FOR_DEPLOYMENT

844-Word Canonical Breakdown

The 844-word count comes from the Wave-1 demo run which streams all 10 modalities. This table shows how it has been verified at multiple checkpoints.

Checkpoint	Word Count	Evidence
Runtime snapshot anchor	844	172/174 tests, determinism hash match
Post-lane integration anchor	844	62/62 regression PASS
Family contract freeze	844	wave1.0 metric authority
CLI surface (non-canonical)	780	Tracked split; demo path remains authority

Per-Lane Verification

Lane	Verification	Key Metric
TEXT_EMOJI	pytest=9 passed; determinism cases 12	Mixed-stream text count 52
DIAGRAM_IMAGE	pytest=16 passed; mean distance 0.44–0.79	Enhancement PSNR 45.95 dB
MUSIC	Events 4; packed words 34	Mixed-stream music count 42
VOICE	all_pass=true; replay all_same=true	Mixed-stream voice count 70
MENTAL	pytest=28 passed	Mixed-stream mental count 7
TOUCH	pytest=20 passed	Raw:549 → ZPE:87
SMELL	Comparator cases 116	Mixed-stream smell count 6
TASTE	Merged unique InChIKey 13510; anchor cases 6	Mixed-stream taste count 29

---

Open Risks (Non-Blocking)

• CLI and demo surfaces report different stream counts (780 vs 844 words); canonical authority remains the demo path at 844.
• Optional audio dependency chain may fail on Python 3.14; Python 3.11/3.12 is the practical baseline.
• Some scripts and docs still include machine-absolute paths and need portability cleanup.
• Taste regression coverage contains 2 failing tests tied to legacy hardcoded paths.
• Legal text finalization is pending owner-supplied content in LICENSE.

Risk	Current State
Authority metric	844 canonical
Audio toolchain	Python 3.11/3.12 baseline
Path portability	Cleanup pending
Taste regression	2 failing tests remain
Legal text	Owner-supplied content pending

Contributing, Security, Support

• Contribution workflow: CONTRIBUTING.md
• Security policy and reporting: SECURITY.md
• User support channel guide: docs/SUPPORT.md
• Frequently asked questions: docs/FAQ.md
• Contact: [e-mail]