ModalityCheck
Test the tests that protect your multimodal AI pipelines.
ModalityCheck is a lightweight, zero-dependency Python library for mutation testing and fault injection in multimodal AI/ML data pipelines (video, audio, text, sensor streams).
Why Use ModalityCheck?
In multimodal AI systems (like Video-LLMs, Speech-Vision assistants, or robotics perception), data pipelines combine disparate media streams that easily fall out of sync:
- Hardware latency or Bluetooth lag causes audio to drift hundreds of milliseconds behind video.
- Sensor dropouts or network glitches cause a required modality (e.g. depth map, subtitles) to silently drop.
- Payloads become empty or truncated during batch ingestion.
When fed corrupted or misaligned data, multimodal AI models fail silently: attention mechanisms hallucinate, cross-attention weights degrade, and model accuracy drops without throwing an explicit crash.
Data engineers write validators to guard against these issues. But how do you know your validators actually work?
ModalityCheck tests your validators. It acts like a "fire drill" for your pipeline by deliberately injecting controlled faults into your data and verifying that your validators catch every single defect.
Where Should You Use This?
| Stage | Where It Fits | Benefit |
|---|---|---|
| CI/CD Pipelines | Run as part of your automated test suite alongside pytest. |
Prevents regressions in your data validation rules before deploying new pipeline code. |
| Data Ingestion Guards | Place validators in streaming workers (Kafka, AWS SQS, GCP Pub/Sub). | Quarantines corrupted or desynchronized samples before they reach model inference. |
| Dataset Curation & Pretraining | Audit multimodal training datasets (WebVid, Ego4D, AudioSet, etc.). | Identifies edge cases and blind spots in dataset filtering scripts. |
| Edge & Robotics Systems | Test timestamp synchronization between cameras, LiDAR, and IMU sensors. | Guarantees multi-sensor alignment within sub-second tolerance thresholds. |
How It Works: The Workflow
[ Healthy Multimodal Sample ]
(Video + Audio + Text)
│
▼
┌───────────────────────────┐
│ ModalityCheck │
│ Fault Scenarios │
└─────────────┬─────────────┘
│ (Injects controlled mutations:
│ dropped audio, 400ms time drift)
▼
[ Mutated / Corrupted Sample ]
│
▼
┌───────────────────────────┐
│ Your Pipeline Guards │
│ (Required, Skew, etc.) │
└─────────────┬─────────────┘
│
┌───────────┴───────────┐
▼ ▼
[ CAUGHT? ] [ MISSED? ]
│ │
▼ ▼
Report: 100% Score Flags Pipeline Blind Spot!
Safe to feed model (Tighten validation rules)
How Does It Look?
Running ModalityCheck produces a clear, structured mutation report:
======================================================================
REAL-WORLD DEMO: MODALITYCHECK MUTATION TESTING
======================================================================
1. Testing clean sample in production pipeline:
Status: success
Result: Model successfully analyzed clip_104.mp4 with audio_104.flac
2. Running ModalityCheck fault injection (testing the pipeline's guards):
- Scenarios Tested: 3
- Defects Caught: 3
- Detection Rate: 100%
Case #1: [CAUGHT]
Scenario: drop_modality
Expected issue code: missing_modality
Reported issues: ["missing_modality: Required modality 'audio' is missing."]
Case #2: [CAUGHT]
Scenario: shift_timestamp
Expected issue code: timestamp_skew
Reported issues: ['timestamp_skew: Timestamp skew 0.420000s exceeds limit 0.150000s.']
Case #3: [CAUGHT]
Scenario: shift_timestamp
Expected issue code: timestamp_skew
Reported issues: ['timestamp_skew: Timestamp skew 0.580000s exceeds limit 0.150000s.']
3. Demonstrating that corrupted data is prevented from reaching the AI model:
Status: REJECTED_BY_PIPELINE
Errors: ['[timestamp_skew] Timestamp skew 0.420000s exceeds limit 0.150000s.']
Protected AI model from processing out-of-sync audio!
======================================================================
Quickstart Guide
1. Installation
Create a virtual environment and install the package:
python -m venv .venv
# On Linux/macOS:
source .venv/bin/activate
# On Windows:
# .venv\Scripts\activate
python -m pip install -e ".[dev]"
2. Basic Example
from modalitycheck import (
DropModality,
ModalityData,
MultimodalSample,
RequiredModalities,
ShiftTimestamp,
TimestampSkew,
evaluate_mutations,
)
# 1. Define your sample (lightweight payload references without heavy media loading)
sample = MultimodalSample(
sample_id="clip-001",
modalities={
"video": ModalityData(value="s3://bucket/video.mp4", timestamp=1.00),
"audio": ModalityData(value="s3://bucket/audio.wav", timestamp=1.05),
"text": ModalityData(value="A person opens a door.", timestamp=1.02),
},
)
# 2. Define your pipeline's validation rules
validators = [
RequiredModalities({"video", "audio", "text"}),
TimestampSkew(max_skew_seconds=0.10), # Max 100ms drift allowed
]
# 3. Define the fault scenarios to test
scenarios = [
DropModality(modality="audio"),
ShiftTimestamp(modality="video", seconds=0.50), # 500ms video lag
]
# 4. Evaluate whether your validators catch the faults
report = evaluate_mutations(
samples=[sample],
scenarios=scenarios,
validators=validators,
)
print(f"Detection rate: {report.detection_rate:.0%}")
for case in report.cases:
status = "CAUGHT" if case.detected else "MISSED"
print(f"[{status}] Scenario '{case.scenario}' -> Expected: {case.expected_issue_code}")
Writing Custom Validators and Scenarios
ModalityCheck is designed to be easily extensible using Python Protocol interfaces:
Custom Validator Example
from modalitycheck import Issue, MultimodalSample, ValidationResult
class NonEmptyPayloads:
"""Ensure no modality contains an empty or null payload."""
def validate(self, sample: MultimodalSample) -> ValidationResult:
issues = []
for name, data in sample.modalities.items():
if not data.value:
issues.append(
Issue(
code="empty_payload",
message=f"Modality '{name}' is empty.",
sample_id=sample.sample_id,
modality=name,
)
)
return ValidationResult(tuple(issues))
Custom Fault Scenario Example
from dataclasses import dataclass
from modalitycheck import ModalityData, MultimodalSample
@dataclass(frozen=True)
class CorruptPayload:
"""Simulates payload corruption in a target modality."""
modality: str
name: str = "corrupt_payload"
expected_issue_code: str = "empty_payload"
def apply(self, sample: MultimodalSample) -> MultimodalSample:
modalities = dict(sample.modalities)
if self.modality in modalities:
modalities[self.modality] = ModalityData(value="") # wipe payload
return sample.with_modalities(modalities)
In-Depth Documentation
Explore the comprehensive guides in the docs/ directory:
- 🏛️ Architecture & Internal Design: Zero-copy memory model, structural subtyping (PEP 544), mutation testing theory, and immutability guarantees.
- 🔧 Troubleshooting & Common Pitfalls: Debugging 0% detection rates, floating-point timestamp precision, Mypy protocols, and common gotchas.
- 📖 API Reference: Complete specification of all classes, methods, parameters, and return types.
Running the Real-World Pipeline Demo
A complete end-to-end simulation showing an audio-visual AI model guarded by an ingestion pipeline is included in docs/examples/real_world_pipeline.py:
python docs/examples/real_world_pipeline.py
Development & Publishing
Run all quality checks and verify the package build:
# Run unit tests
pytest
# Static type checking (Mypy strict mode)
mypy src tests docs/examples
# Code formatting and linting
ruff check .
# Build distribution wheels & source archive
python -m build
# Validate package metadata for PyPI
python -m twine check dist/*
Roadmap
- Add built-in validators for duplicate IDs, empty payloads, and monotonic timestamps.
- Add property-based testing (Hypothesis) and mutation coverage reports.
- Add optional lightweight adapters for popular dataset formats (HuggingFace
datasets, WebDataset). - Provide standard CI/CD GitHub Action templates for pipeline validation.
License
MIT. See LICENSE.
Metadata
Release files for modalitycheck 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| modalitycheck-0.1.0.tar.gz | 19.8 kB | Without provenance | Details for modalitycheck-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| modalitycheck-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for modalitycheck-0.1.0-py3-none-any.whl |
Total release size: 33.6 kB
Release files / modalitycheck-0.1.0.tar.gz
| Download URL | modalitycheck-0.1.0.tar.gz |
|---|---|
| Size | 19.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
82d56557a76abc25900a71c8430e62bddb40db75a033d97e934996647abbf630
|
|
BLAKE2b-256 checksum How to use checksums |
879f4880aad97d9dd0d5b1ade6b8d424782313787953e8616a7f4350fd14928d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.0
|
Release files / modalitycheck-0.1.0-py3-none-any.whl
| Download URL | modalitycheck-0.1.0-py3-none-any.whl |
|---|---|
| Size | 13.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
7fe89c947117ba081a85d719c637b9314f06fae91881b4f72694e9612429a394
|
|
BLAKE2b-256 checksum How to use checksums |
86cdb84948327ddf43e17b11d21a93dc07eff06a70fce5f1b90f163d1ac27bb7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.0
|
Security for modalitycheck 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.