Skip to main content

ModalityCheck

Test the tests that protect your multimodal AI pipelines.

Python 3.10+ Type Checked with mypy Linted with Ruff License: MIT

ModalityCheck is a lightweight, zero-dependency Python library for mutation testing and fault injection in multimodal AI/ML data pipelines (video, audio, text, sensor streams).


Why Use ModalityCheck?

In multimodal AI systems (like Video-LLMs, Speech-Vision assistants, or robotics perception), data pipelines combine disparate media streams that easily fall out of sync:

  • Hardware latency or Bluetooth lag causes audio to drift hundreds of milliseconds behind video.
  • Sensor dropouts or network glitches cause a required modality (e.g. depth map, subtitles) to silently drop.
  • Payloads become empty or truncated during batch ingestion.

When fed corrupted or misaligned data, multimodal AI models fail silently: attention mechanisms hallucinate, cross-attention weights degrade, and model accuracy drops without throwing an explicit crash.

Data engineers write validators to guard against these issues. But how do you know your validators actually work?

ModalityCheck tests your validators. It acts like a "fire drill" for your pipeline by deliberately injecting controlled faults into your data and verifying that your validators catch every single defect.


Where Should You Use This?

Stage Where It Fits Benefit
CI/CD Pipelines Run as part of your automated test suite alongside pytest. Prevents regressions in your data validation rules before deploying new pipeline code.
Data Ingestion Guards Place validators in streaming workers (Kafka, AWS SQS, GCP Pub/Sub). Quarantines corrupted or desynchronized samples before they reach model inference.
Dataset Curation & Pretraining Audit multimodal training datasets (WebVid, Ego4D, AudioSet, etc.). Identifies edge cases and blind spots in dataset filtering scripts.
Edge & Robotics Systems Test timestamp synchronization between cameras, LiDAR, and IMU sensors. Guarantees multi-sensor alignment within sub-second tolerance thresholds.

How It Works: The Workflow

       [ Healthy Multimodal Sample ]
           (Video + Audio + Text)
                     │
                     ▼
       ┌───────────────────────────┐
       │     ModalityCheck         │
       │     Fault Scenarios       │
       └─────────────┬─────────────┘
                     │ (Injects controlled mutations:
                     │  dropped audio, 400ms time drift)
                     ▼
       [ Mutated / Corrupted Sample ]
                     │
                     ▼
       ┌───────────────────────────┐
       │   Your Pipeline Guards    │
       │   (Required, Skew, etc.)  │
       └─────────────┬─────────────┘
                     │
         ┌───────────┴───────────┐
         ▼                       ▼
    [ CAUGHT? ]             [ MISSED? ]
         │                       │
         ▼                       ▼
  Report: 100% Score      Flags Pipeline Blind Spot!
  Safe to feed model      (Tighten validation rules)

How Does It Look?

Running ModalityCheck produces a clear, structured mutation report:

======================================================================
 REAL-WORLD DEMO: MODALITYCHECK MUTATION TESTING
======================================================================

1. Testing clean sample in production pipeline:
   Status: success
   Result: Model successfully analyzed clip_104.mp4 with audio_104.flac

2. Running ModalityCheck fault injection (testing the pipeline's guards):
   - Scenarios Tested: 3
   - Defects Caught:   3
   - Detection Rate:   100%

   Case #1: [CAUGHT]
     Scenario:            drop_modality
     Expected issue code: missing_modality
     Reported issues:     ["missing_modality: Required modality 'audio' is missing."]

   Case #2: [CAUGHT]
     Scenario:            shift_timestamp
     Expected issue code: timestamp_skew
     Reported issues:     ['timestamp_skew: Timestamp skew 0.420000s exceeds limit 0.150000s.']

   Case #3: [CAUGHT]
     Scenario:            shift_timestamp
     Expected issue code: timestamp_skew
     Reported issues:     ['timestamp_skew: Timestamp skew 0.580000s exceeds limit 0.150000s.']

3. Demonstrating that corrupted data is prevented from reaching the AI model:
   Status: REJECTED_BY_PIPELINE
   Errors: ['[timestamp_skew] Timestamp skew 0.420000s exceeds limit 0.150000s.']

   Protected AI model from processing out-of-sync audio!
======================================================================

Quickstart Guide

1. Installation

Create a virtual environment and install the package:

python -m venv .venv

# On Linux/macOS:
source .venv/bin/activate
# On Windows:
# .venv\Scripts\activate

python -m pip install -e ".[dev]"

2. Basic Example

from modalitycheck import (
    DropModality,
    ModalityData,
    MultimodalSample,
    RequiredModalities,
    ShiftTimestamp,
    TimestampSkew,
    evaluate_mutations,
)

# 1. Define your sample (lightweight payload references without heavy media loading)
sample = MultimodalSample(
    sample_id="clip-001",
    modalities={
        "video": ModalityData(value="s3://bucket/video.mp4", timestamp=1.00),
        "audio": ModalityData(value="s3://bucket/audio.wav", timestamp=1.05),
        "text": ModalityData(value="A person opens a door.", timestamp=1.02),
    },
)

# 2. Define your pipeline's validation rules
validators = [
    RequiredModalities({"video", "audio", "text"}),
    TimestampSkew(max_skew_seconds=0.10),  # Max 100ms drift allowed
]

# 3. Define the fault scenarios to test
scenarios = [
    DropModality(modality="audio"),
    ShiftTimestamp(modality="video", seconds=0.50),  # 500ms video lag
]

# 4. Evaluate whether your validators catch the faults
report = evaluate_mutations(
    samples=[sample],
    scenarios=scenarios,
    validators=validators,
)

print(f"Detection rate: {report.detection_rate:.0%}")
for case in report.cases:
    status = "CAUGHT" if case.detected else "MISSED"
    print(f"[{status}] Scenario '{case.scenario}' -> Expected: {case.expected_issue_code}")

Writing Custom Validators and Scenarios

ModalityCheck is designed to be easily extensible using Python Protocol interfaces:

Custom Validator Example

from modalitycheck import Issue, MultimodalSample, ValidationResult

class NonEmptyPayloads:
    """Ensure no modality contains an empty or null payload."""

    def validate(self, sample: MultimodalSample) -> ValidationResult:
        issues = []
        for name, data in sample.modalities.items():
            if not data.value:
                issues.append(
                    Issue(
                        code="empty_payload",
                        message=f"Modality '{name}' is empty.",
                        sample_id=sample.sample_id,
                        modality=name,
                    )
                )
        return ValidationResult(tuple(issues))

Custom Fault Scenario Example

from dataclasses import dataclass
from modalitycheck import ModalityData, MultimodalSample

@dataclass(frozen=True)
class CorruptPayload:
    """Simulates payload corruption in a target modality."""

    modality: str
    name: str = "corrupt_payload"
    expected_issue_code: str = "empty_payload"

    def apply(self, sample: MultimodalSample) -> MultimodalSample:
        modalities = dict(sample.modalities)
        if self.modality in modalities:
            modalities[self.modality] = ModalityData(value="")  # wipe payload
        return sample.with_modalities(modalities)

In-Depth Documentation

Explore the comprehensive guides in the docs/ directory:

  • 🏛️ Architecture & Internal Design: Zero-copy memory model, structural subtyping (PEP 544), mutation testing theory, and immutability guarantees.
  • 🔧 Troubleshooting & Common Pitfalls: Debugging 0% detection rates, floating-point timestamp precision, Mypy protocols, and common gotchas.
  • 📖 API Reference: Complete specification of all classes, methods, parameters, and return types.

Running the Real-World Pipeline Demo

A complete end-to-end simulation showing an audio-visual AI model guarded by an ingestion pipeline is included in docs/examples/real_world_pipeline.py:

python docs/examples/real_world_pipeline.py

Development & Publishing

Run all quality checks and verify the package build:

# Run unit tests
pytest

# Static type checking (Mypy strict mode)
mypy src tests docs/examples

# Code formatting and linting
ruff check .

# Build distribution wheels & source archive
python -m build

# Validate package metadata for PyPI
python -m twine check dist/*

Roadmap

  1. Add built-in validators for duplicate IDs, empty payloads, and monotonic timestamps.
  2. Add property-based testing (Hypothesis) and mutation coverage reports.
  3. Add optional lightweight adapters for popular dataset formats (HuggingFace datasets, WebDataset).
  4. Provide standard CI/CD GitHub Action templates for pipeline validation.

License

MIT. See LICENSE.

Metadata

Release files for modalitycheck 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for modalitycheck 0.1.0
File Size Uploaded Provenance Actions
modalitycheck-0.1.0.tar.gz 19.8 kB Without provenance Details for modalitycheck-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for modalitycheck 0.1.0
File Interpreter ABI Platform Actions
modalitycheck-0.1.0-py3-none-any.whl Python 3 none any Details for modalitycheck-0.1.0-py3-none-any.whl

Total release size: 33.6 kB

Release files / modalitycheck-0.1.0.tar.gz

Download URL modalitycheck-0.1.0.tar.gz
Size 19.8 kB
Tags Source
SHA-256 checksum
How to use checksums
82d56557a76abc25900a71c8430e62bddb40db75a033d97e934996647abbf630
BLAKE2b-256 checksum
How to use checksums
879f4880aad97d9dd0d5b1ade6b8d424782313787953e8616a7f4350fd14928d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.0

Release files / modalitycheck-0.1.0-py3-none-any.whl

Download URL modalitycheck-0.1.0-py3-none-any.whl
Size 13.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
7fe89c947117ba081a85d719c637b9314f06fae91881b4f72694e9612429a394
BLAKE2b-256 checksum
How to use checksums
86cdb84948327ddf43e17b11d21a93dc07eff06a70fce5f1b90f163d1ac27bb7
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.0

Security for modalitycheck 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page