MediaRef

The portable frame-level media reference primitive — container-agnostic, fps-free, RFC-based.

(uri, pts_ns) is the entire schema. URIs follow RFC 3986 (with RFC 2397 for embedded data); pts_ns is an int64 nanosecond presentation timestamp. The schema is frozen for the life of MediaRef Spec 1.x. Works in any container (Parquet, mcap, rosbag, HDF5) and any standard media format (JPEG, PNG, H.264, H.265, AV1).

Quick Start

from mediaref import MediaRef, DataURI, batch_decode
import numpy as np

# 1. Create references — local file, HTTP(S), cloud, or video frame.
ref = MediaRef(uri="image.png")
ref = MediaRef(uri="https://example.com/image.jpg")
ref = MediaRef(uri="s3://bucket/image.jpg")             # any fsspec scheme
ref = MediaRef(uri="video.mp4", pts_ns=1_000_000_000)   # frame at 1.0s

# 2. Load.
rgb = ref.to_ndarray()      # (H, W, 3) RGB
pil = ref.to_pil_image()

# Private storage and either video backend use the same fsspec path.
frame = MediaRef(uri="s3://bucket/video.mp4", pts_ns=0).to_ndarray(
    decoder="torchcodec",
    storage_options={"anon": False},
)

# 3. Embed bytes inside a MediaRef (self-contained reference).
ref = MediaRef(uri=DataURI.from_image(rgb, format="png"))

# 4. Batch-decode many frames from one video — opens the container once.
refs = [MediaRef(uri="video.mp4", pts_ns=int(i*1e9)) for i in range(10)]
frames = batch_decode(refs)

# 5. Serialize for storage in any string-based format.
json_str = ref.model_dump_json()   # '{"uri":"...","pts_ns":...}'

See API Reference for full details — DataURI, batch_decode, cloud URIs, HuggingFace datasets integration, lerobot interop, the mediaref CLI.

Why MediaRef?

1. Separate heavy media from lightweight metadata. Store 1 TB of videos separately and keep only a few KB of references in your tables. MediaRef is decoupled, format-agnostic, and works wherever you can store a string. Already used in production: the D2E research project stores 1 TB+ of gameplay data referenced by MediaRef via OWAMcap.

2. Permanent schema built on RFCs. (uri, pts_ns) is frozen for the life of Spec 1.x. No proprietary formats, no breaking changes.

3. Sparse-frame batch decoding. When loading many frames from a single video, batch_decode() opens the container once and seeks monotonically — 4.9× faster decoding throughput and 2.2× better I/O efficiency vs per-frame decoding on a sparse-frame ML dataloader workload. Methodology: D2E paper Section 3 / Appendix A.

Installation

pip install mediaref                  # core: image loading + cloud-storage URIs (fsspec)
pip install 'mediaref[video]'         # + PyAV for video frame decoding
pip install 'mediaref[torchcodec]'    # + TorchCodec video backend, compatible with 0.7+
pip install 'mediaref[torchcodec-image]'  # + TorchCodec 0.16+ image/video (Python 3.10+)
pip install 'mediaref[hf]'            # + HuggingFace datasets feature registration
pip install 'mediaref[video,torchcodec-image,hf]'  # all extras

For uv: uv add 'mediaref[video,torchcodec-image,hf]'. MediaRef follows semantic versioning; the wire schema (uri, pts_ns) is frozen for the life of Spec 1.x.

Optional TorchCodec backend. On Python 3.10+, install mediaref[torchcodec-image] for the 0.16+ image API; PyAV is not required. It decodes JPEG, PNG, WebP, GIF, AVIF, and HEIC images without FFmpeg via ref.to_ndarray(image_decoder="torchcodec"). Use image_decoder_options={"output_dtype": "auto"} to preserve native high-bit-depth image data as uint16. The broader mediaref[torchcodec] extra keeps TorchCodec 0.7+ support for video-only users, including Python 3.9.

batch_decode(refs, decoder="torchcodec") uses TorchCodec for video on CPU. To opt into CUDA decoding, pass decoder_options={"device": "cuda"}; MediaRef moves the result back to host memory for its NumPy return type. Video decoding still requires an FFmpeg installation with shared libraries. Verify the runtime, not just the import:

python -c 'from torchcodec._core import get_ffmpeg_library_versions; print(get_ffmpeg_library_versions())'

Follow TorchCodec's official FFmpeg installation instructions first. The standalone verifier is pip install patch-torchcodec && patch-torchcodec --verify. On Linux, if verification fails and you intentionally want to reuse PyAV's bundled FFmpeg, install the optional patch dependencies with pip install 'patch-torchcodec[patch]', then run patch-torchcodec.

Documentation

• API Reference — full API: MediaRef, DataURI, batch_decode, cloud URIs, HuggingFace integration, lerobot interop, the CLI.
• MediaRef Specification 1.0 — wire format, URI grammar, pts_ns semantics, conformance criteria.
• Comparisons — how MediaRef relates to datasets.Video and lerobot's VideoFrame.
• Playback Semantics — how frame selection works at specific timestamps.

Examples

• ROS bag conversion — convert ROS1/ROS2 bags with CompressedImage topics to MediaRef-referenced video, recovering 70–90% storage via inter-frame compression. Works without a ROS install (uses rosbags).

Datasets shipped with MediaRef

These are projects from the author's own work that use MediaRef on the storage path. External adopters welcome — open a PR to add yours.

Tagging a HuggingFace dataset with mediaref makes it discoverable at huggingface.co/datasets?other=mediaref.

Citation

If you reference MediaRef in writing, the CITATION.cff file at repo root has the canonical metadata. BibTeX:

@software{mediaref,
  author  = {Choi, Suhwan},
  title   = {MediaRef: a portable frame-level media reference primitive},
  version = {1.0.0},
  year    = {2026},
  doi     = {10.5281/zenodo.19892316},
  url     = {https://github.com/open-world-agents/MediaRef}
}

The doi above is the Zenodo concept DOI — it always resolves to the latest published release. To cite v1.0.0 specifically, use 10.5281/zenodo.19892317.

Acknowledgments

The video decoder interface design references TorchCodec's API design.

License

MediaRef is released under the MIT License.