MediaRef

The portable frame-level media reference primitive — container-agnostic, fps-free, RFC-based.

(uri, pts_ns) is the entire schema. URIs follow RFC 3986 (with RFC 2397 for embedded data); pts_ns is an int64 nanosecond presentation timestamp. The schema is frozen for the life of MediaRef Spec 1.x. Works in any container (Parquet, mcap, rosbag, HDF5) and any standard media format (JPEG, PNG, H.264, H.265, AV1).

Quick Start

from mediaref import MediaRef, DataURI, batch_decode
import numpy as np

# 1. Create references — local file, HTTP(S), cloud, or video frame.
ref = MediaRef(uri="image.png")
ref = MediaRef(uri="https://example.com/image.jpg")
ref = MediaRef(uri="s3://bucket/image.jpg")             # any fsspec scheme
ref = MediaRef(uri="video.mp4", pts_ns=1_000_000_000)   # frame at 1.0s

# 2. Load.
rgb = ref.to_ndarray()      # (H, W, 3) RGB
pil = ref.to_pil_image()

# Private storage and either video backend use the same fsspec path.
frame = MediaRef(uri="s3://bucket/video.mp4", pts_ns=0).to_ndarray(
    decoder="torchcodec",
    storage_options={"anon": False},
)

# 3. Embed bytes inside a MediaRef (self-contained reference).
ref = MediaRef(uri=DataURI.from_image(rgb, format="png"))

# 4. Batch-decode many frames from one video — opens the container once.
refs = [MediaRef(uri="video.mp4", pts_ns=int(i*1e9)) for i in range(10)]
frames = batch_decode(refs)

# 5. Serialize for storage in any string-based format.
json_str = ref.model_dump_json()   # '{"uri":"...","pts_ns":...}'

See API Reference for full details — DataURI, batch_decode, cloud URIs, HuggingFace datasets integration, lerobot interop, the mediaref CLI.

Why MediaRef?

1. Separate heavy media from lightweight metadata. Store 1 TB of videos separately and keep only a few KB of references in your tables. MediaRef is decoupled, format-agnostic, and works wherever you can store a string. Already used in production: the D2E research project stores 1 TB+ of gameplay data referenced by MediaRef via OWAMcap.

2. Permanent schema built on RFCs. (uri, pts_ns) is frozen for the life of Spec 1.x. No proprietary formats, no breaking changes.

3. Sparse-frame batch decoding. When loading many frames from a single video, batch_decode() opens the container once and seeks monotonically — historical PyAV measurements showed 4.9× faster decoding throughput and 2.2× better I/O efficiency vs per-frame decoding on a sparse-frame ML dataloader workload. Methodology: D2E paper Section 3 / Appendix A.

Installation

pip install mediaref                  # core: image loading + cloud-storage URIs (fsspec)
pip install 'mediaref[video]'         # + TensorCodec: default CPU video decoding, no Torch/PyAV
pip install 'mediaref[pyav]'          # legacy PyAV backend; select decoder='pyav'
pip install 'mediaref[torchcodec]'    # + TorchCodec 0.16+ image/video (Python 3.10+)
pip install 'mediaref[hf]'            # + HuggingFace datasets feature registration
pip install 'mediaref[video,torchcodec,hf]'  # video + image + HF extras

For uv: uv add 'mediaref[video,torchcodec,hf]'. MediaRef follows semantic versioning; the wire schema (uri, pts_ns) is frozen for the life of Spec 1.x.

Default video backend: TensorCodec. ref.to_ndarray() and batch_decode(refs) use TensorCodec's CPU playback selection and NumPy output. TensorCodec 0.1.4+ bundles FFmpeg in Linux (glibc 2.17+) and macOS 14+ wheels for x86_64/ARM64. Other platforms require a source build with Rust and FFmpeg 7 headers, or the optional PyAV backend: install mediaref[pyav] and pass decoder="pyav". MediaRef's native grayscale/depth path still uses PyAV. Historical benchmark numbers above were measured with PyAV, not TensorCodec.

For time-based windows, TensorCodec's opt-in timestamp mode avoids the initial full packet scan and selects frames by actual PTS:

frames = batch_decode(refs, decoder_options={"seek_mode": "timestamp"})

The default remains exact, matching TorchCodec. timestamp mode does not support frame-index queries; use exact when accessing the decoder by frame number.

Optional TorchCodec backend. On Python 3.10+, install mediaref[torchcodec] for the 0.16+ image API; PyAV is not required. It decodes JPEG, PNG, WebP, GIF, AVIF, and HEIC images without FFmpeg via ref.to_ndarray(image_decoder="torchcodec"). Use image_decoder_options={"output_dtype": "auto"} to preserve native high-bit-depth image data as uint16.

batch_decode(refs, decoder="torchcodec") uses TorchCodec for video on CPU. To opt into CUDA decoding, pass decoder_options={"device": "cuda"}; MediaRef moves the result back to host memory for its NumPy return type. Video decoding still requires an FFmpeg installation with shared libraries. Verify the runtime, not just the import:

python -c 'from torchcodec._core import get_ffmpeg_library_versions; print(get_ffmpeg_library_versions())'

Follow TorchCodec's official FFmpeg installation instructions first. The standalone verifier is pip install patch-torchcodec && patch-torchcodec --verify. On Linux, if verification fails and you intentionally want to reuse PyAV's bundled FFmpeg, install the optional patch dependencies with pip install 'patch-torchcodec[patch]', then run patch-torchcodec.

Documentation

• API Reference — full API: MediaRef, DataURI, batch_decode, cloud URIs, HuggingFace integration, lerobot interop, the CLI.
• MediaRef Specification 1.0 — wire format, URI grammar, pts_ns semantics, conformance criteria.
• Comparisons — how MediaRef relates to datasets.Video and lerobot's VideoFrame.
• Playback Semantics — how frame selection works at specific timestamps.

Examples

• ROS bag conversion — convert ROS1/ROS2 bags with CompressedImage topics to MediaRef-referenced video, recovering 70–90% storage via inter-frame compression. Works without a ROS install (uses rosbags).

Datasets shipped with MediaRef

These are projects from the author's own work that use MediaRef on the storage path. External adopters welcome — open a PR to add yours.

Tagging a HuggingFace dataset with mediaref makes it discoverable at huggingface.co/datasets?other=mediaref.

Citation

If you reference MediaRef in writing, the CITATION.cff file at repo root has the canonical metadata. BibTeX:

@software{mediaref,
  author  = {Choi, Suhwan},
  title   = {MediaRef: a portable frame-level media reference primitive},
  version = {1.0.0},
  year    = {2026},
  doi     = {10.5281/zenodo.19892316},
  url     = {https://github.com/open-world-agents/MediaRef}
}

The doi above is the Zenodo concept DOI — it always resolves to the latest published release. To cite v1.0.0 specifically, use 10.5281/zenodo.19892317.

Acknowledgments

The video decoder interface design references TorchCodec's API design.

License

MediaRef is released under the MIT License.