QuadEmbed

Multimodal embeddings for text, image, audio, and video in one shared 768-dimensional vector space. Compare any modality against any other with a dot product.

Trained end-to-end on a single RTX 4060 laptop GPU (8GB VRAM) by freezing three pretrained encoders and training only two small projection heads, reproducing the GELATO architecture behind Jina AI's jina-embeddings-v5-omni.

⚠️ Non-commercial license. QuadEmbed builds on jina-embeddings-v5-text-nano, which is CC-BY-NC-4.0. That restriction carries through to this package and its weights. Research and educational use is fine; commercial use is not, without a separate license from Jina AI.

Install

pip install quadembed              # text + image + audio
pip install quadembed[video]       # adds video support (OpenCV)
pip install quadembed[all]         # everything, plus audio file loading helpers

Quickstart

from quadembed import QuadEmbed
from PIL import Image

model = QuadEmbed.from_pretrained()   # downloads weights from the Hub on first run

texts = ["a dog running on the beach", "a bowl of ramen noodles"]
text_embeds = model.embed_text(texts)
image_embeds = model.embed_image([Image.open("photo.jpg").convert("RGB")])

# embeddings are L2-normalized, so this is cosine similarity
similarity = text_embeds @ image_embeds.T
for text, score in zip(texts, similarity[:, 0].tolist()):
    print(f"{score:.3f}  {text}")

Audio

Pass mono float32 arrays at 16 kHz (what the frozen Whisper encoder expects):

import soundfile as sf

audio, sr = sf.read("clip.wav")
audio_embeds = model.embed_audio([audio.astype("float32")], sampling_rate=sr)
similarity = model.embed_text(["a dog barking"]) @ audio_embeds.T

Video

model = QuadEmbed.from_pretrained(modalities=("text", "video"))
video_embeds = model.embed_video_file("clip.mp4", num_frames=4)
similarity = model.embed_text(["a person skateboarding"]) @ video_embeds.T

Loading only what you need

Each encoder costs memory and download time. Load a subset:

model = QuadEmbed.from_pretrained(modalities=("text", "vision"))   # skip audio
model = QuadEmbed.from_pretrained(device="cpu")                    # force CPU

How it works

Three frozen encoders, two trained projectors:

Only ~3.3M parameters were ever trained, against nearly a billion frozen ones. Video needs no encoder or projector of its own: frames are sampled, run through the vision path, and mean-pooled over time.

Training used bidirectional in-batch InfoNCE (temperature 0.02) plus Matryoshka representation learning over prefix dims {32, 64, 128, 256, 768}, so truncated embeddings remain usable if you need a smaller index.

Measured performance

Cross-modal retrieval recall@k on held-out splits, text-query direction:

Random-chance R@1 on the 1024-candidate image eval is ~0.1%.

These are honest small-scale numbers. Vision was trained on ~172k image-caption pairs, orders of magnitude less than production embedding models, and image R@1 is the weakest metric as a result. R@5 and R@10 are considerably stronger, so this is more useful for candidate retrieval and reranking than for exact top-1 matching.

Links

• Source and training code: github.com/mithilai/QuadEmbed
• Model weights: huggingface.co/Mithil-21/quadembed-nano
• Full write-up: I Built a Multimodal Embedding Model From Scratch on an RTX 4060

Credit

QuadEmbed reproduces the architecture described in Jina AI's GELATO paper (arXiv:2605.08384). All credit for the original architecture and training recipe belongs there. This is an independent reproduction, not affiliated with or endorsed by Jina AI.