🎨 Emosaic

Convert any GIF or MP4 video into an emoji mosaic art video using FFmpeg.

Each frame is broken into a configurable grid of pixel blocks. Every block maps to the closest-matching emoji by perceptual color distance (CIE L*a*b* ΔE). The processed frames are stitched back into an MP4 with the original frame rate preserved, and optional audio passthrough for MP4 inputs.

emosaic input.mp4 output.mp4

🟥🟥🟧🟨🟩🟦🟪 ← each cell = one emoji
🟥🟫🟨🟩🟦🟪🟥
🟧🟨🟩🟦🟪🟥🟧

---

Features

• GIF & MP4 input — any format FFmpeg can read (.gif, .mp4, .mov, .mkv, .webm, .avi)
• MP4 output — H.264, yuv420p, maximum compatibility
• Audio passthrough — copies or re-encodes audio from MP4 inputs
• Perceptual color matching — CIE L*a*b* ΔE76 for accurate emoji selection
• Configurable grid — --resolution N sets emoji columns (rows auto-scale)
• Three quality modes — fast / standard / hq
• Speed control — --speed 0.5 or --speed 2.0 (adjusts both video and audio)
• Custom emoji palettes — JSON file with your own emoji + RGB definitions
• Single-frame preview — --preview outputs a PNG before committing to full render
• Progress bar — real-time frame progress with FPS counter
• Prints every FFmpeg command — copy and run manually if needed
• Zero network calls — fully offline, no API keys

---

Requirements

Dependency	Version	Notes
Python	≥ 3.9
FFmpeg	≥ 4.4	Must be on PATH
Pillow	≥ 10.0	Installed automatically
numpy	≥ 1.24	Installed automatically

Optional (strongly recommended): A system color emoji font for best visual output.

OS	Font	Install
macOS	Apple Color Emoji	Pre-installed
Windows	Segoe UI Emoji	Pre-installed
Linux	Noto Color Emoji	sudo apt install fonts-noto-color-emoji
Linux	Twemoji	Download from twitter/twemoji

Without a color emoji font, Emosaic falls back to Pillow's built-in bitmap font, which renders ASCII approximations. The output still works, but won't look as good.

---

Installation

pip install emosaic

With pipx (isolated environment)

pipx install emosaic

Verify

emosaic --help
emosaic --help

---

Usage

Basic

# Output defaults to emojioutput.mp4
emosaic input.mp4

# Explicit output name
emosaic input.mp4 output.mp4

# GIF input
emosaic animation.gif emoji_animation.mp4

Quality modes

# Fast mode: 16-column grid, minimal palette — good for quick previews
emosaic input.mp4 out.mp4 --mode fast

# Standard mode: 32-column grid, 26-emoji palette (default)
emosaic input.mp4 out.mp4 --mode standard

# High quality: 48+ column grid, 47-emoji palette
emosaic input.mp4 out.mp4 --mode hq

Grid resolution

# 16 columns = chunky / pixel-art look
emosaic input.mp4 out.mp4 --resolution 16

# 64 columns = fine mosaic, slower to render
emosaic input.mp4 out.mp4 --resolution 64

# Custom cell pixel size (overrides auto-sizing)
emosaic input.mp4 out.mp4 --resolution 32 --cell-size 24

Speed control

# Half speed (slow motion)
emosaic input.mp4 out.mp4 --speed 0.5

# Double speed (timelapse)
emosaic input.mp4 out.mp4 --speed 2.0

Audio is adjusted proportionally via atempo when speed ≠ 1.0. For speed outside the [0.5, 2.0] range, chain multiple atempo filters manually.

Audio

# Strip audio entirely
emosaic input.mp4 out.mp4 --no-audio

# Audio is preserved by default when input has audio
emosaic input_with_audio.mp4 out.mp4

Preview (single frame)

# Saves frame_001.png instead of a full video — fast feedback loop
emosaic input.mp4 frame_001 --preview

# Preview with HQ settings
emosaic input.mp4 preview --preview --mode hq --resolution 48

The output path suffix is replaced with .png automatically.

Custom emoji palette

emosaic input.mp4 out.mp4 --palette examples/nature_palette.json

Custom palette JSON format:

[
  {"emoji": "🌊", "name": "ocean-blue",   "rgb": [28,  107, 186]},
  {"emoji": "🌿", "name": "forest-green", "rgb": [68,  148, 74]},
  {"emoji": "🔥", "name": "fire-orange",  "rgb": [220, 80,  0]},
  {"emoji": "⬛", "name": "black",        "rgb": [23,  23,  23]},
  {"emoji": "⬜", "name": "white",        "rgb": [230, 230, 230]}
]

Face emojis

# Adds 😀 😐 😡 😢 🥰 to the palette (mapped to skin-tone/yellow zones)
emosaic input.mp4 out.mp4 --faces

Video quality

# CRF 0 = lossless, 51 = worst quality, 18 = default (visually near-lossless)
emosaic input.mp4 out.mp4 --crf 23

Debugging

# Keep the raw and processed frame directories in /tmp/emojiart_*/
emosaic input.mp4 out.mp4 --keep-temp

---

FFmpeg Commands Explained

Emosaic prints every FFmpeg command it runs. Here's what they mean:

Frame extraction

ffmpeg -y \
  -i input.mp4 \
  -vf "setpts=0.5000*PTS" \   # speed: 0.5 = half speed (2× more frames)
  -vsync 0 \                   # prevent duplicate/dropped frames
  -frame_pts 1 \               # embed PTS in output filenames
  /tmp/emojiart_xxx/raw_frames/frame_%06d.png

• -vf copy is used when no scaling/speed is applied
• -vsync 0 is critical for GIFs which have variable frame timing
• setpts=FACTOR*PTS adjusts timing: factor < 1 = faster, > 1 = slower

Video rebuild

ffmpeg -y \
  -framerate 25.0000 \         # output FPS (original × speed multiplier)
  -i frame_%06d.png \          # processed emoji frames
  -i input.mp4 \               # original (for audio mux, if applicable)
  -c:v libx264 \               # H.264 encoder
  -crf 18 \                    # quality (lower = better)
  -preset medium \             # encoding speed vs compression tradeoff
  -pix_fmt yuv420p \           # max player compatibility
  -movflags +faststart \       # web-optimized: moov atom at front
  -c:a copy \                  # copy audio stream (or 'aac' + atempo for speed)
  -map 0:v:0 -map 1:a:0 \     # explicit stream mapping
  -shortest \                  # end when shortest stream ends
  output.mp4

---

Architecture

emojiart/
├── __init__.py          # Package exports
├── __main__.py          # python -m emojiart entry point
├── cli.py               # Argument parsing + orchestration pipeline
├── ffmpeg_utils.py      # FFmpeg subprocess wrappers
│                           probe_video()   → metadata dict
│                           extract_frames() → PNG files
│                           rebuild_video()  → MP4 output
├── palette.py           # Color science + emoji mapping
│                           EmojiEntry       → dataclass
│                           load_palette()   → List[EmojiEntry]
│                           EmojiMapper      → RGB → emoji (CIE Lab ΔE)
└── renderer.py          # Per-frame image rendering
                            EmojiFrameRenderer.render() → PIL Image
                            render_frames_batch()        → List[Path]

Data flow

Input file
    │
    ▼
[ffprobe] probe_video()
    → fps, resolution, has_audio
    │
    ▼
[FFmpeg] extract_frames()
    → /tmp/.../raw_frames/frame_000001.png ...
    │
    ▼
[Pillow] EmojiFrameRenderer.render() × N frames
    For each cell (col × row):
        avg_color(region) → (R, G, B)
        EmojiMapper.map(R, G, B) → emoji via CIE ΔE
        draw.rectangle(fill=avg_color)
        draw.text(emoji, font=emoji_font)
    → /tmp/.../emoji_frames/frame_000001.png ...
    │
    ▼
[FFmpeg] rebuild_video()
    → output.mp4 (H.264 + audio)

Color matching algorithm

1. Source pixel block averaged to one (R, G, B) triple
2. Quantized to nearest 8 (reduces cache misses by ~8×)
3. Converted to CIE L*a*b* (perceptual color space)
4. CIE ΔE76 distance computed against every palette entry's pre-computed Lab value
5. Minimum-distance entry wins → its emoji is rendered
6. Result cached — typical video needs only ~200–500 unique cache entries

---

Performance

Resolution	Cell px	Frames/sec (CPU)	Notes
16	32	~80 fps	Fast mode, great for quick drafts
32	16	~30 fps	Standard mode default
48	16	~15 fps	HQ mode
64	12	~8 fps	Very detailed, slow

Measurements on a modern laptop (Apple M2 / Ryzen 7). Performance scales linearly with grid size (O(cols × rows) per frame).

To speed up rendering:

• Use --mode fast or a lower --resolution
• For long videos, split into segments and render in parallel (shell & or xargs)
• A GPU-accelerated emoji renderer (via torch or cupy) would be the next optimization step

---

Edge Cases & Known Limitations

Situation	Behavior
GIF with variable frame timing	FFmpeg normalizes to constant FPS via vsync 0
Input has no audio	Audio flags are ignored silently
--speed outside [0.5, 2.0]	Audio uses clamped atempo; video speed is unaffected
No emoji font on system	Falls back to Pillow bitmap font (visible ASCII-like glyphs)
Very small input (< 16px)	Grid auto-reduces to avoid zero-size cells
Unicode filenames	Fully supported on Python 3.9+ on all platforms
Transparent GIF frames	Converted to RGB (transparency → black background)
Very long video (> 30min)	Works, but may use significant disk space in /tmp for frames
CRF 0 (lossless)	Produces very large files; --crf 18 is recommended

---

Custom Palette Examples

Neon / cyberpunk

[
  {"emoji": "🟥", "name": "hot-pink",    "rgb": [255, 0, 128]},
  {"emoji": "🟦", "name": "cyber-cyan",  "rgb": [0, 255, 220]},
  {"emoji": "🟨", "name": "neon-yellow", "rgb": [220, 255, 0]},
  {"emoji": "🟪", "name": "electric-purple", "rgb": [180, 0, 255]},
  {"emoji": "⬛", "name": "void-black",  "rgb": [5, 0, 20]},
  {"emoji": "⬜", "name": "grid-white",  "rgb": [200, 220, 255]}
]

Grayscale

[
  {"emoji": "⬛", "name": "black",  "rgb": [0,   0,   0]},
  {"emoji": "🖤", "name": "dark",   "rgb": [50,  50,  50]},
  {"emoji": "🩶", "name": "gray",   "rgb": [128, 128, 128]},
  {"emoji": "🤍", "name": "light",  "rgb": [200, 200, 200]},
  {"emoji": "⬜", "name": "white",  "rgb": [255, 255, 255]}
]

---

License

MIT — see LICENSE.