mllm-annotator

A small, resumable tool for sending folders of audio or image files to a multimodal LLM for automatic annotation, plus an embedding + UMAP visualization workflow. Gemini is the current backend; the design keeps the provider behind a thin seam so others can be added later.

It ships both a command-line tool and a desktop GUI.

Install

# CLI only
pip install mllm-annotator

# with the desktop GUI and the embed/visualize feature
pip install "mllm-annotator[ui,viz]"

Or, for development from a clone:

uv sync --extra ui --extra viz

The embed/visualize feature also needs ffmpeg on your PATH to handle audio formats Gemini can't embed directly (e.g. .aac, .opus). It is an optional system dependency, not a pip package; without it those files are skipped.

API key

Provide a Gemini API key in any one of these ways (checked in this order):

1. environment variable GEMINI_API_KEY (or GOOGLE_API_KEY):

   $env:GEMINI_API_KEY="your_api_key"
   

2. a .env file in the current working directory:

   GEMINI_API_KEY=your_api_key
   

3. saved from inside the GUI — click Set API Key in the top-left corner (it also opens automatically on first launch if no key is found), paste the key, and it is stored securely in your OS keyring (Windows Credential Manager / macOS Keychain / Linux Secret Service). No plaintext file is written.

.env is ignored by git, and keys are never written into the built package.

Command line

mllm-annotator --help

Examples

Horse cough annotation:

mllm-annotator `
  --input-folder "C:\data\horse_audio" `
  --media-type audio `
  --instruction "Annotate if the audio contains a horse cough or another sound such as the horse smacking the microphone." `
  --daily-limit 500

Swiss German transcription validation:

mllm-annotator `
  --input-folder "C:\data\swiss_german_audio" `
  --media-type audio `
  --labels-csv "C:\data\transcriptions.csv" `
  --instruction "Confirm whether the attached Swiss German audio matches the associated transcription. If it is wrong, rewrite the correct transcription." `
  --daily-limit 500

Image captioning:

mllm-annotator `
  --input-folder "C:\data\images" `
  --media-type image `
  --instruction "Caption the attached image." `
  --daily-limit 500

Desktop GUI

mllm-annotator-ui

The GUI lets you browse for the data folder, choose audio or image mode, optionally select a CSV, write the instruction, and preview the file table.

The Annotation Spec panel drives the run in two steps:

1. Propose with Gemini — Gemini designs a rewritten instruction, the CSV columns to use, and the output fields (with any enum labels). They appear in an editable form: a rewritten-instruction box and one row per output field (name, type, allowed values, description). With "Rewrite prompt" unchecked, a default spec is proposed without an API call.
2. Approve & Build Table — once you are happy with the (optionally edited) spec, this builds the Results table around your output fields.

Then Start / Resume processes the files using the approved spec, updating each row as Gemini responses arrive and writing the same JSONL result/state files as the CLI. Resuming a previous run loads and re-uses its stored spec automatically. A second tab embeds the media and shows an interactive 2-D UMAP projection (zoom/pan toolbar, hover a point for its file name). If you provide a CSV, a "Color by" dropdown lets you color the points by any of its columns; switching columns re-colors instantly using cached coordinates. Embeddings are written next to the data as {folder}_embeddings.jsonl.

CSV format

The optional labels CSV may have any number of columns; only filename is required, and every media file must have exactly one matching row. Empty cells are allowed. Other columns are passed to Gemini, which decides which to use and how based on your instruction.

filename,manual_label,model_label
audio_001.wav,cough,cough
audio_002.wav,,not_cough
audio_003.wav,snort,cough

For --recursive, filename must be the relative path with forward slashes, for example speaker_a/audio_001.wav.

Resume behavior

On the CLI, results default to runs/results.jsonl and progress to runs/state.json (override with --output-file / --state-file). The GUI instead defaults both next to the data folder ({folder}_results.jsonl / {folder}_state.json). Either way, if the daily limit is reached or the API returns a quota/rate limit, run the same command again later or the next day — already processed files are skipped.

On the first run, gemini-3.5-flash designs an annotation spec from your instruction and any CSV columns: a rewritten instruction, the columns to pass per sample, and the output fields (with enum labels when your instruction implies them). The spec is stored in the state file and reused on resume; you can edit state.json (or the GUI Spec panel) to adjust it. Media files are processed with gemini-3.1-flash-lite. Use --no-rewrite to skip the design call and use a default schema (annotation, confidence, notes) with the instruction as-is.

Example: closed-set audio labeling

Instruction:

Review these audio clips. Each clip is part of a recording of a horse. The labels CSV has a manual_label column (the sound a human annotated, may be empty) and a model_label column (a neural network's prediction of whether the clip contains a cough). Output an annotator_label chosen from cough, snort, or grunt, and a confidence between 0 and 1.

Gemini will design a spec whose output_fields includes annotator_label (string with enum_values: [cough, snort, grunt]) and confidence (number), and pass manual_label and model_label to each per-sample call.

Before spending API calls, you can validate the folder and optional CSV:

mllm-annotator `
  --input-folder "C:\data\images" `
  --media-type image `
  --instruction "Caption the attached image." `
  --dry-run