multimodal-mllog

Experiment logbook pipeline for researchers training ML models with coding agents.

An external observer that automatically captures per-run facts (agent events, git state, optional MLflow info) as JSONL records via a sensor/adapter/sink architecture, and on demand generates a PDF experiment report from them via the coding agent.

coding agent  --> mllog capture (automatic via hook)   -->  JSONL store
coding agent  --> /logbook --from yesterday            -->  PDF report
researcher    --> mllog dashboard                       -->  local web UI

Architecture

Record envelope (stored as one JSONL line per record in ./mllog/records/<YYYY-MM-DD>.jsonl):

{
  "schema_version": "0.1",
  "record_id": "<ULID>",
  "display_name": "Fix LR scheduler bug",  // short title for UI (≤60 chars)
  "activity_type": "train | eval | analysis | attempt_failed",
  "status": "ok | failed",
  "started_at": "<iso8601>",
  "ended_at": "<iso8601>",
  "sources": {
    "agent":  { "event_count": 12, "edit_ledger": {}, "config_deltas": [], "commands": [] },
    "git":    { "commit": "abc123", "dirty": false, "changed_files": [] },
    "mlflow": { "run_id": "...", "params": {}, "metrics": {} },  // present only if active
    "env":    { "python_version": "3.13", "platform": "..." }
  },
  "digest": null,           // optional, evidence-gated
  "transcript_ref": null    // pointer to session JSONL, never raw content
}

Installation

pip install multimodal-mllog              # core (capture, query)
pip install "multimodal-mllog[mlflow]"    # + optional MLflow sensor/sink
pip install "multimodal-mllog[dashboard]" # + local web dashboard (Flask + PDF rendering)

Quickstart

1. Open your ML project in a coding agent (Claude Code, Codex, etc.).
2. Run your experiment: train, evaluate, or analyze.
3. The Stop hook captures the run automatically, or run /mllog manually.
4. Run /logbook --from yesterday to generate a PDF experiment report from stored records.
5. Run mllog dashboard to browse records in a local web UI.

CLI reference

# Capture a run (sensors gather git, agent events, optional MLflow automatically)
mllog capture --type {train|eval|analysis|attempt_failed} --status {ok|failed} \
              [--name "short title"] [--session-path <path>] [--mlflow-run-id <id>] [--auto]

# Query records
mllog get-logs --from <when> [--to <when>] [--json]
# <when>: ISO date (2026-07-01), 'yesterday', 'today'

# Local dashboard
mllog dashboard [--port 8265] [--mllog-dir ./mllog]

# Sensor / adapter / sink status
mllog doctor

# Checkpoint
mllog checkpoint [--show | --advance]

Reports

Reports are agent-generated: the /logbook skill instructs the coding agent to read session records, analyze them, and produce a professional PDF report using reportlab. This produces rich narrative documents with tables, cross-run comparisons, failure analysis, and recommendations — not a mechanical data dump.

/logbook --from 2026-07-01 --to 2026-07-19

The dashboard also offers a quick summary generator for simple tabular overviews.

Storage

• Records: ./mllog/records/<YYYY-MM-DD>.jsonl (append-only, one line per record, fsynced)
• Checkpoint: ./mllog/checkpoint.json
• Logbooks: ./mllog/logbooks/<from>_<to>.pdf
• Override root with MLLOG_DIR env var.
• Override logbooks dir with MLLOG_LOGBOOKS_DIR env var.

Dashboard

A local read-only web UI for browsing records and viewing experiment details.

pip install "multimodal-mllog[dashboard]"
mllog dashboard
# opens http://localhost:8265

Two views: Sessions (table, filterable) and Record Detail (metrics, config deltas, agent activity).

Development

git clone https://github.com/jean-johnson-zwix/multimodal_mllogger
cd multimodal_mllogger
make install         # create venv + install deps
make test            # run pytest
make lint            # check core purity (no banned imports)
make frontend        # rebuild dashboard frontend

License

MIT