multimodal-mllog 1.2.1

Experiment logbook pipeline: automatic per-run capture + on-demand reporting

multimodal-mllog

Experiment logbook pipeline for researchers training ML models with coding agents.

An external observer that automatically captures per-run facts (agent events, git state, optional MLflow info) as JSON records via a sensor/adapter/sink architecture, and on demand renders a markdown logbook from them.

coding agent  -->  mllog capture (automatic via hook)  -->  local JSON store
researcher    -->  /logbook (manual)                    -->  markdown logbook

Architecture

sensors (data producers) --> core (deterministic) --> sinks (optional side effects)

  Sensors:   agent (always), git, env, mlflow (optional)
  Adapters:  Claude Code JSONL --> AgentEvent normal form
  Core:      validate, store (fsync), query, project to markdown
  Sinks:     mlflow_attach (runs after write+fsync, failures warn)

Record envelope (./mllog/records/<YYYY-MM-DD>/<record_id>.json):

{
  "schema_version": "0.1",
  "record_id": "<ULID>",
  "activity_type": "train | eval | analysis | attempt_failed",
  "status": "ok | failed",
  "started_at": "<iso8601>",
  "ended_at": "<iso8601>",
  "comparison_safety": { "code_touched": [], "config_touched": [], "safe_for_delta": true },
  "sources": {
    "agent":  { "event_count": 12, "edit_ledger": {}, "config_deltas": [], "commands": [] },
    "git":    { "commit": "abc123", "dirty": false, "changed_files": [] },
    "mlflow": { "run_id": "...", "params": {}, "metrics": {} },  // present only if active
    "env":    { "python_version": "3.13", "platform": "..." }
  },
  "digest": null,           // optional, evidence-gated
  "transcript_ref": null    // pointer to session JSONL, never raw content
}

Installation

pip install multimodal-mllog              # core (capture, query, render)
pip install "multimodal-mllog[mlflow]"    # + optional MLflow sensor/sink

Quickstart

1. Open your ML project in a coding agent (Claude Code, Codex, etc.).
2. Run your experiment: train, evaluate, or analyze.
3. The Stop hook captures the run automatically, or run /mllog manually.
4. Run /logbook --from yesterday to generate a logbook from stored records.

CLI reference

# Capture a run (sensors gather git, agent events, optional MLflow automatically)
mllog capture --type {train|eval|analysis|attempt_failed} --status {ok|failed} \
              [--session-path <path>] [--mlflow-run-id <id>] [--auto]

# Query / render records
mllog get-logs --from <when> [--to <when>] [--json]
mllog get-logs --from <when> [--to <when>] --render [--out <path>]
# <when>: ISO date (2026-07-01), 'yesterday', 'today'

# Sensor / adapter / sink status
mllog doctor

# Checkpoint
mllog checkpoint [--show | --advance]

Storage

• Records: ./mllog/records/<YYYY-MM-DD>/<record_id>.json (one per run, fsynced)
• Checkpoint: ./mllog/checkpoint.json
• Logbooks: ./mllog/logbooks/<from>_<to>.md
• Override root with MLLOG_DIR env var.

Development

git clone https://github.com/jean-johnson-zwix/multimodal_mllogger
cd multimodal_mllogger
make install         # create venv + install deps
make test            # run pytest
make lint            # check core purity (no banned imports)

Module layout

src/mllog/
  cli.py                    # entrypoint
  core/
    schema.py               # Record, ComparisonSafety, Digest, TranscriptRef
    store.py                # date-partitioned JSON, fsync, time-window queries
    project.py              # Record[] -> markdown (pure, no I/O)
    events.py               # AgentEvent discriminated union
    extract/                # pure functions over list[AgentEvent]
      edits.py  config.py  commands.py  safety.py
  sensors/                  # data producers (entry points)
    agent.py  git.py  mlflow.py  env.py
  adapters/                 # vendor JSONL -> AgentEvent
    claude_code.py  codex.py
  sinks/                    # post-write side effects (entry points)
    mlflow_attach.py

Roadmap

• v1.0: MLflow-centric pipeline with /mllog command.
• v1.1: Local JSON store, automatic capture via hooks, /logbook reporting.
• v1.2 (current): Sensor/adapter/sink architecture, AgentEvent normal form, deterministic extractors, comparison safety, evidence-gated digests, entry points.

License

MIT