Codex Can See

Codex Can See is a local-first, agent-friendly video evidence tool. It extracts verifiable frames, optionally adds local transcription, asks a configured vision-language model for analysis, and emits one JSON object on stdout for agents to parse. People can inspect the same frames, transcript, report, and work directory.

How it works

plugin instructions -> non-mutating doctor -> local evidence extraction -> configured provider -> JSON + report

The plugin never contains Python source code, credentials, state, or an installer. The pinned bootstrap command installs a persistent uv-managed tool environment and records readiness under ~/.config/codex-can-see/.

Requirements

• macOS Apple Silicon, macOS x86_64, or Linux x86_64
• uv
• ffmpeg
• ffprobe
• A provider key for OpenAI or Anthropic

Windows is unsupported in v0.1.0. Linux CUDA is implemented but remains runtime-untested until validated on real NVIDIA hardware.

Direct installation

Use uvx only for first-run bootstrap:

uvx codex-can-see@0.1.0 bootstrap \
  --provider openai \
  --transcription yes

Bootstrap then performs the persistent installation itself. After setup, invoke the installed CLI directly:

codex-can-see doctor --mode invocation --json
codex-can-see analyze "$HOME/Movies/demo.mp4" \
  --between 00:10:00 00:13:00 \
  --question "What does the person do with the package?"

For a smaller frame-provider installation without local transcription:

uvx codex-can-see@0.1.0 bootstrap \
  --provider anthropic \
  --transcription no

A provider is required even when transcription is disabled because Codex Can See always uses frame-backed provider analysis.

Non-interactive key input

Export the selected provider key before bootstrap:

read -r -s OPENAI_API_KEY
export OPENAI_API_KEY
uvx codex-can-see@0.1.0 bootstrap --provider openai --transcription yes
unset OPENAI_API_KEY

Alternatively, pipe exactly one key:

printf '%s\n' "$OPENAI_API_KEY" | \
  uvx codex-can-see@0.1.0 bootstrap \
    --provider openai \
    --transcription yes \
    --key-stdin

The key is stored only in ~/.config/codex-can-see/.env. It is not placed in command arguments, state, runtime configuration, logs, stdout, or the plugin.

Install as a Codex skill plugin

This is a three-step process: register the catalog, install the plugin payload, then bootstrap the Python runtime.

# 1. Register the marketplace/catalog
codex plugin marketplace add monodeepdas1215/codex-can-see --ref v0.1.0

# 2. Install the actual plugin
codex plugin add codex-can-see@codex-can-see-marketplace

# 3. Bootstrap the persistent runtime
uvx codex-can-see@0.1.0 bootstrap \
  --provider openai \
  --transcription yes

Install as a Claude Code skill plugin

# 1. Register the marketplace/catalog
claude plugin marketplace add monodeepdas1215/codex-can-see

# 2. Install the actual plugin
claude plugin install codex-can-see@codex-can-see-marketplace

# 3. Bootstrap the persistent runtime
uvx codex-can-see@0.1.0 bootstrap \
  --provider anthropic \
  --transcription yes

The installed skill begins with codex-can-see doctor --mode invocation --json. It does not install dependencies, download models, mutate state, or ask for credentials during analysis.

Runtime profiles

Platform/profile	Local runtime	Fixed model	Package
Darwin/arm64	MLX Whisper	mlx-community/whisper-small-mlx	codex-can-see[full-mlx]
Darwin/x86_64	WhisperX CPU	small	codex-can-see[full-cpu]
Linux/x86_64 CPU	WhisperX CPU	small	codex-can-see[full-cpu]
Linux/x86_64 CUDA 12.6+	WhisperX CUDA	small	codex-can-see[full-cuda]
Transcription disabled	none	none	codex-can-see

Whisper model selection is fixed and intentionally not a public setting. A disabled transcription profile performs no model warm-up and checks no model cache, but frame extraction and provider analysis continue.

URL sources use the yt-dlp executable installed inside the persistent Codex Can See tool environment. Local files do not make network requests.

Command interface

codex-can-see analyze SOURCE \
  (--start TIME --end TIME | --at TIME | --after TIME | --before TIME | --between TIME TIME) \
  [--question TEXT] \
  [--out-dir DIR]

Only one time-filter form is allowed:

Form	Window
--start 10:00 --end 13:00	exactly [10:00, 13:00]
--between 10:00 13:00	[10:00, 13:00]
--after 10:00	[10:00, end-of-media]
--before 13:00	[media-start, 13:00]
--at 14:32	[14:32, 14:37]

Timestamps remain on the original media timeline.

--out-dir is request-specific:

• missing directories, including parents, are created;
• an existing empty directory is accepted;
• a non-directory is rejected;
• a non-empty directory is rejected to prevent stale artifact mixing;
• work_dir is always an absolute path.

Unknown arguments use normal command-line usage errors. Provider, model, frame count, frame quality, frame dimensions, and output format are setup-time settings, not analyze arguments.

Readiness and exit codes

Always start with:

codex-can-see doctor --mode invocation --json

The doctor checks state, package version, runtime configuration, provider/key presence, ffmpeg, ffprobe, yt-dlp, backend/profile consistency, and fixed model cache when transcription is enabled. It performs no installation, provider request, model download, state repair, or prompt.

0  ready/success
2  usage or provider resolution error
3  state/schema/package mismatch or incomplete setup
4  missing ffmpeg, ffprobe, or yt-dlp
5  provider configuration error
6  backend/profile mismatch
7  bootstrap/runtime installation failure
8  media/evidence processing failure

Provider failure during analysis is different from a setup error: Codex Can See preserves local evidence, writes the report, emits valid JSON, sets answer: null, adds a redacted warning, and exits 0.

Provider and frame overrides

Bootstrap accepts one non-secret YAML override:

schema: codex-can-see/runtime-override/v1
provider: openai

auth:
  env: EXAMPLE_API_KEY
endpoint:
  base_url: https://replacement.example/v1
model:
  default: compatible-model

frames:
  count: 16
  quality: 0.80
  max_dimension: 1280

Install it with:

export EXAMPLE_API_KEY="..."
uvx codex-can-see@0.1.0 bootstrap \
  --provider openai \
  --transcription yes \
  --provider-overrides ./runtime-override.yaml

Allowed bounds are:

frames.count          1..100
frames.quality        0.0..1.0
frames.max_dimension  16..4096

Defaults are 12, 0.75, and 1024. A custom auth.env is a variable name, never a secret value.

Private configuration

Readiness and profile metadata are stored in ~/.config/codex-can-see/state.toml.

~/.config/codex-can-see/
├── state.toml
├── runtime.json
├── provider-overrides.json
└── .env

The directory mode is 0700; files are mode 0600 and are replaced atomically. state.toml contains no key values. Switching providers writes the newly selected key and removes the previous provider's saved key.

Output contract

stdout contains exactly one JSON object. Diagnostics go to stderr.

{
  "source": {
    "url": "/absolute/path/demo.mp4",
    "duration_sec": 1200.0,
    "title": "demo.mp4"
  },
  "analysis_window": {
    "requested_form": "between",
    "start_seconds": 600.0,
    "end_seconds": 780.0
  },
  "answer": {
    "provider": "openai",
    "model": "gpt-4o",
    "text": "The person lifts the package..."
  },
  "evidence": {
    "frames": [
      {
        "path": "/absolute/path/frames/frame_000.jpg",
        "timestamp_sec": 600.0
      }
    ],
    "transcript": [],
    "transcript_source": "none",
    "transcript_status": "disabled",
    "transcript_error": null
  },
  "warnings": [],
  "work_dir": "/absolute/path"
}

Interpret transcript status exactly:

Status	Meaning
ok	Transcript segments are available.
no_speech	Local Whisper completed and found no speech.
unavailable	Local transcription failed; inspect transcript_error.
disabled	The installed profile intentionally excludes transcription.

disabled is not a media observation. unavailable does not invalidate frame evidence.

Work directory

work_dir contains:

• frames/frame_000.jpg and sibling keyframes;
• frames.json;
• transcript.txt;
• transcription.json when local Whisper runs;
• downloaded media and metadata for URL sources;
• report.md.

Frame paths in JSON are absolute. Open them directly when visual verification is possible.

Development

Run tests without creating a public development extra:

uv run --with pytest --with PyYAML python -m pytest -q

Build Python and plugin artifacts:

uv build
uv run python scripts/build_plugin.py

Validate both plugin surfaces:

claude plugin validate ./plugin --strict
claude plugin validate ./ --strict

Uninstall

Remove the Python runtime and private configuration:

uv tool uninstall codex-can-see
rm -rf ~/.config/codex-can-see

Remove the agent plugin through Codex or Claude. Agent-plugin removal does not delete user media or Python state.

License

MIT. See LICENSE.