agent-ear
给没有音频通道的文本 Agent 一副离线耳朵:把音频/视频编译成同一时间轴上的 audio-score.json,再整理成层级化、带时间戳的 hearing.md。任何文本模型都能读,不需要模型本身支持音频。
状态:basic(Omni 详细转录)与 precise(Omni 前置 + 按需精修)已实现。
强依赖:Qwen Omni 模型 API。 basic/ask/precise/music 都依赖qwen3.8-omni-flash;没有 Omni API Key 和网络时,只有privacy本地降级可用。precise 还额外依赖 Qwen ASR。
边界:只做离线文件;不做 Live Translation、实时流、空间声学、声纹身份、完整音乐总谱。
这不是“可选云增强”。 Omni 是基础模式的必需前置;
privacy是唯一不调用 Qwen 的模式(MOSS 仍未实现)。
安装
先装 CLI,再装 skill。
skills/agent-ear/scripts/agent-ear只是一个 6 行的薄包装,它execPATH 里的agent-ear命令。 如果你只用npx skills add HudsonHuang/agent-ear、或手动 clone 来装 skill,必须另外执行uv tool install agent-ear, 否则第一条命令就会报exec: agent-ear: not found。agent-ear doctor会用NO-CLI标出这种情况。
macOS / Linux
curl -fsSL https://raw.githubusercontent.com/HudsonHuang/agent-ear/main/install.sh -o /tmp/agent-ear-install.sh
sh /tmp/agent-ear-install.sh
或手动:
uv tool install "agent-ear" # 只装 basic(推荐先这样)
# uv tool install "agent-ear[local]" # precise 需要:torch/BEATs/Beat This!,体积较大
agent-ear setup
Windows(原生,PowerShell)
winget install Gyan.FFmpeg
uv tool install agent-ear
agent-ear setup
如果 GitHub 脚本可用:
irm https://raw.githubusercontent.com/HudsonHuang/agent-ear/main/install.ps1 | iex
Windows 需要 ffmpeg/ffprobe 在 PATH;uv tool install 默认把 agent-ear.exe 放到 %USERPROFILE%\.local\bin。
安装向导会问什么
agent-ear setup 会:
- 检查 ffmpeg/ffprobe;
- 询问 Qwen API Key(隐藏输入,不回显);
- 可选自定义 base URL(默认
https://dashscope.aliyuncs.com); - 检查 Key;
- 选择要安装的 Agent:Codex / DSH / Claude Code;
- 写
~/.config/agent-ear/config.json(Windows:%APPDATA%\agent-ear\config.json,0600); - 安装 Skill,跑 doctor + 本地 smoke test。
Key 也可以走环境变量:DASHSCOPE_API_KEY(优先级最高)。
快速使用
# 问答:一次 Omni 调用,同时返回完整分析 + 答案
agent-ear ask input.mp4 "这段音频里发生了什么?"
# 详细转录:完整 transcript/events/music,0.5–2s 时间戳
agent-ear listen input.mp4 --mode basic --out .ear/my-video
# 精确:词级 ASR + 40ms 事件 + beat grid
agent-ear listen input.mp4 --mode precise --need transcript,events,beat --out .ear/my-video-precise
# 长视频只要笑声:不转写,只跑目标事件
agent-ear listen long.mp4 --mode basic --request "把视频里所有笑声都剪出来"
agent-ear listen long.mp4 --mode precise --request "把视频里所有笑声都剪出来"
# 问已保存的转录,不重新调 Omni
agent-ear ask --score .ear/my-video "谁在什么时候笑了?"
读结果:
cat .ear/my-video/hearing.md
agent-ear digest .ear/my-video/audio-score.json --level 1
输出
.ear/<name>-<hash>/
├── audio-score.json # 唯一事实源,schema 校验
├── audio-score.json.gz
├── hearing.md # 层级 Markdown:概览 / 章节 / 音乐 / 事件索引 / 限制
├── digest.md # 章节级短摘要
├── context.md # 给文本 LM 的完整上下文
└── work/ # 16k/32k wav
audio-score.json:speech/events/music/relations/timeline/views;事件带precision(omni-coarse/refined-40ms)。hearing.md:给 Agent/人读;# 音频纪要+## 概览/## 章节/## 音乐/## 事件索引/## 限制。context.md:给文本 LM 的完整 transcript/events/music;agent-ear ask --score直接使用,不再调 Omni。
模式
| 模式 | 用途 | 时间精度 | 依赖 |
|---|---|---|---|
basic(默认) |
详细转录、摘要、粗剪 | 0.5–2s | Qwen API |
precise |
字幕、打轴、卡点、精剪 | 词级 + 40ms | Qwen + local extra |
music |
音乐描述 + beat grid | BPM/beats | Qwen + local extra |
privacy |
不上传 | 本地 | local extra(MOSS 未实现) |
precise 用 --need transcript,events,beat 显式指定;不指定时默认三项都跑。--request 会按自然语言只跑需要的层。
实测(2026-11-11)
| 项 | 结果 |
|---|---|
| 1 小时 basic 结构化 | 231s,RTF 0.064,¥0.211/音频小时 |
| 5 分钟 precise 全量 | 57s,RTF 0.19 |
| 5 分钟只要笑声 basic / precise | 15.2s / 24.1s;53/65 事件 refine |
| Omni 单模型 QA(180 条) | 87.2% |
| 直接 ask vs 一次转录+回答 | 87.2% vs 80.0%(全量) |
| 转录后强文本模型(40 条子集) | 77.5% |
诚实结论:basic 的价值是完整转录/时间轴/可审计,不是“QA 超过 Omni”。单问题最高准确率仍是直接 ask;多问题/剪辑才值得先转录。详见 直接 ask vs 转录后 ask。
Codex / DSH / Claude
安装后 Skill 位置:
- Codex:
$CODEX_HOME/skills/agent-ear(默认~/.codex/skills/agent-ear) - DSH:
$DSH_HOME/skills/agent-ear(默认~/.dsh/skills/agent-ear) - Claude Code:
~/.claude/skills/agent-ear
Windows 下对应 %USERPROFILE%\.codex\skills\agent-ear 等。
Codex 重启后可用 /skills 查看;DSH 新会话可见;Claude 输入 /agent-ear。Skill 内部通过 scripts/agent-ear(Windows:scripts\agent-ear.cmd)调用 CLI。
DSH 的 Skill 与 Plugin 不是一回事:Skill 放目录即可,不需要
dsh plugin add或dsh.bundle.patch。
Doctor / 配置 / 卸载
agent-ear doctor --json # 依赖/config/skill/模型,不联网
agent-ear doctor --check-api # 额外检查 key
agent-ear config show # 不显示 key
agent-ear config set-key # 隐藏输入
agent-ear skill list
agent-ear uninstall --target codex
agent-ear uninstall --target all --purge
已知限制
- basic 是 0.5–2s 时间戳,不能精确打轴;要精确用 precise。
- precise 事件 refine 目前只覆盖可映射标签(如 Laughter);开放标签仍是
omni-coarse。 - 1 小时 basic 已实测;1 小时 precise 未实测。
- 5 分钟 precise 的 Qwen diarization 会过切说话人;长音频显式
--speakers 2或合并。 - AED 选型不是永久的;
ear/aed_registry.py的LAST_REVIEWED超过 6 个月要重查。 - Windows 原生支持 basic/precise;
privacy的 MOSS 仍未实现。
开发
uv venv .venv --python 3.12
uv pip install --python .venv/bin/python -e ".[dev,local]"
.venv/bin/python -m pytest -q
.venv/bin/python -m ear.cli --help
模型缓存:~/.cache/ear/models(EAR_MODEL_DIR 可覆盖)。下载支持 HTTPS_PROXY / EAR_DOWNLOAD_PROXY。
文档
Metadata
Release files for agent-ear 0.2.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| agent_ear-0.2.2-py3-none-any.whl | Python 3 | none | any | Details for agent_ear-0.2.2-py3-none-any.whl |
Release files / agent_ear-0.2.2-py3-none-any.whl
| Download URL | agent_ear-0.2.2-py3-none-any.whl |
|---|---|
| Size | 115.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
9423daecc6e931ed622a15cd9c9f7fb2bb84fcd6b92711de8524555830e2be6d
|
|
BLAKE2b-256 checksum How to use checksums |
b0f24383464287e5c9a89e0853b784e115f71ddb1671dcb1b3003e3e8357dfc0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 11, 2026.
Transparency logSecurity for agent-ear 0.2.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
1 file in this release has been uploaded with provenance metadata.
Attestation for agent_ear-0.2.2-py3-none-any.whl
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 11, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.