webact-kakashi 0.1.0

Record human web interactions and replay them with AI agents

WebAct Kakashi 🎭

Language / 语言

English · 中文

---

WebAct Kakashi 🎭

Record human web interactions → give AI agents a precise map of every element

WebAct Kakashi bridges the gap between unreliable AI web automation and real-world page complexity. Instead of asking an agent to blindly guess buttons and forms, you demonstrate the workflow once — WebAct Kakashi captures every click, input, scroll, and the exact CSS selector / XPath for each element — then feeds that rich context back to your agent.

Human demo  →  kakashi records  →  Agent reads context  →  Accurate automation

Works on Chrome and Microsoft Edge (same Manifest V3 extension, no separate builds).

---

Why WebAct Kakashi?

Problem	Without kakashi	With kakashi
Agent can't find the right button	Prompt engineering, trial-and-error	Exact CSS / XPath from real session
Page uses dynamic ids / React keys	Fragile selectors break often	Multi-attribute selectors recorded live
F12 inspection is tedious	Dev manually annotates every element	One-time human demo, reusable forever
Different flows for different tasks	Describe every step in the prompt	Load the relevant recording, agent decides

---

Quick start

1. Install the Python library

pip install webact-kakashi

2. Install the browser extension

Chrome / Edge (manual load):

1. Clone this repo, cd extension/
2. Run python create_icons.py once to generate icons
3. Open chrome://extensions (Chrome) or edge://extensions (Edge)
4. Enable Developer mode → Load unpacked → select the extension/ folder

3. Record a workflow

# Start the recording server first
kakashi record --output my_workflow.json

# Optional flags:
#   --port 7892      WebSocket port (default 7892)
#   --name "Login"   Human-readable name
#   --quiet          Suppress per-step logging

Then in your browser:

1. Navigate to the target website
2. Click Start Recording in the WebAct Kakashi popup
3. Perform all actions you want to demonstrate
4. Click ■ Stop (floating toolbar on page) or Stop Recording (popup)

Both .json and .md files are saved automatically on stop.

4. Use with an agent

from kakashi import Recording

recording = Recording.load("my_workflow.json")
context = recording.to_agent_context()

system_prompt = f"""
{context}

Task: {user_task}

Use element selectors above as trusted locators.
Do NOT replay the recorded steps — use your own judgment.
"""

---

CLI reference

kakashi record   [--output FILE] [--port PORT] [--name NAME] [--quiet]
kakashi show     RECORDING_FILE
kakashi export   RECORDING_FILE [--format json|markdown|compact] [--output FILE] [--max-steps N]
kakashi context  RECORDING_FILE [--output FILE] [--max-steps N]

---

Python API

from kakashi import Recording, export_markdown, export_compact

r = Recording.load("workflow.json")
context  = r.to_agent_context()        # plain-text block for agent prompts
compact  = export_compact(r)           # token-efficient JSON string
markdown = export_markdown(r)          # human-readable Markdown

---

Architecture

Browser (Chrome / Edge)
  └── content.js       captures events, generates CSS/XPath selectors
        │  WebSocket (ws://localhost:7892)
        ▼
  background.js        state management, popup ↔ content relay
        │
        ▼
  popup.html/js        Start / Stop / Add Note UI

Python (local)
  └── kakashi record   WebSocket server → saves .json + .md on stop
        ▼
  kakashi show / export / context
        ▼
  Your AI agent / LLM prompt

---

Project structure

webact_kakashi/
├── extension/          Chrome & Edge extension (Manifest V3)
│   ├── manifest.json
│   ├── background.js
│   ├── content.js
│   ├── popup.html / popup.js / popup.css
│   ├── create_icons.py
│   └── icons/
├── kakashi/            Python library
│   ├── __init__.py
│   ├── models.py
│   ├── server.py
│   ├── exporter.py
│   └── cli.py
├── skill/
│   └── SKILL_PROMPT.md
├── tests/
├── pyproject.toml
└── README.md

---

Contributing

git clone https://github.com/your-org/webact-kakashi
cd webact_kakashi
pip install -e ".[dev]"
pytest

License

MIT — see LICENSE.

Roadmap

• Chrome Web Store / Edge Add-ons listing
• Screenshot capture per step (optional)
• Shadow DOM & iframe support
• Playwright code generation (kakashi export --format playwright)
• VS Code extension

---

---

WebAct Kakashi 🎭

录制人工网页操作 → 给 AI Agent 提供精准的元素地图

WebAct Kakashi 解决了 AI 网页自动化中最常见的痛点：agent 找不到正确的按钮、输入框或容器。 你只需人工演示一遍操作流程，WebAct Kakashi 会自动录制每一次点击、输入、滚动，以及每个元素的完整 CSS Selector 和 XPath，再把这些信息作为上下文喂给 agent，大幅提升自动化准确率。

人工演示  →  kakashi 录制  →  Agent 读取上下文  →  精准自动化

同时支持 Chrome 和 Microsoft Edge（同一个 Manifest V3 扩展，无需分别构建）。

---

为什么用 WebAct Kakashi？

问题	不用 kakashi	用 kakashi
Agent 找不到正确的按钮	反复调试提示词，靠猜	真实录制的精确 CSS / XPath
页面用动态 id / React key	选择器容易失效	多属性选择器，录制时生成
F12 手动查元素很繁琐	开发者逐一标注元素	人工演示一次，长期复用
不同任务需要不同流程	每次都要描述所有步骤	加载对应录制，agent 自己决策

---

快速上手

1. 安装 Python 库

pip install webact-kakashi

2. 安装浏览器扩展

Chrome / Edge（开发者模式手动加载）：

1. Clone 本仓库，进入 extension/ 目录
2. 运行一次 python create_icons.py 生成图标
3. 打开 chrome://extensions（Chrome）或 edge://extensions（Edge）
4. 开启右上角开发者模式 → 加载已解压的扩展程序 → 选择 extension/ 文件夹

3. 录制工作流

# 第一步：先启动录制服务
kakashi record --output my_workflow.json

# 可选参数：
#   --port 7892      WebSocket 端口（默认 7892）
#   --name "登录流程"  录制名称
#   --quiet          关闭逐步日志

然后在浏览器中：

1. 打开目标网站
2. 点击扩展弹窗里的 Start Recording
3. 完整演示所有需要录制的操作
4. 点击页面悬浮工具栏的 ■ Stop，或弹窗里的 Stop Recording

停止后自动同时生成 .json 和 .md 两个文件。

4. 查看录制

kakashi show my_workflow.json

5. 与 Agent 配合使用

from kakashi import Recording

recording = Recording.load("my_workflow.json")
context = recording.to_agent_context()

system_prompt = f"""
{context}

任务：{user_task}

请参考上方的元素选择器作为定位依据。
不要逐步复刻录制内容——根据任务自行决定操作顺序。
"""

---

CLI 命令

kakashi record   [--output 文件] [--port 端口] [--name 名称] [--quiet]
kakashi show     录制文件
kakashi export   录制文件 [--format json|markdown|compact] [--output 文件] [--max-steps N]
kakashi context  录制文件 [--output 文件] [--max-steps N]

---

Python API

from kakashi import Recording, export_markdown, export_compact

r = Recording.load("workflow.json")
context  = r.to_agent_context()        # 给 agent 的纯文本上下文块
compact  = export_compact(r)           # 省 token 的紧凑 JSON 字符串
markdown = export_markdown(r)          # 人类可读的 Markdown 文档

---

录制格式示例

每一步是一个 JSON 对象：

{
  "step": 3,
  "action": "click",
  "timestamp": "2026-04-05T10:23:01.456Z",
  "url": "https://example.com/dashboard",
  "page_title": "Dashboard — Example",
  "element": {
    "tag": "button",
    "id": "submit-order",
    "css_selector": "#submit-order",
    "xpath": "//*[@id=\"submit-order\"]",
    "nearest_label": "Place Order",
    "attributes": { "type": "submit", "data-action": "checkout" }
  }
}

支持录制的操作类型： click · input · change · scroll · keydown · navigate · note · page_load

密码字段自动脱敏为 ***，仅录制特殊键（Enter、Escape、Tab 等），不录制普通字符输入。

---

配套 Skill

skill/SKILL_PROMPT.md 是配套的 Agent Skill 提示词，直接复制进你的 Agent 系统指令即可。

Skill 告诉 Agent：

• 如何加载和解析录制文件
• 选择器优先级：id → data-testid → name → CSS → XPath
• 不要逐步复刻录制，而是把它作为元素定位参考

---

贡献

git clone https://github.com/your-org/webact-kakashi
cd webact_kakashi
pip install -e ".[dev]"
pytest

开源协议

MIT — 见 LICENSE

Roadmap

• 上架 Chrome Web Store / Edge 加载项
• 每步可选截图功能
• Shadow DOM 与 iframe 支持
• 导出为 Playwright 脚本（kakashi export --format playwright）
• VS Code 扩展集成