OSWorld-MCP: Benchmarking MCP Tool Invocation in Computer-Use Agents
🔔 Updates
2025-10-28: We released our paper and project page! 🎉
📄 Read the Paper | 🌐 Visit the Project Page
📑 Overview & Key Highlights
OSWorld-MCP is a comprehensive and fair benchmark for evaluating computer-use agents in real-world scenarios.
It jointly measures Model Context Protocol (MCP) tool invocation capabilities, graphical user interface (GUI) operation skills, and decision-making performance.
Designed as an extension of OSWorld, it significantly improves realism, balance, and comparability in evaluation.
Key Features & Findings
- 158 validated MCP tools, spanning 7 common applications (LibreOffice Writer, Calc, Impress, VS Code, Google Chrome, VLC, OS utilities). Among them, 25 distractor tools for robustness testing
- 250 tool-beneficial tasks → 69% of benchmark tasks benefit from MCP tools
- Multi-round tool invocation possible, posing real decision-making challenges
- MCP tools boost model accuracy & efficiency — e.g., OpenAI o3: 8.3% → 20.4% (15 steps)
- Highest observed Tool Invocation Rate (TIR) = 36.3% (Claude-4-Sonnet, 50 steps) → indicating ample room for improvement
- MCP tools improve agent metrics
- Higher tool invocation correlates with higher accuracy
- Combining tools introduces significant challenges
Architecture Overview
Figure: OSWorld-MCP evaluation framework integrating GUI actions and MCP tool invocations.
⚙️ Installation & Usage
1️⃣ Preparation: Code Setup
# Clone OSWorld base repo
git clone https://github.com/xlang-ai/OSWorld.git
# Clone OSWorld-MCP
git clone https://github.com/X-PLUG/OSWorld-MCP.git
Integrate OSWorld-MCP files into OSWorld to enable MCP support.
2️⃣ Preparation: Docker Environment
- Copy MCP files into
/homeinside Docker:
/home/
└── mcp_server/
└── osworld_mcp_client.py
- Install dependencies:
pip install -r requirements.txt
- Install Node.js
- Launch MCP server:
cd mcp_server
bash debug_server.sh
A successful launch opens the local MCP debug UI in your browser.
3️⃣ Running Evaluation
Example: Evaluate Claude 4 Sonnet (15 steps):
python run_multienv_e2e.py \
--api_url <your_api_url> \
--api_key <your_api_key> \
--model 'claude-sonnet-4-20250514-thinking' \
--test_all_meta_path 'evaluation_examples/test_all.json' \
--num_envs 1 \
--action_space mcp \
--max_steps 15 \
--max_trajectory_length 15
📐 Key Metrics
- Task Accuracy (Acc) — % of tasks successfully completed.
- Tool Invocation Rate (TIR) — correct decisions to use a tool or not.
- Average Completion Steps (ACS) — average number of actions per completed task.
📊 Leaderboard (Sorted by Accuracy)
🔗 Live Leaderboard: osworld-mcp.github.io
Max Steps: 15
| Model / Agent | Acc | TIR | ACS |
|---|---|---|---|
| Agent-S2.5 | 42.1 | 30.0 | 10.0 |
| Claude-4-Sonnet | 35.3 | 30.0 | 10.4 |
| Seed1.5-VL | 32.0 | 25.1 | 10.2 |
| Qwen3-VL | 31.3 | 24.5 | 10.5 |
| Gemini-2.5-Pro | 20.5 | 16.8 | 11.4 |
| OpenAI o3 | 20.4 | 16.7 | 11.6 |
| Qwen2.5-VL | 15.8 | 13.1 | 13.5 |
Max Steps: 50
| Model / Agent | Acc | TIR | ACS |
|---|---|---|---|
| Agent-S2.5 | 49.5 | 35.3 | 17.0 |
| Claude-4-Sonnet | 43.3 | 36.6 | 20.1 |
| Qwen3-VL | 39.1 | 29.5 | 21.1 |
| Seed1.5-VL | 38.4 | 29.0 | 23.0 |
| Gemini-2.5-Pro | 27.2 | 21.5 | 29.7 |
| OpenAI o3 | 25.2 | 21.0 | 32.1 |
| Qwen2.5-VL | 14.8 | 10.9 | 37.2 |
📚 Citation
@article{jia2025osworldmcp,
title={OSWorld-MCP: Benchmarking MCP Tool Invocation in Computer-Use Agents},
author={Jia, Hongrui and Liao, Jitong and Zhang, Xi and Xu, Haiyang and Xie, Tianbao and Jiang, Chaoya and Yan, Ming and Liu, Si and Ye, Wei and Huang, Fei},
year={2025},
journal={arXiv preprint arXiv:2510.24563}
}
Metadata
Release files for iflow-mcp_x-plug-osworld-mcp 0.1.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| iflow_mcp_x_plug_osworld_mcp-0.1.2.tar.gz | 1.5 MB | Without provenance | Details for iflow_mcp_x_plug_osworld_mcp-0.1.2.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| iflow_mcp_x_plug_osworld_mcp-0.1.2-py3-none-any.whl | Python 3 | none | any | Details for iflow_mcp_x_plug_osworld_mcp-0.1.2-py3-none-any.whl |
Total release size: 1.6 MB
Release files / iflow_mcp_x_plug_osworld_mcp-0.1.2.tar.gz
| Download URL | iflow_mcp_x_plug_osworld_mcp-0.1.2.tar.gz |
|---|---|
| Size | 1.5 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
3b5d0534aaf921672a64ad278620fe4ebdeff72f1d9f6ff167b14d3d11a1c3c7
|
|
BLAKE2b-256 checksum How to use checksums |
82b2e2d5205d42f1229000e8632b180572a4942f6b48d94d84ed049e697ece1b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.10.2 {"installer":{"name":"uv","version":"0.10.2","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Debian GNU/Linux","version":"13","id":"trixie","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / iflow_mcp_x_plug_osworld_mcp-0.1.2-py3-none-any.whl
| Download URL | iflow_mcp_x_plug_osworld_mcp-0.1.2-py3-none-any.whl |
|---|---|
| Size | 72.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
ae161212f94fd72e0d9acb99f16c4ac5434b8567e4a8a2f21de16fa678fff539
|
|
BLAKE2b-256 checksum How to use checksums |
336bbac1a6d3117adf3d14f29d3bac3a31028413c7421bbcee2d89f4842c6296
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.10.2 {"installer":{"name":"uv","version":"0.10.2","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Debian GNU/Linux","version":"13","id":"trixie","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for iflow-mcp_x-plug-osworld-mcp 0.1.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.