nia-web-eval-agent 1.3.0

NIA AI - Web Evaluation Agent for browser testing and UX analysis

🚀 NIA AI web-eval-agent MCP Server

Let the coding agent debug itself, you've got better things to do.

🔥 Supercharge Your Debugging

NIA AI's MCP Server launches a browser-use powered agent to autonomously execute and debug web apps directly in your code editor.

⚡ Features

• 🌐 Navigate your webapp using BrowserUse (2x faster with NIA backend)
• 📊 Capture network traffic - requests are intelligently filtered and returned into the context window
• 🚨 Collect console errors - captures logs & errors
• 🤖 Autonomous debugging - the Cursor agent calls the web QA agent mcp server to test if the code it wrote works as epected end-to-end.

🧰 MCP Tool Reference

Tool	Purpose
web_eval_agent	🤖 Automated UX evaluator that drives the browser, captures screenshots, console & network logs, and returns a rich UX report.
setup_browser_state	🔒 Opens an interactive (non-headless) browser so you can sign in once; the saved cookies/local-storage are reused by subsequent web_eval_agent runs.

Key arguments

• web_eval_agent

  • url (required) – address of the running app (e.g. http://localhost:3000)
  • task (required) – natural-language description of what to test ("run through the signup flow and note any UX issues")
  • headless_browser (optional, default false) – set to true to hide the browser window

• setup_browser_state

  • url (optional) – page to open first (handy to land directly on a login screen)

You can trigger these tools straight from your IDE chat, for example:

Evaluate my app at http://localhost:3000 – run web_eval_agent with the task "Try the full signup flow and report UX issues".

🏁 Quick Start (macOS/Linux)

1. Pre-requisites (typically not needed):

• brew: /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
• npm: (brew install npm)
• jq: brew install jq

2. Run the installer after getting an api key (free)
   • Installs playwright
   • Installs uv
   • Inserts JSON into your code editor (Cursor/Cline/Windsurf) for you!

curl -LSf https://trynia.ai/install.sh -o install.sh && bash install.sh && rm install.sh

3. Visit your favorite IDE and restart to apply the changes
4. Send a prompt in chat mode to call the web eval agent tool! e.g.

Test my app on http://localhost:3000. Use web-eval-agent.

🛠️ Manual Installation

1. Get your API key at trynia.ai
2. Install uv

curl -LsSf https://astral.sh/uv/install.sh | sh)

4. Install playwright:

npm install -g chromium playwright && uvx --with playwright playwright install --with-deps

6. Add below JSON to your relevant code editor with api key
7. Restart your code editor

🔃 Updating

• uv cache clean
• refresh MCP server

    "web-eval-agent": {
      "command": "uvx",
      "args": [
        "--refresh-package",
        "webEvalAgent",
        "--from",
        "git+https://github.com/nia-ai-app/web-eval-agent.git",
        "webEvalAgent"
      ],
      "env": {
        "NIA_API_KEY": "<YOUR_KEY>"
      }
    }

NIA AI Discord Server

🛠️ Manual Installation (Mac + Cursor/Cline/Windsurf)

1. Get your API key at trynia.ai
2. Install uv

curl -LsSf https://astral.sh/uv/install.sh | sh)

3. Install playwright:

npm install -g chromium playwright && uvx --with playwright playwright install --with-deps

4. Add below JSON to your relevant code editor with api key
5. Restart your code editor

Manual Installation (Windows + Cursor/Cline/Windsurf)

We're refining this, please open an issue if you have any issues!

1. Do all this in your code editor terminal
2. curl -LSf https://trynia.ai/install.sh -o install.sh && bash install.sh && rm install.sh
3. Get your API key at trynia.ai
4. Install uv (curl -LsSf https://astral.sh/uv/install.sh | sh)
5. uvx --from git+https://github.com/nia-ai-app/web-eval-agent.git playwright install
6. Restart code editor

🚨 Issues

• Updates aren't being received in code editors, update or reinstall for latest version: Run uv cache clean for latest
• Any issues feel free to open an Issue on this repo or in the discord!
• 5/5 - static apps without changes weren't screencasting, fixed! uv clean + restart to get fix

Changelog

• 4/29 - Agent overlay update - pause/play/stop agent run in the browser

📋 Example MCP Server Output Report

📊 Web Evaluation Report for http://localhost:5173 complete!
📝 Task: Test the API-key deletion flow by navigating to the API Keys section, deleting a key, and judging the UX.

🔍 Agent Steps
  📍 1. Navigate → http://localhost:5173
  📍 2. Click     "Login"        (button index 2)
  📍 3. Click     "API Keys"     (button index 4)
  📍 4. Click     "Create Key"   (button index 9)
  📍 5. Type      "Test API Key" (input index 2)
  📍 6. Click     "Done"         (button index 3)
  📍 7. Click     "Delete"       (button index 10)
  📍 8. Click     "Delete"       (confirm index 3)
🏁 Flow tested successfully – UX felt smooth and intuitive.

🖥️ Console Logs (10)
  1. [debug] [vite] connecting…
  2. [debug] [vite] connected.
  3. [info]  Download the React DevTools …
     …

🌐 Network Requests (10)
  1. GET /src/pages/SleepingMasks.tsx                   304
  2. GET /src/pages/MCPRegistryRegistry.tsx             304
     …

⏱️ Chronological Timeline
  01:16:23.293 🖥️ Console [debug] [vite] connecting…
  01:16:23.303 🖥️ Console [debug] [vite] connected.
  01:16:23.312 ➡️ GET /src/pages/SleepingMasks.tsx
  01:16:23.318 ⬅️ 304 /src/pages/SleepingMasks.tsx
     …
  01:17:45.038 🤖 🏁 Flow finished – deletion verified
  01:17:47.038 🤖 📋 Conclusion repeated above
👁️  See the "NIA AI Control Center" dashboard for live logs.

Star History

---

Built with <3 @ trynia.ai

NIA Web Eval Agent

AI-powered web application testing and UX evaluation tool. Automatically test user interfaces, analyze user experience, and provide detailed feedback using advanced AI agents.

🚀 Quick Start

Installation

Install from PyPI:

pip install nia-web-eval-agent

Get Your API Key

1. Visit https://trynia.ai
2. Sign up for an account
3. Get your NIA API key from the dashboard

Configuration for Cursor

Add this to your ~/.cursor/mcp.json:

{
  "mcpServers": {
    "NIA Web Eval": {
      "command": "nia-web-eval-agent",
      "env": {
        "NIA_API_KEY": "your_nia_api_key_here"
      }
    }
  }
}

Configuration for Claude Desktop

Add this to your ~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "NIA Web Eval": {
      "command": "nia-web-eval-agent",
      "env": {
        "NIA_API_KEY": "your_nia_api_key_here"
      }
    }
  }
}

Usage

Once configured, restart your IDE and you can use the web evaluation tools:

Evaluate my app at http://localhost:3000 using web_eval_agent

🧰 MCP Tools

Tool	Purpose
web_eval_agent	🤖 Automated UX evaluator that drives the browser, captures screenshots, console & network logs, and returns a rich UX report.
setup_browser_state	🔒 Opens an interactive browser for authentication; saves cookies/state for future runs.

Tool Arguments

• web_eval_agent

  • url (required) – localhost URL of your app (e.g. http://localhost:3000)
  • task (required) – description of what to test ("test the signup flow and note UX issues")
  • headless_browser (optional, default false) – set to true to hide browser window

• setup_browser_state

  • url (optional) – page to open first (useful for login screens)

🔄 Updating

To get the latest version:

pip install --upgrade nia-web-eval-agent

Then restart your IDE to pick up the changes.

🛠️ Development & Contributing

For local development:

git clone https://github.com/nia-ai-app/web-eval-agent.git
cd web-eval-agent
pip install -e .

🚨 Troubleshooting

• Tools not appearing: Ensure you've restarted your IDE after configuration
• API key issues: Get a valid key from trynia.ai
• Browser issues: The tool will auto-install required browser dependencies

📞 Support

• Discord Server
• GitHub Issues
• Documentation

---

Built with ❤️ by NIA AI