Skip to main content

AgentUI

Visual workflow builder for computer vision and AI. Create image processing pipelines by connecting tools in a drag-and-drop interface, then export and run them programmatically.

Part of the Datamarkin ecosystem - Built on PixelFlow and Mozo for production-ready computer vision.

What It Is

AgentUI is a web-first tool builder that lets you:

  • Build visually: Drag and drop tools on a canvas to create workflows
  • Connect tools: Wire outputs to inputs with type-safe connections
  • Execute: Run workflows in the browser and see results instantly
  • Export: Save workflows as JSON for version control and programmatic execution
  • Integrate: Use as a Python library in your own applications

Think of it as a visual programming environment for computer vision tasks.

Quick Start

# Install
pip install agentui

# Start the server
agentui start

# Open http://localhost:8000 in your browser

That's it. The UI is already bundled - no separate build step needed.

Embedding in Your Flask App

AgentUI can be dropped into any Flask application with minimal code:

from flask import Flask
import agentui

app = Flask(__name__)

# Optional: Inject a custom header (e.g., navigation, user menu)
agentui.set_header("my_app_header.html",
                   context_fn=lambda: {"user": current_user.name})

# Mount AgentUI at /workflow-builder
app.register_blueprint(agentui.create_agentui_bp(), url_prefix='/workflow-builder')

# Access the runner view at /workflow-builder/run/<workflow_id>

Key integration features:

  • Custom header injection: Replace the default toolbar with your own UI
  • Runner mode: Serve workflows in execution-only mode via /run/<workflow_id>
  • JS bridge: Control workflow execution from your page via window.AgentUI.run(), .getWorkflow(), etc.

See the Integration Guide below for details.

What You Can Build

ML-Powered Tools

  • Object Detection: Detect objects using YOLOv8 or Detectron2 (80 COCO classes)
  • Instance Segmentation: Get pixel-level masks for detected objects
  • Depth Estimation: Generate depth maps from single images (Depth Anything)

Image Processing

  • Transforms: Rotate, flip, crop images with automatic detection coordinate updates
  • Enhancement: CLAHE enhancement, auto-contrast, gamma correction, image normalization
  • Analysis: Color analysis, quality metrics, dominant color extraction
  • Blending: Combine multiple images with alpha blending

Annotation & Privacy

  • Draw Detections: Bounding boxes, labels, masks, polygons
  • Privacy Protection: Blur or pixelate regions automatically
  • Object Tracking: Track objects across video frames
  • Zone Analysis: Monitor object presence in defined areas

Input/Output

  • Load: Images from files or base64 data
  • Save: Export processed images to disk
  • Web Display: Convert images to base64 for browser display

Usage

Web Interface

  1. Add tools: Drag tools from the left palette onto the canvas
  2. Connect: Click and drag from output ports to input ports
  3. Configure: Select a tool to edit its parameters in the right panel
  4. Execute: Click "Run Workflow" to process
  5. View Results: See outputs in the results panel
  6. Export: Save your workflow as JSON

Programmatic Usage

from agentui import Workflow

# Load a workflow created in the UI
workflow = Workflow.load('my_workflow.json')

# Run with an image
result = workflow.run(image='test.jpg')

# Access outputs
detections = result['detections']  # PixelFlow Detections object
print(f"Found {len(detections)} objects")

# Batch processing (automatic)
result = workflow.run(image=['img1.jpg', 'img2.jpg', 'img3.jpg'])
for i, dets in enumerate(result['detections']):
    print(f"Image {i}: {len(dets)} objects")

Workflow Design Philosophy

AgentUI is designed for visual workflow creation:

  • Create workflows using the drag-and-drop UI
  • Export as JSON for version control
  • Load and execute programmatically with the Python API

Why not build workflows in code? The visual interface is the fastest way to prototype CV pipelines. The Python API focuses on execution (loading and running workflows), not construction. This separation keeps the codebase simple and the workflow format UI-native.

The Datamarkin Ecosystem

AgentUI integrates two powerful libraries:

  • PixelFlow: Computer vision primitives (annotation, tracking, spatial analysis)
  • Mozo: Universal model serving (object detection, segmentation, depth estimation)

These libraries are maintained by the same team and designed to work together seamlessly.

Development

UI Development

Only needed if you're modifying the UI:

cd ui
npm install
npm run dev  # Development server with hot reload at http://localhost:5173

# When done
npm run build  # Builds to ../agentui/static/

Adding Custom Tools

Tools are Python classes that inherit from Tool:

from agentui.core.tool import Tool, ToolOutput, Port, PortType

class MyCustomTool(Tool):
    @property
    def tool_type(self) -> str:
        return "MyCustomTool"

    @property
    def input_ports(self) -> Dict[str, Port]:
        return {"image": Port("image", PortType.IMAGE, "Input image")}

    @property
    def output_ports(self) -> Dict[str, Port]:
        return {"image": Port("image", PortType.IMAGE, "Output image")}

    def process(self) -> bool:
        image = self.inputs["image"].data
        # Do something with the image
        self.outputs["image"] = ToolOutput(processed_image, PortType.IMAGE)
        return True

Tools are automatically discovered by the registry. See CLAUDE.md for detailed development guidance.

Installation for Development

git clone <repository-url>
cd agentui

# Python setup
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate
pip install -e .

# Start server
agentui start

# Optional: UI development (only if modifying Svelte code)
cd ui
npm install
npm run build

Roadmap

Future additions will focus on:

  • Additional ML models (OCR, classification, keypoint detection)
  • Vision-language models (GPT-4V, Claude, Gemini, Qwen-VL)
  • Cloud storage integrations (S3, GCS, Azure)
  • Advanced tracking and analytics
  • Real-time streaming workflows

Integration Guide

Custom Header Injection

Replace AgentUI's default toolbar with your own header:

import agentui

agentui.set_header(
    "templates/agentui_custom_header.html",  # Jinja2 template path
    context_fn=lambda: {                      # Optional: dynamic context per request
        "user": current_user.name,
        "workflows": get_saved_workflows()
    }
)

Your header template has access to window.AgentUI for programmatic control:

<!-- templates/agentui_custom_header.html -->
<nav class="your-custom-navbar">
    <button onclick="window.AgentUI.run()">Run Workflow</button>
    <button onclick="window.AgentUI.exportWorkflow()">Export</button>
    <select onchange="window.AgentUI.loadWorkflow(this.value)">
        {% for wf in workflows %}
        <option value="{{ wf.id }}">{{ wf.name }}</option>
        {% endfor %}
    </select>
</nav>

JS Bridge API

When embedding, control AgentUI from your header via window.AgentUI:

Method Description
run() Execute the current workflow
getWorkflow() Return { nodes, edges } of current workflow
loadWorkflow(id) Load a workflow by ID
saveWorkflow(name, description) Save current workflow to the cloud API
isRunning() Check if workflow is currently executing

Runner Mode

Serve a workflow in execution-only mode (no editor):

# In your Flask app
@app.route('/process/<workflow_id>')
def process_workflow(workflow_id):
    return redirect(url_for('agentui', view_mode='runner', workflow_id=workflow_id))

Or directly via AgentUI's blueprint:

GET /workflow-builder/run/my-workflow-id

The runner view provides:

  • Clean upload interface for input images
  • Execute button with progress indicators
  • Formatted results display (images, detections table, JSON data)

Event Listening

Listen for execution state changes from your header:

window.addEventListener('agentui:statechange', (event) => {
    const { isExecuting } = event.detail;
    if (isExecuting) {
        // Disable buttons, show spinner
    } else {
        // Re-enable buttons
    }
});

Documentation

  • CLAUDE.md: Complete developer guide and architecture documentation

Requirements

  • Python 3.9+
  • Optional: Node.js 18+ (only for UI development)

License

MIT License - see LICENSE file for details

Contributing

Contributions welcome! Please check CLAUDE.md for development guidelines and architecture overview.


Built by Datamarkin - Making computer vision accessible through visual workflows.

Metadata

Release files for agentui 0.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for agentui 0.2.0
File Size Uploaded Provenance Actions
agentui-0.2.0.tar.gz 206.6 kB Without provenance Details for agentui-0.2.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for agentui 0.2.0
File Interpreter ABI Platform Actions
agentui-0.2.0-py3-none-any.whl Python 3 none any Details for agentui-0.2.0-py3-none-any.whl

Total release size: 413.9 kB

Release files / agentui-0.2.0.tar.gz

Download URL agentui-0.2.0.tar.gz
Size 206.6 kB
Tags Source
SHA-256 checksum
How to use checksums
18ab41e0345e5885fbbb7215cc63c4b485956299d14c46234421e0c82adfc088
BLAKE2b-256 checksum
How to use checksums
40d9f8ff1ac1acb1b85269cce84323e478756daa69439040b36dc135ae643890
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.10.19

Release files / agentui-0.2.0-py3-none-any.whl

Download URL agentui-0.2.0-py3-none-any.whl
Size 207.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
b142063e58c0b4557e6ca9bdbc10b235b1af4102ead8dbd51a7d42c3fd7bf8d5
BLAKE2b-256 checksum
How to use checksums
8560eba5250cea8f7b64a2577089f1d1177f416d1d6db18c4b4bbba2d0dc39f8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.10.19

Security for agentui 0.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page