A desktop for AI agents
Project description
AgentDesk
Desktops for AI agents :computer:
Explore the docs »
View Demo
·
Report Bug
·
Request Feature
AgentDesk provides full-featured desktop environments which can be programatically controlled by AI agents. Spin them up locally or in the cloud.
▶ Built on agentd a runtime daemon which exposes a REST API for interacting with the desktop.
▶ Implements the DeviceBay Protocol.
Installation
pip install agentdesk
Quick Start
from agentdesk import Desktop
# Create a desktop container
desktop = Desktop.docker()
# Launch the UI for it
desktop.view(background=True)
# Open a browser to Google
desktop.open_url("https://google.com")
# Take actions on the desktop
desktop.move_mouse(500, 500)
desktop.click()
img = desktop.take_screenshot()
Usage
Create a local desktop
from agentdesk import Desktop
desktop = Desktop.docker()
$ agentdesk create --provider docker
*requires docker
Create a remote desktop on Kubernetes
desktop = Desktop.kube()
$ agentdesk create --provider kube
Create a remote desktop on GCE
desktop = Desktop.gce()
$ agentdesk create --provider gce
Create a remote desktop on EC2
desktop = Desktop.ec2()
$ agentdesk create --provider ec2
View the desktop in the UI
desktop.view()
$ agentdesk view old_mckinny
*requires docker
List desktops
Desktop.find()
$ agentdesk get
Delete a desktop
Desktop.delete("old_mckinny")
$ agentdesk delete old_mckinny
Use the desktop
desktop.open_url("https://google.com")
coords = desktop.mouse_coordinates()
desktop.move_mouse(500, 500)
desktop.click()
desktop.type_text("What kind of ducks are in Canada?")
desktop.press_key('Enter')
desktop.scroll()
img = desktop.take_screenshot()
Processors
Process images to make them more accessible to LMMs.
Grid
Add a coordinate grid on top of the image
from agentdesk.processors import GridProcessor
img = desktop.take_screenshot()
processor = GridProcessor()
grid_img = processor.process_b64(img)
Examples
Drawing Bot
See how to use a web-based drawing app with AgentDesk in our notebook.
GPT-4V
See how to use GPT-4V with AgentDesk in our notebook or agent.
Community
Come join us on Discord.
Developing
Please open an issue before creating a PR.
Changes to the VM happen in agentd.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for agentdesk-0.2.102-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ca9f4f65973ede47eb311da862eee05d7ff31887826e8db8847c33feea80a392 |
|
MD5 | 9934968cc6d8727aa4496842e1dcc821 |
|
BLAKE2b-256 | 3fd0e80181f3b16f1a0d05e528eb0cfa79d058ff2619e5031dd432b02925e7ea |