⧐ Intent Pilot

What can be said can be solved.

Get early access to the PTA model‎ ‎ |‎ ‎ Scale on our shoulders

Intent Pilot

Intent-Pilot is an orchestration of two tools: AskUI's object detector with OpenAI's GPT-4v to achieve automation. It is designed to automate repetitive tasks, and to assist users in performing complex tasks with ease. This repository is our attempt to understand the GPT-4v's potential in automation and building an end-to-end automation tool.

We are inspired by and improve upon Self-Operating-Computer by a more accurate object detection model and an improved prompting strategy. We also provide a more user-friendly interface, and a more intuitive way to interact with the tool. For example, the notification feature to let users know what is happening and what to do next. Also, our tool works across all keyboard layouts - US, German, etc, which was one of the limitations of similar tools.

Demo

https://github.com/askui/intent-pilot/assets/106730702/582d7dec-e3ff-43fd-9ab7-0cad9103b366

Quick Start

Setup

• Python 3.9 or later
• OpenAI Key
• AskUI token
  • For staters, we provide a global AskUI key with 4000 free credits. ASKUI_WORKSPACE_ID: 'e26b43ea-a18f-4cd8-a6f1-e1e41dddec18' and ASKUI_TOKEN: 'gbwdeDjlWBrsr8nLjOVB'
  • After that, you can get your own AskUI key by signing up at AskUI
• You can also copy the .env.example file to .env and fill in the required details OR You can enter the credentials in the terminal when you start the app.

Linux

• In case of linux, you may need to install the following packages:

sudo apt-get install xsel xclip python3-tk python3-dev

MacOS

• In case of MacOS, you will have to grant permissions to the terminal to access the clipboard. You can do this by going to System Preferences -> Security & Privacy -> Privacy -> Accessibility and then adding the terminal to the list of apps that can control your computer.

Windows

We are currently working on the Windows version of the tool. It will be available soon.

Quick Fix: The package also works on Windows but the Windows Defender is deleting the src/intent_pilot/utils/screenshot.py file. You have to restore the file from the quarantine and add it to the exclusion list.

Installation

pip install intent-pilot

Terminal

After installation, simply run intent in your terminal:

intent

In case, you are unable to run the command, try running the following command:

python -m intent_pilot

Join Our Discord Community

For real-time discussions and community support, join our Discord server:

• Join our Discord Server and then navigate to the #intent-pilot channel.

Contributing

Thank you for your interest in contributing! We welcome involvement from the community.

We are still deciding on the contribution guidelines. Please stay tuned for updates.

Roadmap

We are currently in the process of building PTA (Prompt-to-Automation) model, a Multi-Modal Model that can understand and execute commands in natural language, in real-time and faster than any VPA (Virtual Personal Assistant) in the market.