A Universal Platform for Training and Evaluation of Mobile Interaction
Project description
Mobile-Env: An Evaluation Platform and Benchmark for Interactive Agents in LLM Era
Mobile-Env is a interaction platform for evaluation and training of the interaction of information user interface (InfoUI). Our paper is available at arXiv.
Mobile-Env is developed based on AndroidEnv. The agent can take the screenshot and the view hierarchy (disabled defaultly for the long latency) as the observation and take a touch or type a token as the action to interact with the Android apps. Several episode signals like step instructions, rewards, or the episode end will be informed during interaction at some crucial steps. A so-called crucial step may be opening a target page, srolling to a correct area, etc. and is depending on the specific task definition.
The proposed WikiHow task set is available at the Hugging Face Platform.
Index
- Evaluating and Training Agents on Mobile-Env
- Extending a New Environment (App) or a New Task Based on Mobile-Env
- Certificate Pinning Problem & Solutions
- Miscellaneous Auxiliary Tools
Platform Features
Mobile-Env is a flexible, adaptable, and easily-extendable platform for InfoUI interaction with the following features:
- Both screenshot and view hierarchy are provided as the observation. The touch and token typing are provided as the action. Wrappers are also supported to customize the observation and action spaces. Thus, both visual-based and text-based agents, both agents with continuous action space and discrete action space, can be evaluated on Mobile-Env.
- New tasks can be easily extended through task definition files.
- Multiple sources are enabled to parse the task events from the operating system: screen text, screen icon, view hierarchy, and the system log, which makes Mobile-Env capable of adapting to most real-world apps without dedicated development. (Screen text and screen icon will be enabled with an external OCR tool and icon recognition tool. Currently, a wrapper of EasyOCR is integrated in the platform and can be enabled directly. An intergrated icon model will be embedded soon as well.)
Getting Started
Installation
Install from PyPI:
pip insall mobile-env-rl
or clone the repository and build locally.
git clone https://github.com/X-LANCE/Mobile-Env
cd Mobile-Env
pip install .
Load and Run Mobile-Env for Evaluation or Training
Before loading the Mobile-Env environment, you will need to set up an Android
Emulator device. Then you can load the
environment with some existing task definitions and start your experiments. A
detailed guidance is provided in Evaluating and Traning Agents on
Mobile-Env. Several examples with a random agent or a
human agent is also provided under examples
.
Extend a New Environment or a New Task
To extend a new environment for Mobile-Env, the environment designer needs to prepare the app package and ensure that the package manages to launch and run on some versions of Android Emulator. If the app requires varying online data, the necessary data should be crawled and dumped and then be replayed for a consistent evaluation. In such case, the designer is supposed to validate the certain effectiveness of certificate unpinning plan for the package. As regards to extend new tasks, task definition files are just required. Detailed instructions can be found in Extending a New Environment (App) or a New Task Based on Mobile-Env.
Several demo task definitions are provided under demos
. Three of them are
migrated from AndroidEnv:
classic_2048.m.textproto
- Classic 2048 game.accessibility_forwarder_clock_set_timer.m.textproto
- A simple task requiring the agent to reset a running timer.systemui_egg_land_default.m.textproto
- Flappy Droid. An open-sourced implementation of classic game, Flappy Bird.
Another one, openmoneybox.add_billings.textproto
is defined upon an
open-sourced billing app,
OpenMoneyBox.
Details are referred to in the task definition files.
Miscellaneous Auxiliary Tools
We also developed an annotation tool for the human demonstrations, and a suite of template tool to auto-generate task definitions according to templates and to combine multiple task definitions to form a multi-step task. The details are referred to in Miscellaneous Auxiliary Tools.
About
This library is developed and maintained by SJTU X-Lance. The corresponding paper is available at https://arxiv.org/abs/2305.08144.
If you find Mobile-Env useful in your research, you can cite the project using the following BibTeX:
@article{DanyangZhang2023_MobileEnv,
title = {{Mobile-Env}: An Evaluation Platform and Benchmark for Interactive Agents in LLM Era},
author = {Danyang Zhang and
Lu Chen and
Zihan Zhao and
Ruisheng Cao and
Kai Yu},
journal = {CoRR},
volume = {abs/2305.08144},
year = {2023},
url = {https://arxiv.org/abs/2305.08144},
eprinttype = {arXiv},
eprint = {2305.08144},
}
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for mobile_env_rl-2.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4dd9a1a46f7efd02f79ea38ce613f43d6f1cdb50a05b318d807718553e264504 |
|
MD5 | 967cf68d0aefc0a49d8c0862f2e083e6 |
|
BLAKE2b-256 | e95915f4166d4b0bceaf05d6a7591aaa09e57f47e90697e9883cff5c0be1553e |