mlx-use 0.0.3

Make MacOS apps accessible for AI agents

Command your MacBook, and it's done—across EVERY app.
Created by Ofir Ozeri

mlx-use enables AI agents to interact with Apple framework applications through browser-use agent tech see it in action!

Quick start

⚠️ Important: Review the Warning section before proceeding.

Clone first

git clone https://github.com/browser-use/macOS-use.git && cd macOS-use

Don't forget an API
Supported providers: OAI, Anthropic or Gemini (deepseek R1 coming soon!) and set it in the .env file

At the moment, mlx-use works best with OAI or Anthropic API, tho Gemini is free. it works great two, just not as reliably.

cp .env.example .env

We reccomand using mlx-use with uv environment

brew install uv && uv venv && ./.venv/bin/activate

Install locally and that'e good to go! try the first exmaple!

uv pip install --eitable . && pytohn examples/try.py

Demos

Click the GIF for the full video!

prompt: Calculate how much is 5 X 4 and return the result, then call done.

python exmaple/calculate.py

prompt: Can you check what hour is Shabbat in israel today? call done when you finish.

python exmaple/check_time_online.py

prompt: Go to auth0.com, sign in with google auth, choose ofiroz91 gmail account, login to the website and call done when you finish.

python exmaple/login_to_auth0.py

Our Vision:

TLDR: Tell every Apple device what to do, and it gets done. on EVERY APP.

This project aimes to build the AI agent of the MLX framework by Apple that would allow the agent to prefrom any action on any Apple device. Our final goal is a open source that anyone can clone, powered by the mlx and mlx-vlm to run local private infrence at zero cost.

Roadmap goals:

1. Support MacBooks at SOTA reliability

• Refine the Agent prompting.
• Release the first working version to pypi.
• Improve self-correction.
• Add feature to allow the agent to check existing apps if failing, e.g. calendar app actual name is iCal.
• Add action for the agent to ask input from the user.
• Test Test Test! and let us know what and how to improve!

2. Support local infrence with small fine tuned model.

• Add support for infrence with local models using mlx and mlx-vlm.
• Fine tune a small model that every device can run infrence with.
• SOTA reliability.

3. Support iPhone/iPad

WARNING

This project is stil under developmeant and user discretion is advised! mlx-use can and will use your private credentials, auth services or stored passwords to complete its task, launch and interact WITH EVERY APP and UI component in your MacBook and restrictions to the model are still under active development! It is not recommended to operate it unsupervised YET mlx-use WILL NOT STOP at captha or any other forms of bot identifications, so once again, user discretion is advised.

Disclamr:

As this is an early stage release, You might experience varying success rates depending on task prompt, we're actively working on improvements.
talk me on X/Twitter or contact me on Discord, your input is crucial and highly valuable!

Contributing

We are a new project and would love contributors! Feel free to PR, open issues for bugs or feature requests.

Thanks

I would like to extend our heartfelt thanks to and for their incredible work in developing Browser Use. Their dedication and expertise have been invaluable, especially in helping with the migration process and I couldn't have done it without them!