browser-use-recorder 0.1.41

Make websites accessible for AI agents

Enable AI to control your browser 🤖

This is a fork of Browser-Use that adds the ability to hook a custom function between each agent step.

🌐 Browser-use is the easiest way to connect your AI agents with the browser.

💡 See what others are building and share your projects in our Discord! Want Swag? Check out our Merch store.

🌤️ Skip the setup - try our hosted version for instant browser automation! Try the cloud ☁︎.

Quick start

With pip (Python>=3.11):

pip install browser-use-recorder

Install Playwright:

playwright install chromium

Spin up your agent:

# import prettyprinter  # Use it if you want to prettyprint the history objects.

# ! pip install -U pyobjtojson

gpt4o_model = ChatOpenAI(model="gpt-4o")


def send_agent_history_step(data):
    url = "http://127.0.0.1:8000/post_agent_history_step"
    response = requests.post(url, json=data)
    return response.json()


async def record_activity(agent_obj):
    website_html = None
    website_screenshot = None
    urls_json_last_elem = None
    model_thoughts_last_elem = None
    model_outputs_json_last_elem = None
    model_actions_json_last_elem = None
    extracted_content_json_last_elem = None

    print('--- BEFORE STEP FUNC ---')
    website_html: str = await agent_obj.browser_context.get_page_html()
    website_screenshot: str = await agent_obj.browser_context.take_screenshot()

    print("--> History:")
    if hasattr(agent_obj, "state"):
        history = agent_obj.state.history
    else:
        history = None

    model_thoughts = obj_to_json(
        obj=history.model_thoughts(),
        check_circular=False
    )

    # print("--- MODEL THOUGHTS ---")
    if len(model_thoughts) > 0:
        model_thoughts_last_elem = model_thoughts[-1]
        # prettyprinter.cpprint(model_thoughts_last_elem)

    # print("--- MODEL OUTPUT ACTION ---")
    model_outputs = agent_obj.state.history.model_outputs()
    model_outputs_json = obj_to_json(
        obj=model_outputs,
        check_circular=False
    )

    if len(model_outputs_json) > 0:
        model_outputs_json_last_elem = model_outputs_json[-1]
        # prettyprinter.cpprint(model_outputs_json_last_elem)

    # print("--- MODEL INTERACTED ELEM ---")
    model_actions = agent_obj.state.history.model_actions()
    model_actions_json = obj_to_json(
        obj=model_actions,
        check_circular=False
    )

    if len(model_actions_json) > 0:
        model_actions_json_last_elem = model_actions_json[-1]
        # prettyprinter.cpprint(model_actions_json_last_elem)

    # print("--- EXTRACTED CONTENT ---")
    extracted_content = agent_obj.state.history.extracted_content()
    extracted_content_json = obj_to_json(
        obj=extracted_content,
        check_circular=False
    )
    if len(extracted_content_json) > 0:
        extracted_content_json_last_elem = extracted_content_json[-1]
        # prettyprinter.cpprint(extracted_content_json_last_elem)

    print("--- URLS ---")
    urls = agent_obj.state.history.urls()
    prettyprinter.cpprint(urls)
    urls_json = obj_to_json(
        obj=urls,
        check_circular=False
    )

    if len(urls_json) > 0:
        urls_json_last_elem = urls_json[-1]
        prettyprinter.cpprint(urls_json_last_elem)

    model_step_summary = {
        "website_html": website_html,
        "website_screenshot": website_screenshot,
        "url": urls_json_last_elem,
        "model_thoughts": model_thoughts_last_elem,
        "model_outputs": model_outputs_json_last_elem,
        "model_actions": model_actions_json_last_elem,
        "extracted_content": extracted_content_json_last_elem
    }

    print("--- MODEL STEP SUMMARY ---")
    # prettyprinter.cpprint(model_step_summary)

    send_agent_history_step(data=model_step_summary)

    # response = send_agent_history_step(data=history)
    # print(response)

    # print("--> Website HTML:")
    # print(website_html[:200])
    # print("--> Website Screenshot:")
    # print(website_screenshot[:200])


agent = Agent(
    task=task,
    llm=gpt4o_model,
)

task = "Compare the price of gpt-4o and DeepSeek-V3"


async def run_agent():
    # Example of accessing history
    try:
        await agent.run(
            before_step_func=record_activity,
            max_steps=30
        )
    except Exception as e:
        print(e)

asyncio.run(run_agent())

Add your API keys for the provider you want to use to your .env file.

OPENAI_API_KEY=

For other settings, models, and more, check out the documentation 📕.

Test with UI

You can test browser-use with a UI repository

Or simply run the gradio example:

uv pip install gradio

python examples/ui/gradio_demo.py

Demos

Task: Add grocery items to cart, and checkout.

Prompt: Add my latest LinkedIn follower to my leads in Salesforce.

Prompt: Read my CV & find ML jobs, save them to a file, and then start applying for them in new tabs, if you need help, ask me.'

https://github.com/user-attachments/assets/171fb4d6-0355-46f2-863e-edb04a828d04

Prompt: Write a letter in Google Docs to my Papa, thanking him for everything, and save the document as a PDF.

Prompt: Look up models with a license of cc-by-sa-4.0 and sort by most likes on Hugging face, save top 5 to file.

https://github.com/user-attachments/assets/de73ee39-432c-4b97-b4e8-939fd7f323b3

More examples

For more examples see the examples folder or join the Discord and show off your project.

Vision

Tell your computer what to do, and it gets it done.

Roadmap

Agent

• Improve agent memory (summarize, compress, RAG, etc.)
• Enhance planning capabilities (load website specific context)
• Reduce token consumption (system prompt, DOM state)

DOM Extraction

• Improve extraction for datepickers, dropdowns, special elements
• Improve state representation for UI elements

Rerunning tasks

• LLM as fallback
• Make it easy to define workfows templates where LLM fills in the details
• Return playwright script from the agent

Datasets

• Create datasets for complex tasks
• Benchmark various models against each other
• Fine-tuning models for specific tasks

User Experience

• Human-in-the-loop execution
• Improve the generated GIF quality
• Create various demos for tutorial execution, job application, QA testing, social media, etc.

Contributing

We love contributions! Feel free to open issues for bugs or feature requests. To contribute to the docs, check out the /docs folder.

Local Setup

To learn more about the library, check out the local setup 📕.

Cooperations

We are forming a commission to define best practices for UI/UX design for browser agents. Together, we're exploring how software redesign improves the performance of AI agents and gives these companies a competitive advantage by designing their existing software to be at the forefront of the agent age.

Email Toby to apply for a seat on the committee.

Swag

Want to show off your Browser-use swag? Check out our Merch store. Good contributors will receive swag for free 👀.

Citation

If you use Browser Use in your research or project, please cite:

@software{browser_use2024,
  author = {Müller, Magnus and Žunič, Gregor},
  title = {Browser Use: Enable AI to control your browser},
  year = {2024},
  publisher = {GitHub},
  url = {https://github.com/browser-use/browser-use}
}

Made with ❤️ in Zurich and San Francisco