unreal-llm-sandbox 0.0.2

LLM Sandbox For Unreal Engine

Unreal LLM Sandbox

This project has two components:

• Web-Interface: unreal-llm-sandbox <--- You are here
• Unreal Plugin: unreal-llm-sandbox-plugin

What is this?

An Unreal Engine plugin that exposes a Python execution server, enabling:

• A web-based coding interface connected directly to Unreal
• LLM-assisted code review with full context of your code and errors
• Creation and deployment of agentic tools that operate inside the engine

Features

• Code Execution - All Code is executed directly in Unreal Engine
• LLM Execution - Ask an LLM for help, with your code/errors in context
• Agentic Tool Use - LLM can use tools directly in Unreal Engine
• User Tools - Build and register custom agentic tools instantly

Installation

Clone and Install

Create a new conda env, then:

git clone https://github.com/NeuralVFX/unreal-llm-sandbox 
cd unreal-llm-sandbox
pip install -e .

Set your OPENAI_API_KEY

In Windows, open Powershell and then:

setx OPENAI_API_KEY *Your_api_key*

In Linux, I bet you'll figure it out...

Unreal Server

Follow instructions at unreal-llm-sandbox-unreal to install Unreal side

Usage

Starting the Server

From the menu bar: LLM Sandbox → Start Server

• The server runs at http://127.0.0.1:8765

Starting the Web Interface

• Start unreal-llm-sandbox from command line ( outside of Unreal )
• Open http://localhost:5001/notebook/notebook.ipynb ( or any ipynb name )
• If the notebook doesn't exist, a blank one is created
• Notebooks auto-save every 2 seconds
• Standard .ipynb format, compatible with Jupyter

The web app provides:

• Notebook Interface - Jupyter-style interface
• Code Cells - Write and executing Python code in Unreal
• Markdown Cells - Write notes in Markdown
• LLM Prompt Cells - Chat with LLMs that have full context of your notebook + agentic control of Unreal

Registering Custom Agent Tools

Syntax to register a new agentic tool:

@register_tool
def spawn_cube(location_x: float, location_y: float, location_z: float):
    """Spawn a cube at the specified world location.
    
    Args:
        location_x: X coordinate
        location_y: Y coordinate  
        location_z: Z coordinate
    
    Returns:
        Name of the spawned actor
    """
    import unreal
    # Your Unreal Python code here
    ...

Either:

• Run this in a Code Cell
• Or create a new python file in your project's Content/Python/tools directory

Tool Discovery:

• Tools registered in Code Cells are instantly availible to the LLM
• Tools added to Content/Python/tools are discovered on project restart

To use:

• Open a Prompt Cell, Click the 🛠️ icon to activate Unreal tools, and write a prompt!

Requirements

• Unreal Engine 5.6
• A conda env for the web-server
• An Open AI API key