unreal-llm-sandbox 0.0.1

LLM Sandbox For Unreal Engine

Unreal LLM Sandbox

A plugin for Unreal Engine that provides a Flask server for LLM-powered code execution directly in the editor.

Features

• Code Execution - All Code is executed directly in Unreal Engine
• LLM Execution - Ask an LLM for help, with your code/errros in context
• Agentic Tool Use - LLM can use tools directly in Unreal Engine
• User Tools - Build and register custom agentic tools

Installation

git clone https://github.com/NeuralVFX/unreal-llm-sandbox 
cd unreal-llm-sandbox
pip install -e .

or

pip install unreal-llm-sandbox

Unreal Server

Follow instructions at unreal-llm-sandbox-unreal to install Unreal side

Usage

Starting the Server ( Unreal )

From the menu bar:

LLM Sandbox -> Start Server

• The server runs at http://127.0.0.1:8765

Starting the Web Interface

• Start unreal-llm-sandbox from command line ( outside of Unreal )
• Open http://localhost:5001/notebook/notebook.ipynb ( or any ipynb name )
• If the notebook doesn’t exist, a blank one is created
• Notebooks auto-save every 2 seconds
• Standard .ipynb format, compatible with Jupyter

The web app provides: - Notebook Interface - Jupyter-style interface - Code Cells - Write and executing Python code in Unreal - Markdown Cells - Write notes in Markdown - LLM Prompt Cells - Chat with LLMs that have full context of your notebook + agentic control of Unreal - Agent Cells - Agentic code generation with automatic unit testing and iteration

Registering Custom Agent Tools

Create a new python file in your project’s Content/Python/tools directory:

@register_tool
def spawn_cube(location_x: float, location_y: float, location_z: float):
    """Spawn a cube at the specified world location.
    
    Args:
        location_x: X coordinate
        location_y: Y coordinate  
        location_z: Z coordinate
    
    Returns:
        Name of the spawned actor
    """
    import unreal
    # Your Unreal Python code here
    ...

Tools are automatically discovered by LLM, and may be used via a Prompt Cell

Requirements

• Unreal Engine 5.6
• Python 3.11+ (bundled with UE)

License

MIT