portal-env 0.3.0

A package that simplifies RL development by conveniently running RL environments in isolated Docker containers, avoiding dependencies conflicts. The agent interacts with the isolated environment through a Gymnasium interface that functions as a relay.

Portal-Env 🤖🪞✨➖✨🪞🌍

A tool for Reinforcement Learning development that separates the runtime environments of agents and RL environments. This tool addresses the challenges of dependency management in RL development (e.g., conflicting dependencies) by serving RL environments through isolated Docker containers.

• Experimenting with new RL environments without irreversible changes to your Python / Conda environment is now possible!
• Want to use that one environment that requires an ancient Ubuntu & Python 2.7? No problem!
• Multiple RL environments with conflicting dependencies can coexist without any issues!

Portal-Env creates a clean separation between:

1. The agent's runtime environment - where the RL algorithm is implemented and executed.
2. The RL environment's runtime environment - containing the RL environment and its dependencies.

A communication "portal" enables seamless interaction between the agent and the environment while keeping their runtime environments isolated.

Installation

Requirements

• Docker
• Unix-based OS if not using Docker to run your agent

pip install portal-env

Usage

Basic Agent Usage

After starting the environment-side portal (detailed below), you can interact with the environment using the AgentSidePortal, which requires <env_name> (detailed below) as the first argument and takes optional arguments and keyword arguments for environment setup:

from portal_env import AgentSidePortal
from stable_baselines3 import PPO


# Initialize the agent-side portal and the environment
env = AgentSidePortal(env_name="ale", env_args=["ALE/Pong-v5"])  # pass environment setup arguments here

# Initialize the agent
agent = PPO("MlpPolicy", env, verbose=1)
agent.learn(total_timesteps=10000)
...

Or

from portal_env import AgentSidePortal
from my_agent import Agent


# Initialize the agent-side portal and the environment
env = AgentSidePortal(env_name="ale", env_args=["ALE/Pong-v5"])  # pass environment setup arguments here

# Initialize the agent
agent = Agent(env.action_space)

# Run an episode
obs, info = env.reset()
done = False
while not done:
    action = agent.act(obs)
    obs, reward, terminated, truncated, info = env.step(action)
    done = terminated or truncated

If your agent is launched through a Docker container, please apply the following two modifications:

1. When calling AgentSidePortal, set the agent_in_docker argument to True (e.g., AgentSidePortal(..., agent_in_docker=True)). This is important for establishing the portal connection.
2. Add the portal network name either to your docker run command via the --network portal_env_net argument:

docker run --network portal_env_net ... (rest of your command)

or to your docker-compose.yaml file if you use one:

services:
    <agent_service_name>:
        ...
        networks:
            - portal_env_net
        
        
networks:
    portal_env_net:
        external: true

We highly recommend using Docker!

Launching an Environment Portal

We provide a collection of pre-built environment portals for popular environments, together with a cli tool portal-env for launching them (and also custom environment portals). Currently, we support the following environments:

• Atari Learning Environment (ale)
• Mujoco and Gymnasium environments (mujoco)
• OpenAI Retro (retro)
• Craftium (craftium)

We hope to support more environments in the future. Contributions are welcome!

To launch a supported environment using the cli tool, use:

portal-env start <env_name>

Here, <env_name> denotes a unique environment name. It should be supplied to the agent-side portal, AgentSidePortal, as the first argument during initialization.

This command will start the environment portal by automatically building the Docker image and starting a corresponding Docker container. As in the example above, environment setup arguments should be passed to the AgentSidePortal (agent-side).

Custom Environment Portals

To interact with a custom environment, you need to provide two files:

1. Environment Main Script (env_main.py): A script that starts the environment-side portal (server) and provides it with an environment factory, a callable that creates and returns a new environment instance upon call.

from portal_env import EnvSidePortal
from your_env import YourEnvironment  # Your custom environment

portal = EnvSidePortal(env_factory=YourEnvironment)
portal.run()

E.g., to set up an Atari environment portal:

from portal_env import EnvSidePortal
import gymnasium
import ale_py


def main():
    portal = EnvSidePortal(env_factory=gymnasium.make)
    portal.start()


if __name__ == '__main__':
    main()

Note that the environment's dependencies (e.g., ale_py) should only be installed through the environment Dockerfile (see below).

2. Environment Dockerfile (Dockerfile.env): A Dockerfile for building the Docker image of the environment. This Dockerfile should contain the following:

• Install environment-specific dependencies
• Install Portal-Env (RUN pip install portal-env)
• Copy your environment code
• Run the main script from step 1 above using CMD ["python", "env_main.py"].

FROM python:3.12-slim

# Install environment-specific dependencies
COPY requirements.txt .
RUN pip install -r requirements.txt

# Copy your environment code
COPY . .

# Run the environment portal
CMD ["python", "env_main.py"]

You can launch your custom environments automatically using the cli tool:

portal-env start -p <path-to-custom-env-dir> <env-name>

where <path-to-custom-env-dir> is the path to the directory containing the Dockerfile.env and env_main.py files, and <env-name> is the name of the environment (should be unique).

License

MIT License

Credits

• Portal (https://github.com/danijar/portal)