gymnasium-cartpole-swingup 0.1.0

CartPole SwingUp environment for Gymnasium

Gymnasium CartPole SwingUp

A more challenging version of the classic CartPole environment for Gymnasium where the pole starts in a downward position.

Description

This package provides a port of the CartPole SwingUp environment to the modern Gymnasium API. It is based on:

• zuoxingdong/DeepPILCO
• hardmaru/estool

The environment has been updated to work with the latest Gymnasium interface and includes enhanced rendering capabilities.

Installation

# Using pip
pip install gymnasium-cartpole-swingup

# Using uv
uv add gymnasium-cartpole-swingup

For the development version, install directly from GitHub:

pip install git+https://github.com/nkiyohara/gymnasium_cartpole_swingup.git

Usage

import gymnasium as gym
import gymnasium_cartpole_swingup  # This import is required to register the environment, even if unused

# Create the environment
env = gym.make("CartPoleSwingUp-v0", render_mode="human")
observation, info = env.reset(seed=42)

for _ in range(1000):
    action = env.action_space.sample()
    observation, reward, terminated, truncated, info = env.step(action)
    
    if terminated or truncated:
        observation, info = env.reset()

env.close()

Note: The import gymnasium_cartpole_swingup line is necessary to register the environment with Gymnasium, even though it may appear unused. If you're using auto-formatters or linters that remove unused imports, you can add a # noqa comment or disable that specific check:

import gymnasium_cartpole_swingup  # noqa: F401

Environment Details

• State: Initially, the pole hangs downward ($\theta \approx \pi$)
• Goal: Swing the pole upright and maintain balance
• Action Space: Force applied to cart $[-1, 1]$
• Observation Space: $[x, \dot{x}, \cos(\theta), \sin(\theta), \dot{\theta}]$
• Reward: Higher when pole is upright and cart is centered

Observation Space Detail

The observation is a 5-dimensional vector:

Index	Observation	Description	Min	Max
0	$x$	Cart position along the track	$-2.4$	$2.4$
1	$\dot{x}$	Cart velocity	$-\infty$	$\infty$
2	$\cos(\theta)$	Cosine of the pole angle	$-1.0$	$1.0$
3	$\sin(\theta)$	Sine of the pole angle	$-1.0$	$1.0$
4	$\dot{\theta}$	Angular velocity of the pole	$-\infty$	$\infty$

Notes:

• The trigonometric representation $(\cos(\theta), \sin(\theta))$ is used instead of the raw angle to avoid discontinuities in the state space.
• When the pole is upright, $\cos(\theta) = 1$ and $\sin(\theta) = 0$.
• When the pole is hanging down, $\cos(\theta) = -1$ and $\sin(\theta) = 0$.

Action Space Detail

The action is a 1-dimensional continuous value:

Index	Action	Description	Min	Max
0	$F$	Horizontal force applied to the cart	$-1.0$	$1.0$

Notes:

• The force is scaled internally by a factor of $10.0$
• Positive values move the cart to the right
• Negative values move the cart to the left

Reward Function

The reward function is a product of two components:

1. Pole angle component: $\frac{\cos(\theta) + 1}{2}$

   • Maximum value of $1.0$ when the pole is upright ($\cos(\theta) = 1$)
   • Minimum value of $0.0$ when the pole is hanging down ($\cos(\theta) = -1$)

2. Cart position component: $\cos\left(\frac{x}{x_{threshold}} \cdot \frac{\pi}{2}\right)$

   • Maximum value of $1.0$ when the cart is centered ($x = 0$)
   • Decreases to $0.0$ as the cart approaches the boundaries ($x = \pm 2.4$)

Total reward = pole angle component $\times$ cart position component

System Dynamics

The system dynamics follow the standard cart-pole physics model. The state update equations are:

$\ddot{x} = \frac{-2m_p l \dot{\theta}^2 \sin(\theta) + 3m_p g \sin(\theta)\cos(\theta) + 4F - 4b\dot{x}}{4(m_c + m_p) - 3m_p \cos^2(\theta)}$

$\ddot{\theta} = \frac{-3m_p l \dot{\theta}^2 \sin(\theta)\cos(\theta) + 6(m_c + m_p)g\sin(\theta) + 6(F - b\dot{x})\cos(\theta)}{4l(m_c + m_p) - 3m_p l \cos^2(\theta)}$

Where:

• $m_c = 0.5$ (kg): Mass of the cart
• $m_p = 0.5$ (kg): Mass of the pole
• $l = 0.6$ (m): Half-length of the pole
• $g = 9.82$ (m/s²): Gravitational acceleration
• $b = 0.1$: Friction coefficient
• $F$: Applied force, scaled from action value