POMDPPlanners

POMDPPlanners is a set of reliable implementations of POMDP (Partially Observable Markov Decision Process) planning algorithms and environments in Python. It provides standardized simulation studies for research and production-quality planners for industrial applications — from classic benchmarks like Tiger and RockSample to photorealistic autonomous driving and robotic manipulation.

Rendered by the package itself. First row: the CARLA driving environment (left) and the Isaac Sim / IsaacLab Franka reach environment (right). Realistic environments are integrated from the open-source simulators CARLA and NVIDIA Isaac Lab — credit to their authors. Below: environments implemented by the package: Firefighting and Capture the Flag (second row), Light-Dark and Push (third row).

Show more environments

Row by row: Tiger and RockSample; LaserTag and PacMan; Battleship and Maze; T-Maze and Snake; Chicheck Invaders and CartPole; Mountain Car and Safety Ant Velocity; Racetrack.

Show planner episode videos

Episodes planned online by PFT-DPW and replayed in the package's 3D viewer. Left: Firefighting, where the two firefighters put the fire out. Right: Capture the Flag, where the blue team defends its half and tags a red attacker; it does not capture the flag in this episode.

Show environment space types

Which space types each environment supports. A world with several variants (for example a discrete and a continuous Light-Dark) gets a check for every type one of its variants uses. Isaac Lab's action and observation types depend on the task.

Environment	Discrete state	Continuous state	Discrete actions	Continuous actions	Discrete obs.	Continuous obs.
Tiger	✔️	❌	✔️	❌	✔️	❌
RockSample	✔️	❌	✔️	❌	✔️	❌
Battleship	✔️	❌	✔️	❌	✔️	❌
Capture the Flag	✔️	❌	✔️	❌	❌	✔️
Chicheck Invaders	✔️	❌	✔️	❌	✔️	❌
Firefighting	✔️	❌	✔️	❌	✔️	❌
Snake	✔️	❌	✔️	❌	✔️	❌
PacMan	✔️	❌	✔️	❌	✔️	❌
Sanity	✔️	❌	✔️	❌	✔️	❌
Maze / T-Maze	✔️	✔️	✔️	✔️	✔️	❌
Light-Dark	✔️	✔️	✔️	✔️	✔️	✔️
LaserTag	✔️	✔️	✔️	✔️	❌	✔️
Push	❌	✔️	✔️	✔️	❌	✔️
CartPole	❌	✔️	✔️	❌	❌	✔️
Mountain Car	❌	✔️	✔️	❌	❌	✔️
Safety Ant Velocity	❌	✔️	✔️	❌	❌	✔️
Racetrack	❌	✔️	✔️	❌	❌	✔️
CARLA	❌	✔️	✔️	❌	❌	✔️
Isaac Lab	❌	✔️	✔️	✔️	✔️	✔️
nuPlan	❌	✔️	✔️	❌	❌	✔️

Main Features

Features	POMDPPlanners
State-of-the-art online POMDP planners	✔️
Classic benchmarks & realistic simulator environments	✔️
Easy to define custom environments	✔️
Rich belief representations	✔️
GPU-vectorized planning & belief updates	✔️
Risk-sensitive (CVaR) & constrained planning	✔️
Parallel experiment framework with persistent caching	✔️
Hyperparameter tuning (Optuna)	✔️
Progress tracking & Slack notifications	✔️
Tutorial notebooks	✔️
Documentation	✔️
Comprehensive test suite & type hints	✔️

Documentation

Documentation is available online: https://yaacovpariente.github.io/POMDPPlanners/

Installation

Note: POMDPPlanners requires Python 3.10+.

Install the latest release from PyPI:

pip install POMDPPlanners

The release is a source distribution, so pip compiles the package's C++ modules and needs a C++ compiler.

Or install from source:

# Clone the repository
git clone https://github.com/yaacovpariente/POMDPPlanners.git
cd POMDPPlanners

# Create and activate a virtual environment
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install the package
pip install -e .

Example

Plan with POMCP on the classic Tiger problem in just a few lines:

from POMDPPlanners.environments.tiger_pomdp import TigerPOMDP
from POMDPPlanners.planners.mcts_planners.pomcp import POMCP
from POMDPPlanners.utils.belief_factory import create_environment_belief

env = TigerPOMDP(discount_factor=0.95)
planner = POMCP(environment=env, discount_factor=0.95, depth=20,
                exploration_constant=10.0, n_simulations=1000,
                name="POMCP")
belief = create_environment_belief(env, n_particles=200)

actions, _ = planner.action(belief)
print(f"Recommended action: {actions[0]}")

Running Planner Evaluation Experiment

The recommended entry point for end-to-end experiments is LocalSimulationsAPI, which runs parallel episodes, applies persistent caching, and returns aggregated statistics (mean return, CVaR, VaR, confidence intervals).

The example below compares POMCPOW and PFT-DPW on Light-Dark: a robot must reach a goal, but its position readings are only precise near a light source, so it has to detour toward the light before it can find the goal. Each planner gets 2 seconds per decision, and the episodes (30 per planner) run in parallel on every CPU core. Save it as compare_planners.py and run python compare_planners.py.

from pathlib import Path

from POMDPPlanners.environments import ContinuousLightDarkPOMDPDiscreteActions
from POMDPPlanners.planners.mcts_planners.pomcpow import POMCPOW
from POMDPPlanners.planners.mcts_planners.pft_dpw import PFT_DPW
from POMDPPlanners.utils.action_samplers import DiscreteActionSampler
from POMDPPlanners.utils.belief_factory import create_environment_belief
from POMDPPlanners.simulations.simulation_apis.local_simulations_api import LocalSimulationsAPI
from POMDPPlanners.core.simulation import EnvironmentRunParams

env = ContinuousLightDarkPOMDPDiscreteActions(discount_factor=0.95)
sampler = DiscreteActionSampler(env.get_actions())

pomcpow = POMCPOW(environment=env, discount_factor=0.95, depth=10,
                  exploration_constant=200.0, k_o=1.0, k_a=1.0,
                  alpha_o=0.5, alpha_a=0.5, time_out_in_seconds=2,
                  action_sampler=sampler, name="POMCPOW")
pft_dpw = PFT_DPW(environment=env, discount_factor=0.95, depth=10,
                  exploration_constant=200.0, time_out_in_seconds=2,
                  action_sampler=sampler, name="PFT_DPW")
belief = create_environment_belief(env, n_particles=200)

api = LocalSimulationsAPI()
_, stats = api.run_multiple_environments_and_policies(
    environment_run_params=[
        EnvironmentRunParams(
            environment=env,
            belief=belief,
            policies=[pomcpow, pft_dpw],
            num_episodes=30,
            num_steps=30,
        )
    ],
    alpha=0.1,
    confidence_interval_level=0.95,
    experiment_name="LightDark_Evaluation",
    n_jobs=-1,  # run episodes in parallel, one per CPU core
    cache_dir_path=Path("results"),
)
print(stats[["policy", "average_return", "task_completion_rate"]])

Viewing the results

The run writes its results under results/. Open them in the local results site:

pomdp-report serve results --port 8765

Then browse to http://127.0.0.1:8765, using the port you passed to --port. Open the LightDark_Evaluation experiment to compare the two planners side by side on expected return and task completion rate, and replay any episode in 3D.

The video below walks through the whole example: copying the code from this README, running it, and comparing the planners in the results site. It also shows the List and Table views, editing a chart's title and axes, exporting the chart as an image, and playing every episode at once with Live.

https://github.com/user-attachments/assets/3b305863-5180-454d-bb47-0a9dd13d76cf

Long-running experiments can report progress to Slack and a local progress database, including detection of crashed or stalled runs — set SLACK_WEBHOOK_URL in your environment and notifications are picked up automatically. See NotificationConfig for details.

Parameter tuning

LocalSimulationsAPI.run_optimize_and_evaluate tunes each planner with an Optuna study and evaluates the tuned planners over multiple episodes.

The example below tunes POMCPOW and PFT-DPW on two environments, RockSample and Push, and compares the tuned planners on each. Everything the study uses is written out so you can edit it in place: the search range of each parameter, the planner settings that stay fixed, and the objectives each trial is scored on (average return and task completion rate). Early stopping ends a study once its best trials stop improving. Each planner's name carries its environment, because the evaluation requires planner names to be unique across environments. Save it as tune_planners.py and run python tune_planners.py.

from pathlib import Path

from POMDPPlanners.environments import PushPOMDP, RockSamplePOMDP
from POMDPPlanners.planners.mcts_planners.pft_dpw import PFT_DPW
from POMDPPlanners.planners.mcts_planners.pomcpow import POMCPOW
from POMDPPlanners.core.simulation.hyperparameter_tuning import (
    EarlyStoppingConfig,
    HyperParameterOptimizationDirection,
    HyperParameterRunParams,
    HyperParamPlannerConfig,
    NumericalHyperParameter,
)
from POMDPPlanners.simulations.simulation_apis.local_simulations_api import LocalSimulationsAPI
from POMDPPlanners.utils.action_samplers import DiscreteActionSampler
from POMDPPlanners.utils.belief_factory import create_environment_belief

N_TRIALS = 50            # at most this many Optuna trials per planner
EPISODES_PER_TRIAL = 20  # episodes that score one trial
EVAL_EPISODES = 30       # fresh episodes for each tuned planner
NUM_STEPS = 30           # step limit per episode
DISCOUNT_FACTOR = 0.95
DEPTH = 10               # search depth, fixed rather than tuned

# What each trial is scored on, and in which direction.
OBJECTIVES = [
    ("average_return", HyperParameterOptimizationDirection.MAXIMIZE),
    ("task_completion_rate", HyperParameterOptimizationDirection.MAXIMIZE),
]


def search_space(env):
    max_exploration = (env.reward_range[1] - env.reward_range[0]) * DEPTH
    return [
        NumericalHyperParameter(0.0, max_exploration, "exploration_constant"),  # UCB exploration
        NumericalHyperParameter(1, 10, "k_a"),  # action widening coefficient
        NumericalHyperParameter(0.01, 0.5, "alpha_a"),  # action widening exponent
        NumericalHyperParameter(1, 10, "k_o"),  # observation widening coefficient
        NumericalHyperParameter(0.01, 0.5, "alpha_o"),  # observation widening exponent
    ]


studies = []
for env in (
    RockSamplePOMDP(discount_factor=DISCOUNT_FACTOR),
    PushPOMDP(discount_factor=DISCOUNT_FACTOR),
):
    belief = create_environment_belief(env, n_particles=200)
    sampler = DiscreteActionSampler(env.get_actions())
    for policy_cls, name in ((POMCPOW, "POMCPOW"), (PFT_DPW, "PFT_DPW")):
        planner = HyperParamPlannerConfig(
            policy_cls=policy_cls,
            hyper_parameters=search_space(env),
            constant_parameters={  # settings that stay fixed in every trial
                "discount_factor": DISCOUNT_FACTOR,
                "depth": DEPTH,
                "name": f"{name}_{env.name}",
                "environment": env,
                "action_sampler": sampler,
                "time_out_in_seconds": 1,
            },
        )
        studies.append(
            HyperParameterRunParams(
                environment=env,
                belief=belief,
                hyper_param_planner_config=planner,
                num_episodes=EPISODES_PER_TRIAL,
                num_steps=NUM_STEPS,
                n_trials=N_TRIALS,
                parameters_to_optimize=OBJECTIVES,
                early_stopping=EarlyStoppingConfig(patience=20, min_trials=20),
            )
        )

api = LocalSimulationsAPI()
_, stats = api.run_optimize_and_evaluate(
    configs=studies,
    evaluation_episodes=EVAL_EPISODES,
    evaluation_steps=NUM_STEPS,
    optimization_n_jobs=-1,  # use every CPU core
    evaluation_n_jobs=-1,
    experiment_name="Tuning_RockSample_Push",
    cache_dir_path=Path("results"),
)
print(stats[["environment", "policy", "average_return", "task_completion_rate"]])

These constants allow up to 1,000 tuning episodes per planner (50 × 20), then 30 evaluation episodes for each tuned planner. Early stopping ends a planner's study once 20 trials in a row fail to improve its best results, but never before 20 trials have run, so N_TRIALS is an upper bound.

In pomdp-report serve results, the Tuning_RockSample_Push experiment shows the whole study as one card. Its page has a "Compare Tuned planners" block for each environment, listing every evaluation metric with its confidence interval and filters for metrics, planners and environments. Each tuned planner opens a tuning view with its trials, diagnostic charts and evaluation episodes, and both pages can build a figure and download it as SVG, PNG or CSV.

The video below walks through the whole example: copying the code from this README, running the full study, and reading the results on PushPOMDP in the results site. It shows the planner comparison and its filters, a tuning view with its trials and diagnostic charts, and the evaluation episodes played Live.

https://github.com/user-attachments/assets/1d0f551d-d2f1-48f1-83b7-af31c5fd2778

Tutorial Notebooks

Self-contained Jupyter notebooks with executable end-to-end examples live in docs/examples/:

Notebook	What it covers
basic_usage.ipynb	Environment setup, belief initialization, single-planner evaluation
planners_comparison.ipynb	Side-by-side comparison of POMCP / POMCPOW / PFT-DPW on a shared environment
belief_representations.ipynb	Particle, Gaussian, and Gaussian-mixture beliefs
hyperparameter_tuning.ipynb	End-to-end Optuna search via run_optimize_and_evaluate
advanced_optimization.ipynb	Multi-config tuning, custom search spaces
custom_environment.ipynb	Implementing a new Environment subclass
tree_analysis_debugging.ipynb	Inspecting and debugging search trees

Implemented Algorithms

Algorithm	Description
POMCP	Monte Carlo tree search with unweighted particle beliefs (Silver & Veness, 2010)
POMCP-DPW	POMCP with double progressive widening for large action/observation spaces
POMCPOW	Weighted-particle MCTS for continuous observation spaces (Sunberg & Kochenderfer, 2018)
PFT-DPW	Particle filter tree with double progressive widening (Sunberg & Kochenderfer, 2018)
Sparse PFT	Particle filter tree with sparse observation branching
Sparse Sampling	Depth-limited sparse sampling of the belief MDP (Kearns et al., 2002)
BetaZero	Neural-network-guided belief-state MCTS with learned policy and value
ConstrainedZero	Safety-constrained variant of BetaZero
Constrained POMCPOW / Constrained PFT-DPW	Cost-constrained online planning
iCVaR POMCPOW / iCVaR PFT-DPW / iCVaR Sparse Sampling	Risk-averse planning with iterated CVaR objectives
VOPP	Fully GPU-vectorized online POMDP planning (Hoerger et al., 2025)
Discrete Action Sequences	Open-loop baseline planner

Implemented Environments

Environment	Description
Tiger	Classic information-gathering benchmark
Light-Dark	Navigation under state-dependent observation noise (continuous & discrete variants)
RockSample	Rover science mission with sensing trade-offs
LaserTag	Pursuit with laser range-finder observations
PacMan	Arcade-style pursuit-evasion with rendering
Battleship	Probe a hidden fleet one cell at a time and sink every ship in as few probes as possible
Maze / T-Maze	Memory task: a one-time noisy cue at the start says which distant goal pays (T-shaped corridor, or a generated maze with discrete & continuous movement)
Snake	Arcade Snake with hidden food, located through a short-range vision window and a noisy scent
Chicheck Invaders	Grid shooter: clear a flock of chickens whose dives start unseen
CartPole / MountainCar	Partially observable versions of the Gym classics
Push	Object manipulation under contact uncertainty
Firefighting	Several firefighters put out a grid fire spread by a hidden wind
Capture the Flag	Team game against hidden opponents and a hidden flag, sensed through noisy range readings
Safety-Ant-Velocity	Safety-constrained quadruped locomotion
Racetrack	Racing on HighwayEnv's racetrack, with matched fully and partially observed modes
CARLA	Photorealistic autonomous driving in the CARLA simulator
Isaac Lab	Franka reach manipulation in NVIDIA Isaac Lab / Isaac Sim
nuPlan	Autonomous driving planning on real-world driving logs
Sanity	Minimal environment for quick sanity checks

Custom environments are first-class: subclass Environment, implement the transition, observation, and reward models, and every planner and the whole experiment framework work with it out of the box. See custom_environment.ipynb.

Belief Representations

Beliefs are pluggable and planner-independent: unweighted and weighted particle filters, batched particle beliefs, Gaussian and Gaussian-mixture beliefs, and GPU-vectorized particle belief updaters for large-scale simulation.

Citing the Project

If you use POMDPPlanners in your research, please cite:

@misc{pariente2026pomdpplannersopensourcepackagepomdp,
      title={POMDPPlanners: Open-Source Package for POMDP Planning}, 
      author={Yaacov Pariente and Vadim Indelman},
      year={2026},
      eprint={2602.20810},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2602.20810}, 
}

Contributing & Support

Questions, bug reports, and feature requests are welcome on the issue tracker.

License

This project is licensed under the MIT License — see the LICENSE.md file for details.