gr-libs 1.0.3

Package with goal recognition frameworks baselines

gr-libs

gr-libs is a Python package that implements Goal Recognition (GR) algorithms using Markov Decision Processes (MDPs) to model decision-making processes. These implementations adhere to the Gymnasium API. All agents in these algorithms interact with environments registered to the Gym API as part of the initialization process of the gr-envs package, on which gr-libs depends. More details on gr-envs can be found at: GR Envs Repository.

Setup (click to expand)

Setup

Note: If you are using Windows, use Git Bash for the following commands. Otherwise, any terminal or shell will work. You will need Microsoft Visual C++ 14.0 or greater. If you experiment issues while trying to download the package, you can download the latest version of it here: https://visualstudio.microsoft.com/visual-cpp-build-tools/

gr_libs depends on gr_envs, which registers a set of Gym environments. Ensure your Python environment is set up with Python >= 3.11.

Setting Up a Python Environment (if needed)

Using Pip (click to expand)

1. Find Your Python Installation:
   To locate your Python 3.12 executable, run:

   py -3.12 -c "import sys; print(sys.executable)"
   

2. Create a New Virtual Environment:
   Using the path found above, create a new empty venv:

   C:/Users/path/to/Programs/Python/Python312/python.exe -m venv test_env
   

3. Activate the Virtual Environment:

   source test_env/Scripts/activate
   

4. Verify the Active Environment:
   Since there is no direct equivalent to conda env list, you can check your active environment via:

   echo $VIRTUAL_ENV
   

Using Conda (click to expand)

1. Create a New Conda Environment:
   Replace 3.12 with your desired Python version if necessary.

   conda create -n new_env python=3.12
   

2. Activate the Environment:

   conda activate new_env
   

Upgrade Basic Package Management Modules:

Run the following command (replace /path/to/python.exe with the actual path):

/path/to/python.exe -m pip install --upgrade pip setuptools wheel versioneer

Install the gr-libs Package:

The extras install the custom environments defined in gr_envs. (For editable installation, add the -e flag by cloning the repo and cd'ing to it https://github.com/MatanShamir1/gr_libs.git)

• Minigrid Environment:

  pip install gr_libs[minigrid]
  

• Highway Environment (Parking):

  pip install gr_libs[highway]
  

• Maze Environment (Point-Maze):

  pip install gr_libs[maze]
  

• Panda Environment:

  pip install gr_libs[panda]
  

(For editable installation, add the -e flag.)

cd /path/to/clone/of/GoalRecognitionLibs
pip install -e .

Usage Guide (click to expand)

Usage Guide

After installing gr_libs, you will have access to custom Gym environments, allowing you to set up and execute an Online Dynamic Goal Recognition (ODGR) scenario with the algorithm of your choice.

Working with an initial dataset of trained agents

gr_libs also includes a library of trained agents for the various supported environments within the package.

Dataset Download (New CLI)

To download the dataset of trained agents and caches, use the new CLI tool after installing gr_libs:

download-gr-libs-dataset

You can also specify a custom extraction directory:

download-gr-libs-dataset --extract_to /path/to/target/dir

This will download and extract the required files into the specified directory (by default, the package directory).

You may also use the following alternative methods:

• Clone the repository and run the script directly:

  git clone https://github.com/MatanShamir1/gr_libs.git
  cd gr_libs
  pip install gdown
  python download_dataset.py
  

• Manual download from Google Drive:

  • Download gr_cache.zip
  • Download trained_agents.zip

  After downloading, unzip both files into your desired directory within the package.

  After extraction, you should observe the folders trained_agents and gr_cache in your current working directory. These folders contain the necessary datasets for running experiments and using the provided tools.

• Use our docker image, which includes the dataset in it: First, make sure you have docker downloaded, installed and that the docker daemon is running. Then,

1. pull the image:

docker pull ghcr.io/matanshamir1/gr_image:latest

2. run a container:

docker run -it ghcr.io/matanshamir1/gr_image:latest bash

3. You end up with a container that has the datasets in it, but doesn't have the packages. Install the package from within the container, go back to 'Setup' for that.

Method 1: Writing a Custom Script

1. Create a recognizer

   Specify the domain name and specific environment for the recognizer, effectively telling it the domain theory - the collection of states and actions in the environment.

   import gr_libs.environment # Triggers gym env registration - you must run it!
   recognizer = Graql(
       domain_name="minigrid",
       env_name="MiniGrid-SimpleCrossingS13N4"
   )
   

2. Domain Learning Phase (For GRAQL)

   GRAQL does not accumulate information about the domain or engage in learning activities during this phase. Other algorithms don't require any data for the phase and simply use what's provided in their intialization: the domain and environment specifics, excluding the possible goals.

3. Goal Adaptation Phase

   The recognizer receives new goals and corresponding training configurations. GRAQL trains goal-directed agents and stores their policies for inference.

   recognizer.goals_adaptation_phase(
       dynamic_goals=[(11,1), (11,11), (1,11)],
       dynamic_train_configs=[(QLEARNING, 100000) for _ in range(3)]  # For expert sequence generation
   )
   

4. Inference Phase

   This phase generates a partial sequence from a trained agent, simulating suboptimal behavior with Gaussian noise.

   actor = TabularQLearner(
       domain_name="minigrid",
       problem_name="MiniGrid-SimpleCrossingS13N4-DynamicGoal-11x1-v0",
       algorithm=QLEARNING,
       num_timesteps=100000
   )
   actor.learn()
   full_sequence = actor.generate_observation(
       action_selection_method=stochastic_amplified_selection,
       random_optimalism=True  # Adds noise to action values
   )
   partial_sequence = random_subset_with_order(full_sequence, int(0.5 * len(full_sequence)), is_consecutive=False)
   closest_goal = recognizer.inference_phase(partial_sequence, (11,1), 0.5)
   

5. Evaluate the result

   print(f"Closest goal returned by Graql: {closest_goal}\nActual goal actor aimed towards: (11, 1)")
   

Method 2: Using a Configuration File

The consts.py file contains predefined ODGR problem configurations. You can use existing configurations or define new ones.

To execute an ODGR problem using the configuration file, you specify a recognizer, a domain, a gym environment within that domain and the task:

python odgr_executor.py --recognizer ExpertBasedGraml --domain minigrid --task L1 --env_name MiniGrid-SimpleCrossingS13N4

If you also add the flag:

 --collect_stats

to the cmd, 3 kinds of outputs will be generated from the ODGR problem's execution: a. Into:

outputs\\minigrid\MiniGrid-SimpleCrossingS13N4\MiniGrid-SimpleCrossingS13N4\L1\experiment_results

a .pkl and a .txt summary in a dictionary format will be generated, including the summary of all ODGR executions, including runtime and overall accuracies for all lengths and types of input sequences.

b. Into:

outputs\ExpertBasedGraml\minigrid\MiniGrid-SimpleCrossingS13N4\policy_sequences\MiniGrid-SimpleCrossingS13N4-DynamicGoal-1x11-v0_inference_seq/plan_image.png

a visulzation of the sequence the agent generated will be dumped, either in a png or an mp4 format, depending on the domain, for debugability.

c. Into: either:

outputs\ExpertBasedGraml\minigrid\MiniGrid-SimpleCrossingS13N4\goal_embeddings

In Graml algorithms, or:

outputs\Graql\minigrid\MiniGrid-SimpleCrossingS13N4\confidence

In GRAsRL algorithms, pickled results from which confidence of the results can be obtained, for offline analysis.

For GRAsRL outputs, for every possible goal, the likelihood of it being the true goal from the input sequence, based on the policy distance metric.

For GRAML outputs, the embeddings of the sequences are pickled for every goal-directed sequence. Offline, since, since in the embdding space of GRAML's metric model- sequences towards the same sequences are close and vice versa, one could reproduce the most likely goal by measuring the elementwise vector distance of the embeddings, and retrieve a confidence of it.

Running Experiments (click to expand)

Running Experiments

In light of the previous section, the user should already know how to scale the experiments using odgr_executor, and they should also understand how to use the 3 types of outputs for offline analysis of the algorithms. gr_libs also provides another scaling method to run odgr_executor on multiple domains and environments, for many ODGR problems, as well as python scripts for analysis of these results, to create plots and statistics over the executions.

Scaling odgr_executor runs

A part of the contribution of this package is standardizing the evaluations of MDP-based GR frameworks. consts.py provides a set of ODGR problems on which the framework can be evaluated. The 'evaluations' sub-package provides scripts to analyze the results of the all_experiments.py execution, done over the ODGR the problems defined at consts.py.

Running all_experiments.py

You can now run all_experiments.py with your desired combination of domains, environments, tasks, and recognizers directly from the command line, without editing the script:

python gr_libs/all_experiments.py \
    --domains minigrid parking \
    --envs MiniGrid-SimpleCrossingS13N4 Parking-S-14-PC- \
    --tasks L1 L2 L3 L4 L5 \
    --recognizers ExpertBasedGraml Graql \
    --n 5

• --domains: List of domains to run experiments on.
• --envs: List of environments (must be in the same order as domains).
• --tasks: List of tasks (applied to all domain/env pairs).
• --recognizers: List of recognizers/algorithms to evaluate.
• --n: Number of times to execute each task (default: 5).

This script uses multiprocessing to simultaneously execute many odgr_executor.py runs as child processes. It logs failures and successful executions for debugability.

After execution summary files are generated in outputs/summaries/ for further analysis and plotting.

another execution example:

python gr_libs/all_experiments.py --domains parking --envs Parking-S-14-PC- --tasks L1 L2 L3 L4 L5 --recognizers GCAura GCGraml GCDraco BGGraml Draco --n 5

Using analysis scripts

The repository provides benchmark domains and scripts for analyzing experimental results. The evaluation directory contains tools for processing and visualizing the results from odgr_executor.py and all_experiments.py. Please follow the README.md file in the 'evaluation' directory for more details.

Supported Algorithms

Algorithm	Supervised	Reinforcement Learning	Discrete States	Continuous States	Discrete Actions	Continuous Actions	Model-Based	Model-Free	Action-Only	Goal Conditioned	Fine-Tuning	Supported Environments
Graql	❌	✅	✅	❌	✅	❌	❌	✅	❌	❌	❌	Minigrid
Draco	❌	✅	✅	✅	✅	✅	❌	✅	❌	❌	❌	MinigridSimple, MinigridLava, PointMazeObstacles, PointMazeFourRooms, PandaReach, Parking
GCDraco	❌	✅	✅	✅	✅	✅	❌	✅	❌	✅	❌	PointMazeObstacles, PointMazeFourRooms, PandaReach, Parking
GCAura	❌	✅	✅	✅	✅	❌	✅	❌	✅	✅	PointMaze, PandaReach, Parking
ExpertBasedGraml	✅	✅	✅	✅	✅	✅	❌	✅	✅	❌	❌	MinigridSimple, MinigridLava, PointMazeObstacles, PointMazeFourRooms, PandaReach, Parking
GCGraml	✅	✅	✅	✅	✅	✅	❌	✅	✅	✅	❌	PointMazeObstacles, PointMazeFourRooms, PandaReach, Parking

Supported Domains

Domain	Action Space	State Space
Minigrid	Discrete	Discrete
PointMaze	Continuous	Continuous
Parking	Continuous	Continuous
Panda	Continuous	Continuous

Supported Environments

Domain	Environment name	GC Adaptable
Minigrid	MiniGrid-SimpleCrossingS13N4	❌
Minigrid	MiniGrid-LavaCrossingS9N2	❌
PointMaze	PointMaze-FourRoomsEnvDense-11x11	✅
PointMaze	PointMaze-ObstaclesEnvDense-11x11	✅
Parking	Parking-S-14-PC-	✅
Panda	PandaMyReachDense	✅

Do note one can create other environments outside the supported environments, but they're not a part of the benchmark.

For Developers

Developers will need to work slightly different: instead of installing the packages, they need to clone the repos and either install them as editables or add their paths to PYTHONPATH so they will function as packages effectively. Additional packages to install as a developer:

pip install pre-commit
pre-commit install

These will activate pre-commit hooks to keep the code readable and organized.

Using docker (click to expand)

Naviagte to the README.md under the CI folder for more information on managing docker images with datasets. In the 'actions' folder, you can find which docker images are being used in the CI pipeline and it's possible to imitate the behavior there locally.