hnp 1.1.2

Features

• Innovative discretization method: The provided solution leverages an innovation in state space design proposed by TELUS called Hyperspace Neighbour Penetration (HNP), which allows slowly-changing continuous variables to be approximated as discrete variables.

• Support for building control environments: The HNP package has built-in support for Sinergym - a building control simulation environment, as well as Beobench - a toolkit providing unified access to building control environments.

• Ready-to-use RL agents: A HNP-enabled Q-Learning agent and two baseline agents are provided in the package.

HNP

A key assumption in HNP is that continuous state spaces with very incremental, locally linear transitions can be effectively discretized into relatively coarse ranges or bands of values as tiles. When this type of coarse discretization is applied to a slowly-changing variable (e.g. temperature), it can lead to situations where an action (e.g. changing the setpoints) results in such a small change to the subsequent state observation that no discrete state transition has actually occurred. A naïve solution could be to increase the granularity of the state space, i.e. to consider much smaller changes in temperature as distinct elements of the state space, but it is computationally expensive or impossible to establish an extremely granular grid system. Alternatively, HNP computes multiple values from tile boundaries and then aggregates them using a weighted norm. This enables state-action pairs to result in steps towards other states and their corresponding values.

HNP is fully described in its foundational paper.

Supported Environments

Supported Agents

The HNP package provides the following agents:

• Random Action Agent: An agent that takes a random action

• Fixed Action Agent: An agent that always take a pre-defined action

• HNP-enabled Q-Learning Agent: A Q-learning agent with built-in HNP that allows different types of observation variables

Quickstart

pip install hnp

import numpy as np

from hnp.agents import QLearningAgent
from hnp.environment import ObservationWrapper, create_env

config = {
    "agent": {
        "num_episodes": 100,
        "horizon": 24,
        "gamma": 0.99,
        "num_tiles": 20,
        "initial_epsilon": 1,
        "epsilon_annealing": 0.999,
        "learning_rate": 0.1,
        "learning_rate_annealing": 0.999
    },
    "env": {
        "name": "Eplus-5Zone-hot-discrete-v1",
        "normalize": True,
        "obs_to_keep": [0, 1, 8, 10],
        "mask": [0, 0, 0, 0]
    }
}

obs_to_keep = np.array(config["env"]["obs_to_keep"])
mask = np.array(config["env"]["mask"])

env = create_env(config["env"])
env = ObservationWrapper(env, obs_to_keep)

agent = QLearningAgent(
    env,
    config["agent"]["params"],
    mask,
)
agent.train()
agent.save_results()
env.close()

Documentation

Detailed package documentation: HNP Docs

License

The code in this repository is published under 3-Clause BSD license (see LICENSE file).