aniate

Approximate Numerics in Applied Transition Environments

aniate is a Python library for finite decision problems. It covers Markov decision processes (MDPs) and problems in which the reward depends on the history of the episode.

The user specifies the states, actions, transition probabilities, rewards and discount factor. The library then:

• validates the model when it is built
• solves it exactly
• simulates it
• tests other simulators against it
• writes one log line per operation
• saves plots as PDF files

A guide to choosing the right function for a task is in tree.md.

Installation

pip install aniate            # numpy and scipy
pip install 'aniate[vis]'     # adds matplotlib, required for plots

Python 3.10 or later is required.

Example

import aniate as an
from aniate import vis

m = an.from_functions(
    states=range(5), actions=["left", "right"],
    transition=lambda s, a: {max(s - 1, 0): 1.0} if a == "left" else {min(s + 1, 4): 0.8, s: 0.2},
    reward=lambda s, a, s_next: 10.0 if s_next == 4 else -1.0,
    gamma=0.9, terminal=[4], initial=0,
)

sol = m.solve()                                   # optimal policy, its exact value, an error bound
runs = an.simulate(m, sol, episodes=10_000)       # 10,000 episodes, run in parallel
report = an.check(m, policy=sol)                  # compares simulated behaviour with the model
vis.save(vis.overview(runs), "overview.pdf")      # four standard plots in one PDF
m.describe()                                      # the model as a JSON-compatible dict

A complete workflow uses one constructor and four functions: solve, simulate, check and vis.save.

Each operation writes one line to standard error:

aniate | model    | 5 states | 2 actions | gamma 0.9 | horizon inf | 14 transitions | valid
aniate | solve    | policy_iteration | 4 iterations | 2.8 ms | start value 3.20876 | bound 8.9e-15
aniate | simulate | 10,000 episodes | 49,768 steps | 4 ms | mean return 3.23989 +/- 0.014 | 10,000 terminated | 0 truncated
aniate | check    | PASSED | 300 episodes | 1,505 steps | 4 pairs tested | return 3.1863 vs model 3.2088
aniate | vis      | saved overview.pdf

Model validation

A model is validated at construction. Construction fails, and the error names each offending state and action, if any of the following hold:

• a row of transition probabilities does not sum to 1
• a state-action pair has no successor
• a terminal state moves to another state or pays a reward
• the discount factor is 1 and no terminal state is reachable

Unreachable states and rewards on transitions of probability zero produce warnings.

an.check covers the case where a separate simulator exists. It runs episodes through that simulator and compares the observed transitions, rewards, terminations and returns with the model, using chi-square tests with a Bonferroni correction.

These guarantees rest on runtime checks and a test suite. They have not been formally proved.

History-dependent rewards

A task such as "visit A before B" is expressed with event labels and an automaton. aniate builds the exact product model and solves it.

from aniate import nmdp

world  = an.problems.gridworld(rows=5, cols=5, goals=((0, 4),), start=(4, 0))
labels = nmdp.Labels(world).at("r4c4", "A").at("r0c4", "B")
task   = an.NMDP(world, labels, nmdp.Ordering(["A", "B"], violation_penalty=-1))
policy = task.solve()        # a MemoryPolicy: policy.step(state) returns the next action

Main functions

Worked examples

Each folder in tests/ contains a model, two tests and a script that writes PDF plots.

pytest                               # all tests
python tests/stsp/plot.py            # writes route.pdf and overview.pdf into tests/stsp

Import banner

Importing aniate prints a Fibonacci rectangle to standard error once per process. Set ANIATE_BANNER=0 to disable it. python -m aniate prints it together with the version.

Development

pip install -e '.[dev]'
pytest

Released under the MIT License.