amaze-benchmarker 1.0

Benchmark generator for sighted maze-navigating agents

AMaze

A lightweight maze navigation task generator for sighted AI agents.

Mazes

AMaze is primarily a maze generator: its goal is to provide an easy way to generate arbitrarily complex (or simple) mazes for agents to navigate in.

Every maze can be described by a unique, human-readable string:

Clues point agents towards the correct direction, which is required for them to solve intersections. Traps serve the opposite purpose and are meant to provide more challenge to the maze navigation tasks. Finally, Lures are low-danger traps that can be detected using local information only (i.e. go into a wall).

NOTE: The path to solution as well as the finish flag are only visible to the human

Agents

Agents in AMaze are loosely embodied with only access to local physical information (the current cell) and one step-removed temporal information (direction of the previous cell, if any). The input and output spaces can either be discrete or continuous.

Discrete inputs

In the discrete input case, information is provided in an easily intelligible format of fixed size. According to the cell highlighted in the previous example, an agent following the optimal trajectory would receive the following inputs:

Data is provided in direct order. Input vectors for the example above translate to:

• Start: [1 0 1 1 0 0 0 0] (3 walls at E W S and no sign)
• Clue: [.5 0 1 0 0 1 0 0] (1 wall east, coming from the east, sign pointing upwards)
• Lure: [.5 1 1 0 0 0 .25 1] (still coming from east, walls at N E and sign pointing east with magnitude .25)
• Trap: [1 0 .5 0 0 0 0 .5] (coming from west, wall on east, sign pointing down with magnitude .5)
• End: [1 1 .5 0 0 0 0 0] (coming from west, walls at E N)

Continuous inputs

For continuous inputs, a raw grayscale image is directly provided to the agent. It contains wall information on the outer edge, as well as the same temporal information as with the discrete case (centered pixel on the corresponding border). The sign, if any, is thus provided as a potentially complex image that the agent must parse and understand:

While the term continuous is a bit of stretch for coarse-grain grayscale images, it highlights the difference with the discrete case where every possible input is easily enumerable.

In this case, the input buffer is filled first by columns starting in the upper left corner (as depicted in the image). Thus console output of the input buffer is identical to what is displayed in the interface.

Examples

According to the combinations of input and output spaces, the library can work in one of three ways.

Fully discrete

Hybrid (continuous inputs, discrete outputs)

Fully continuous

Maze space complexity

To properly correlate mazes that may look very different, AMaze provides two complementary complexity metrics $S_M$ and $D_M$. The former measures the "surprisingness" of the maze, defined as the entropy over every input state. The latter corresponds to the deceptiveness which, similarly, is an entropy measure but focused on the odds of seeing the same state with different cues. More details can be found in the companion article.

The graph above illustrates the respective distribution of five classes of mazes according to these two metrics. There, one can see that complex mazes (with clues, lures and traps) are very diverse while solely using traps or lures generates mazes with a bias towards Deceptiveness and Surprisingness, respectively.

Further reading

The documentation is available at (https://amaze.readthedocs.io/) including installation instruction (pip) and detailed examples.

Contributing

Contributions to the library increasing its ease of use in both the scientific and students communities are more than welcome. Any such contribution can be made via a pull request while bugs (in the code or the documentation) should be reported in the dedicated tracker.

As a rule of thumb, you should attempt not to add any additional dependencies. Currently, the library uses:

• numpy and pandas for data processing and formatting
• PyQt5 for the interface

Continuous integration and deployment is handled by github directly through the test_and_deploy.yml workflow file. While not directly runnable, the commands used therein are encapsulated in the generalist script commands.sh. Call it with -h to get a list of available instructions.

Before performing a pull request, make sure to run the deployment tests first either through commands.sh before-deploy or by calling the dedicated script (deploy_tests.sh) directly. It will install the package under a temporary folder in multiple configurations (user, test, ...) and check for formatting and pep8 compliance.

Acknowledgement

The lead (and currently sole) developer for this library is K. Godin-Dubois. K. Miras and A. V. Kononova have contributed in an advisory and management capacity.

This research was funded by the Hybrid Intelligence Center, a 10-year programme funded by the Dutch Ministry of Education, Culture and Science through the Netherlands Organisation for Scientific Research, https://hybrid-intelligence-centre.nl, grant number 024.004.022.