🧩 Pyllmits Documentation & User Guide

Last updated: August 5, 2026

📓 This guide is also available as an interactive Google Colab notebook.

Pyllmits tests whether AI models can actually do spatial reasoning, using two very different challenges:

1. Crafter — drop a model into a hand-built 2D survival world and see if it can navigate, gather resources, and complete an objective, using nothing but a text description of what it sees each turn.
2. Paper Folding — a classic spatial puzzle: fold a grid, punch a hole, and ask the model which of five unfolded results matches. You can even swap "north/south/east/west" for made-up words, to check whether a model actually understands direction or just recognizes those specific words.

Both experiments run through the same interface, so testing OpenAI, Gemini, and Hugging Face models side by side is just a matter of adding them to the same run. This guide walks through the website itself — the pages, what they're for, and how they fit together.

Contents

1. Setup
2. Adding your API keys
3. Testing a model in Crafter
4. Testing a model with Paper Folding
5. Putting it to use

1 · Setup

Three lines gets you in:

!pip install pyllmits
import main
main.main()

Running that cell starts a small local server and prints a link — click it and you're in.

(Outside Colab: pip install pyllmits, then run llmits in a terminal. It opens a browser tab for you automatically.)

2 · Adding your API keys

The first thing you'll see is a welcome screen asking for API keys — OpenAI, Gemini, Hugging Face. Paste in whichever ones you have; you don't need all three, just the providers whose models you actually want to test.

Keys are saved locally and never shown back to you in plain text. You can add, remove, or check them at any point later from the Providers page, without restarting anything.

3 · Testing a model in Crafter

Configs is where every experiment you've set up lives, as a card: its world size, its objective, how many trials it's completed, and which models are in it. From each card you can Run it, Edit it, Duplicate it to try a variation, or Delete it.

Hit + New config (or Edit an existing one) and you land in the Editor — this is where an experiment actually gets built. Give it a name, decide how many trials and turns each one gets, and pick a goal (collect wood, craft a pickaxe, defeat a zombie...). The world itself is a grid you paint by hand: click a tile type — grass, water, trees, stone, a zombie, the player's starting spot — and click it onto the map. Below that is the prompt the model will actually see, and the list of models you want to test — add as many as you like, mixing providers freely.

Once it's set up, head to Run, pick the config, and hit Go. It runs in the background — Pause, Resume, or Stop it any time — while a live panel shows exactly what's happening turn by turn: which model is playing, what it's looking at, what it just replied, and how long it took to think.

When a run finishes (or even partway through one), Graphs turns the results into charts — success rate per model, how long each one took to respond, which trials succeeded or failed — so you can compare models at a glance. If you change something and rerun, hit Regenerate graphs to refresh them without rerunning the experiment. Download all saves every chart straight to your computer. Videos does the same thing for the actual gameplay — a replay clip per model, per run.

Want to test more? Raise the trial count on a config you've already run, or drop in a new model, and hitting Run again only does the new work — it never throws out what's already there.

4 · Testing a model with Paper Folding

Paper Folding has its own page, separate from Crafter, with everything on one screen.

Setup is where you name a run, decide how many trials and how many folds each puzzle gets, and pick your models — same mix-and-match across providers as Crafter. The one thing unique to this test is direction names: leave it on real names, type in your own placeholder words (fold "blue-wise" instead of "north"), or let it pick fresh random words for every single trial. That last option is the real test — if a model's accuracy holds up even when the words are meaningless and different every time, that's a much stronger sign it's actually reasoning about the fold rather than just recognizing "north."

Already run something and want to pick up where you left off? The "resume or edit a previous run" menu at the top of Setup loads an old run right back into the form — raise the trial count and only the missing trials run, or add a new model and just that one starts fresh.

Run starts it, with a live status panel showing the current model, trial, its last answer, and — if you're using placeholder words — exactly which words meant which direction on that trial. Graphs, right below it, works exactly like Crafter's: pick a run, view or regenerate its charts, or delete it once you're done with it.

5 · Putting it to use

The point of all this is comparison: run the same setup against several models at once, and the graphs show you who's actually good at this versus who just talks a good game. Because everything (results, graphs, replays) is saved to disk the moment it's produced, you can walk away mid-run, come back later, add a model you forgot, or push the trial count higher — and pick up exactly where you left off instead of starting over.

The Paper Folding side pushes that comparison one step further: run it once with real direction names, once with random placeholder words, and compare the two. A big drop in accuracy between them is the clearest signal you'll get that a model's "spatial reasoning" was leaning on the words themselves, not the geometry.