blindpath 0.1.0

BlindPath: An LLM-Navigation Benchmark for Partial-Observation Grid Worlds

BlindPath 🗺️

An LLM-Navigation Benchmark for Partial-Observation Grid Worlds

BlindPath tests whether LLM agents can navigate a 2D grid world under partial observability — a key challenge in real game AI that most existing benchmarks ignore.

---

Key Distinctions

Property	BlindPath	Typical Benchmarks
Targets	Game AI agents	General reasoning
Observability	Partial (vision window)	Full global state
Input format	Non-rigid (hybrid prompting)	Fixed natural language
Task complexity	Long-horizon planning	Short single-step

---

Benchmark Metrics

Metric	Formula	Measures
Success Rate	Successes / Episodes × 100%	Task completion
Optimal Rate	OptimalPathLen / LegalActions × 100%	Navigation efficiency
Feasible Rate	LegalActions / TotalActions × 100%	Spatial rule adherence
Compliance Ratio	TotalActions / Iterations × 100%	Output format compliance
Exploration Ratio	SeenTiles / AccessibleVisionTiles × 100%	Coverage

---

Installation

pip install -e .

Or install dependencies directly:

pip install anthropic numpy

---

Quick Start

from blindpath import BlindPathEnv, EnvConfig, RandomAgent, Session

config = EnvConfig(
    seed=42,
    env_size=(20, 20),
    num_goals=1,
    obstacle_count=20,
    vision_size=7,       # agent sees a 7×7 window around itself
)

agent = RandomAgent(seed=42)
session = Session(agent, config, verbose=True)

# Single episode
metrics = session.run()
print(metrics.summary())

# Multi-episode benchmark
results = session.run_benchmark(num_episodes=20, base_seed=42)
print(results.summary())

---

Hybrid Prompting Architecture

Developers supply a custom_prompt that is combined with BlindPath's pre-defined static rules before being fed to the LLM. This gives full flexibility over how state information is presented:

from blindpath import LLMAgent, Session, BlindPathEnv, StepResult

class MyAgent(LLMAgent):
    def _prompt(self, prompt: str) -> str:
        # Example: call your LLM's API here
        # return my_llm_client.generate(prompt)
        return '{"action": "Up"}'

def my_prompt(env: BlindPathEnv, result: StepResult) -> str:
    pos = result.agent_position
    return (
        f"You are at column {pos.x}, row {pos.y}. "
        f"Goals remaining: {result.goals_remaining}. "
        "Move towards any visible 'G' tile."
    )

agent = MyAgent(prompt_builder=my_prompt)
session = Session(agent, config)

---

Action Space

{"action": "Up"}

Action	Effect
"Up"	Move agent by (0, -1)
"Down"	Move agent by (0, +1)
"Left"	Move agent by (-1, 0)
"Right"	Move agent by (+1, 0)

---

Grid Labels

Symbol	Meaning
A	Agent
E	Empty
O	Obstacle (impassable)
G	Goal
B	Boundary (edge of map)

---

Hyperparameters

Parameter	Type	Default	Notes
seed	int | None	None	Falls back to system time; ensures reproducibility when set
env_size	(int, int)	(40, 40)	Minimum 5×5
num_goals	int	1	≥1
obstacle_count	int	0	Number of obstacles
vision_size	int	11	Must be odd; ≥3

Each episode uses a unique, procedurally generated map. When running a benchmark with run_benchmark(num_episodes=N, base_seed=S), each episode receives seed = base_seed + i, producing a different grid layout every time while keeping the full run reproducible.

---

Running the Baselines

# Random agent (10 episodes)
python experiments/run_random.py --episodes 10 --seed 42

# POMCP agent (5 episodes)
python experiments/run_pomcp.py --episodes 5 --sims 300

# LLM agent (1 episode)
export ANTHROPIC_API_KEY="sk-ant-..."
python experiments/run_llm.py --model claude-haiku-4-5 --verbose

---

Project Structure

blindpath/
├── Constants/
│   ├── env.py        # Default/minimum config values
│   └── tiles.py      # Tile label constants (A, E, O, G, B)
├── DataClasses/
│   ├── env_config.py  # EnvConfig dataclass
│   ├── generated_map.py # GeneratedMap dataclass
│   ├── step_result.py # StepResult dataclass
│   └── vector2.py     # Vector2 dataclass
├── Util/
│   ├── astar.py       # A* pathfinding
│   └── grid_formatting.py # grid_to_ascii()
├── core/
│   ├── grid.py           # Grid class
│   ├── map_generation.py # Procedural map generation
│   └── env.py            # BlindPathEnv
├── agents/
│   ├── base.py           # BaseAgent abstract class
│   ├── random_agent.py
│   ├── pomcp_agent.py
│   └── llm_agent.py
├── eval/
│   ├── episode_metrics.py # EpisodeMetrics dataclass
│   └── benchmark_results.py # BenchmarkResults dataclass
└── session.py            # Session lifecycle + run_benchmark()
experiments/
├── arguments.py  # Shared CLI argument parsing
├── run_random.py
├── run_pomcp.py
└── run_llm.py

---

Baselines

• Random Agent — Uniform random action selection. Lower bound.
• POMCP Agent — Partially Observable Monte-Carlo Planning with particle filters. Traditional method baseline for partial-info navigation.
• LLM Agent — Claude via the Anthropic API. The primary evaluation target.

---

Iteration Limit

max_iterations = OptimalPathLength × (full_area / vision_area) × 3

Scales to allow for back-tracking proportional to how blind the agent is.

---

Related Work

• PPLN — Similar task, no partial observability
• GWSOT — Spatial awareness probing, full state
• GameTraversalBenchmark — 2D navigation, global state
• GridRoute — Prompt engineering for pathfinding