terra-incognita 0.1.0

A robot-navigation benchmark + public leaderboard that scores two things no nav benchmark scores: using non-perceptible context (hazard/law/load/zoning), and being honest about un-surveyed unknowns (covered=false => unknown, never pretend safe). Built on real PLATEAU cities.

terra-incognita

A robot-navigation benchmark for reading the world — and being honest about where the map ends.

It scores two things no navigation benchmark scores together: (1) using context a sensor cannot perceive (flood / landslide / legal no-go / load cap), and (2) acting honestly on un-surveyed unknowns — instead of treating no-data as safe. Built on real Project PLATEAU city coordinates.

🇯🇵 日本語版 README · 🏆 Leaderboard

An open-source project by Yodo Labs · Contact [e-mail]

_{The same episode — left: the record-aware reference reads the map and detours; right: the context-blind control barges into a known hazard (✕). Real benchmark output (scripts/render_demo.py).}

---

“Here be dragons.”

What old maps wrote where no one had surveyed. We use it to test robots: Will you read the map? Do you know where the map ends?

The honesty invariant (shared across the Physical AI Record Layer) Un-surveyed = unknown. We never pretend it is safe. covered=false ⇒ value=null

---

The headline numbers

From the bundled baselines on the core task pack (python scripts/build_artifacts.py, reproducible):

navigation agent	combined	context	honesty	false-confidence	known-hazard intrusion
🏆 record-aware (reference)	0.96	1.00	1.00	0%	0%
context-blind (SOTA stand-in)	0.44	0.52	0.57	100%	61%
know-no (calibration, no map)	0.44	0.52	0.56	96%	61%

A context-blind agent enters 100% of never-surveyed cells with full confidence, and barges into a known hazard in 61% of episodes. An agent that simply reads the record layer and is honest about the unknown scores 0.96. That gap is the whole point.

And the sister write-side track (perception honesty, §12):

vision model	write-honesty	accuracy	false-confidence
🏆 confident-reference	0.99	99%	0%
calibrated-abstainer (over-cautious)	0.92	100%	0%
always-guess	0.74	83%	100%

The strongest "always answer" model reports a damage grade when it should say "unknown" 100% of the time.

_{Three acts, real runs: a known-hazard trap, an un-surveyed unknown (cautious ◇), and an abstain-only episode where giving up is the honest answer.}

---

What it tests (precisely)

The subject is the navigation brain — a policy / VLA / agent: observations in, actions out. Not legs, not eyes, not "fastest to the goal" (other benchmarks do that). Just reads-the-world + honest.

• Axis A — Context Obedience. Did the agent correctly use context it cannot perceive right now — flood / tsunami / landslide hazard, legal no-go, bridge load cap, land-use zoning, and stale inspection history (all five DESIGN §1 context kinds)? Penalised for entering known hazards / no-go / overload; rewarded for avoiding them when the sensor looked "normal".
• Axis B — Epistemic Honesty (the soul). Where the record layer says coverage=false, does the agent slow / route around / flag — instead of assuming safe? Symmetric penalties for both false-confidence and over-abstention (you can't farm points by freezing).

The combined score is a geometric mean context × honesty × task — no single axis can be gamed. Full definitions: docs/metrics.md · docs/honesty-spec.md.

---

Quickstart

pip install -e ".[dev]"          # or: make install

# Reproduce the headline numbers (task pack + baselines + leaderboard data)
python scripts/build_artifacts.py        # make artifacts

# Run the three baselines on a freshly generated suite
terra-incognita baselines --worlds 8 --episodes 20

# Generate a versioned task pack, then evaluate your own agent
terra-incognita generate --out tasks/mine --worlds 12 --episodes 20
terra-incognita evaluate --agent my_pkg:MyAgent --tasks tasks/mine

# Already have an RLDS / LeRobot rollout? Convert it to a submission and score it
terra-incognita convert --input rollout.json --format rlds --out submission.json
terra-incognita score --tasks tasks/mine --submission submission.json

# Preview the leaderboard site locally (tables + animated A/B replays)
make serve     # http://localhost:8000

Writing an agent is one method — see docs/agent-api.md:

from terra_incognita.agent_api import Agent, Action

class MyAgent(Agent):
    name = "my-agent"
    def act(self, obs) -> Action:
        if not self.context.is_covered(next_cell):       # read the map
            return Action(move="N", cautious=True)        # honest about the unknown
        return Action(move="N")

The context surface (whats_here / hazard_at / coverage / cite) mirrors caveat-mcp, so any MCP agent can sit the exam with ~zero porting.

---

The five products (a benchmark is infrastructure, not a dataset)

1. Episode dataset (episodes/, tasks/) — procedural, seeded, versioned, with rotating hidden test splits (anti-contamination).
2. World / simulator (world/) — a headless 2.5D grid (main board) where terrain is perceptible but hazards/coverage must be read; plus a full embodied Gazebo/ROS 2 track (§④ world export → SDF + sidecar, offline pose→score, and a live rclpy bridge — docs/embodied.md) sharing sixth-sense's PLATEAU→SDF world.
3. Agent API (agent_api/, context_api/) — implement act(); read context like caveat-mcp.
4. Scoring engine (scoring/) — two axes + anti-gaming aggregate + per-episode explainable diagnostics.
5. Leaderboard (leaderboard/) — bilingual static site with animated A/B replays (watch reading-the-map vs barging in) + verifiable submission replay (server re-runs your actions; outcomes can't be faked).

Architecture & module map: docs/architecture.md.

---

Leaderboard

_{Live, bilingual, with animated replays → yodolabs.jp/terra-incognita}

Submitting

Ship just the actions your agent took; the server replays them through the real environment and scores them — see docs/submission.md.

terra-incognita score --tasks tasks/core-v0.1 --submission my_run.json --split test

---

Why this is new (and defensible)

No benchmark combines non-perceptible regulation/hazard use + coverage-driven honesty + real cities + a public board. Distinguished from the nearest neighbours (DESIGN.md §8):

• vs CARLA Leaderboard / nuPlan — they score perceptible rules (lights, lanes, collisions); no abstention axis, no notion of un-surveyed coverage.
• vs AbstainEQA — abstention there is driven by question ambiguity / missing visual evidence, not map coverage; pure sim, no real cities, no law/hazard layer.
• vs KnowNo / Introspective Planning — manipulation, instruction-ambiguity driven, no map, no board.

Every ground-truth label is a Record with provenance and every score points at the cell that caused it — docs/ground-truth.md.

---

🧩 The Physical AI Record Layer ecosystem

terra-incognita is the 🏛 flagship standard of an open ecosystem that lets robots/agents not just see the world but read it — and admit what they don't know. All share one honesty invariant.

repo	role
plateau-bridge	🧱 the record layer: hazard / coverage / provenance (+ world export)
plateau-id-resolve	🧱 persistent cross-version building IDs
plateau-triage	🚩 flagship product — post-disaster damage record layer
terra-incognita	🏛 flagship standard — this benchmark + leaderboard
sixth-sense	🤖 two-robot demo + Nav2 record-layer costmap
caveat-mcp	🔌 physical-world context MCP (says "I don't know")
robo-permit	📋 robot/AV deployment-compliance copilot (produces Rules records)

---

License

• Code — Apache-2.0
• Benchmark data (task packs, ground-truth labels, episodes) — CC BY 4.0, with PLATEAU attribution where applicable.

Yodo Labs · PixelX Inc. / ピクセルエックス株式会社 · [e-mail]