Hero's Journey

A benchmark for testing whether language models can induce hidden rules from demonstrations and act on them in a goal-directed, text-based adventure game.

🔗 Code  |  📄 Paper  |  📦 PyPI  |  📐 Architecture

🔔 Overview

An agent plays an RPG-style game. It sees a set of rules for the current task, but some rules are deliberately hidden. It must infer the missing requirements by studying demonstration episodes (in-context examples) and then apply the inferred pattern to a novel entity — and execute a multi-step plan, not just state the answer.

See the paper for the full design and experiments. This package is the reusable framework; our paper experiments live separately.

🛠️ Install

pip install herosjourney                 # core: task generation + env + eval
pip install "herosjourney[runner]"       # + a generic OpenAI-compatible model adapter
pip install "herosjourney[yaml]"         # + YAML task-definition files
pip install "herosjourney[analysis]"     # + pandas/numpy/matplotlib for metrics & figures

Python 3.10+.

🧩 The four concepts

🚀 Quick start

import json
from herosjourney import get_task, compute_ecsr
from herosjourney.core.elements import fill_elements, load_lexicons
from herosjourney.core.demo_generator import generate_mixed_demos
from herosjourney.runner.adventure_episode import run_single_episode, construct_demo_context

# 1. RULE + TASK — pick a built-in task (additive | compositional | conditional |
#    override | proc_add | proc_comp | proc_cond | proc_over)
spec = get_task("additive")

# 2. Surface-realize the rule into a concrete variant (seed controls names)
sem_lex, nonce_lex = load_lexicons()
with open(spec.rules) as f:
    rule = json.load(f)
elements = fill_elements(rule, sem_lex, nonce_lex, seed=0, split_spec=spec.split)

# Source entities (shown in demos) and gen entities (what we evaluate)
source_tasks = spec.gen_fn(elements, split="source", use_nonce=False)
gen_tasks    = spec.gen_fn(elements, split="gen",    use_nonce=False)

# 3. Build the in-context demonstrations
demos        = generate_mixed_demos(source_tasks, distractor_tasks=[])
demo_context = construct_demo_context(demos)

# 4. AGENT — wrap your model as model_fn(prompt, max_tokens) -> (text, thinking, tokens)
def my_model_fn(prompt, max_tokens=512):
    text = my_llm(prompt)            # call your model however you like
    return text, None, None

# 5. Run one episode on a gen task and score it
result = run_single_episode(
    episode_idx=0,
    task=gen_tasks[0],
    demo_context=demo_context,
    max_runs=None,                   # defaults to reference_length * num_tries
    verbose=False,
    truncate_window=None,
    model_fn=my_model_fn,
    source_tasks=source_tasks,
)
print(result.success, result.efficiency)

# ECSR (efficiency-calibrated success rate) over a set of results
ecsr = compute_ecsr([result], n_tries=spec.max_tries)

Using a hosted/local model without writing a model_fn

Install the runner extra and point the generic OpenAI-compatible adapter at any endpoint (OpenAI, vLLM, LM Studio, Ollama, …):

export OPENAI_BASE_URL="http://localhost:8000/v1"   # your server
export OPENAI_API_KEY="EMPTY"                        # or your real key

result = run_single_episode(..., model_path="my-model-name")

Or from the command line:

adventure-story --task_type additive \
    --elements herosjourney/core/rules/additive.json \
    --model my-model-name --num_tries 2 --num_workers 4

➕ Adding your own task (no Python required)

Write a rule file (my_rule.json, see herosjourney/core/rules/RULE_FORMAT.md) and a task spec:

{
  "name": "my_task",
  "rules": "my_rule.json",
  "process": "property_flat",
  "split": {"fn": "two_offset", "seed": 0},
  "max_tries": 5,
  "validate_fn": "validate_additive_split",
  "eval": {"correct_rule": "…", "description": "…"}
}

import herosjourney
herosjourney.register_task("path/to/my_task.json")   # .yaml also works with [yaml]
spec = herosjourney.get_task("my_task")

For custom validators or per-episode process variation, register_task(...) also accepts keyword arguments. See docs/ARCHITECTURE.md for the full architecture.

🧠 Applying an induction method

# episode_mode selects a steering strategy applied on top of your agent
result = run_single_episode(..., model_fn=my_model_fn, episode_mode="idea")
# "standard" (default), "react", "hr", "idea"

📊 Evaluation

• ECSR (efficiency-calibrated success rate) — herosjourney.compute_ecsr: success_rate × normalized_efficiency, where efficiency = reference_length / num_runs and the floor is 1 / n_tries.
• RV (rule verbalization) — an LLM judge scores a model's free-text rule description; prompts are in herosjourney.eval.judge.

📖 Citation

If you use Hero's Journey in your research, please cite:

@misc{zheng2026herosjourneytestingcomplex,
      title={HERO'S JOURNEY: Testing Complex Rule Induction with Text Games},
      author={Anshun Asher Zheng and Kanishka Misra and David I. Beaver and Junyi Jessy Li},
      year={2026},
      eprint={2606.02556},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2606.02556},
}

📄 License

This project is released under the MIT License.