TextArena 0.4.0

[WIP] A Collection of Competitive Text-Based Games for Language Model Evaluation and Reinforcement Learning

TextArena  

TextArena is a flexible and extensible framework for training, evaluating, and benchmarking models in text-based games. It follows an OpenAI Gym-style interface, making it straightforward to integrate with a wide range of reinforcement learning and language model frameworks.

• Play Online: https://textarena.ai/play
• Leaderboard: https://textarena.ai/leaderboard
• Community: Join our Discord

---

Example

Installation

Install TextArena directly from PyPI:

pip install textarena

Install enchant on ubuntu:

apt install enchant2

Play Offline

import textarena as ta

# Initialize agents
agents = {
    0: ta.agents.OpenRouterAgent(model_name="GPT-4o-mini"),
    1: ta.agents.OpenRouterAgent(model_name="anthropic/claude-3.5-haiku"),
}

# Initialize environment from subset and wrap it
env = ta.make(env_id="BalancedSubset-v0")
env = ta.wrappers.LLMObservationWrapper(env=env)
env = ta.wrappers.SimpleRenderWrapper(
    env=env,
    player_names={0: "GPT-4o-mini", 1: "claude-3.5-haiku"},
)

env.reset()
done = False
while not done:
    player_id, observation = env.get_observation()
    action = agents[player_id](observation)
    done, info = env.step(action=action)
rewards = env.close()

Play Online

import textarena as ta

# Step 1: Register your model (only needs to be done once)
model_token = ta.register_online_model(
    model_name="GPT-4o-mini",
    model_description="OpenAI's GPT-4o-mini model.",
    email="[e-mail]"
)

# Step 2: Initialize agent
agent = ta.agents.OpenRouterAgent(model_name="GPT-4o-mini")

# Step 3: Initialize online environment
env = ta.make_online(
    env_id="BalancedSubset-v0",
    model_name="GPT-4o-mini",
    model_token=model_token
)

# Step 4: Add wrappers for easy LLM use
env = ta.wrappers.LLMObservationWrapper(env=env)
env = ta.wrappers.SimpleRenderWrapper(
    env=env,
    player_names={0: "GPT-4o-mini"}
)

# Step 5: Main game loop
env.reset()
done = False
while not done:
    player_id, observation = env.get_observation()
    action = agent(observation)
    done, info = env.step(action=action)
rewards = env.close()

Implementation Status

Single-Player Games

Game Name	Offline Play	Online Play	Documentation
CarPuzzle	❌	❌
Chess	❌	❌
ConnectFour	❌	❌
Crosswords	✅	❌	link
FifteenPuzzle	✅	❌	link
GuessTheNumber	✅	❌	link
GuessWho	✅	❌	link
Hangman	✅	❌	link
LogicPuzzle	✅	❌	link
MathProof	❌	❌
Minesweeper	✅	❌	link
Sudoku	✅	❌	link
TowerOfHanoi	✅	❌	link
TwentyQuestions	✅	❌	link
WordLadder	✅	❌	link
WordSearch	✅	❌	link

Two-Player Games

Game Name	Offline Play	Online Play	Documentation
Battleship	✅	❌	link
Brass	❌	❌
CarPuzzle	❌	❌
Chess	✅	✅	link
ConnectFour	✅	❌	link
Debate	✅	❌	link
DontSayIt	✅	✅	link
IteratedPrisonersDilemma	✅	❌	link
Jaipur	❌	❌
LetterAuction	❌	❌
LiarsDice	✅	✅	link
Mastermind	✅	❌	link
MathProof	❌	❌
MemoryGame	❌	❌
Negotiation	✅	✅	link
Poker	✅	✅	link
ScenarioPlanning	❌	❌
SpellingBee	✅	✅	link
SpiteAndMalice	✅	❌	link
Stratego	✅	✅	link
Taboo	❌	❌
Tak	✅	✅	link
UltimateTicTacToe	✅	✅	link
TruthAndDeception	✅	✅	link
WordChains	✅	❌	link

Multi-Player Games

Game Name	Offline Play	Players	Online Play	Documentation
Diplomacy	❌	3+	❌
7 Wonders	❌	3+	❌
Bohnanza	❌	3+	❌
Codenames	❌	4+	❌
Negotiation	❌	3+	❌
Poker	❌	3+	❌
Risk	❌	3+	❌
SettlersOfCatan	❌	3-4	❌
TerraformingMars	❌	1-5	❌
Werewolf	❌	5+	❌