FastICRL

In-Context Reinforcement Learning for LLMs — no fine-tuning, no gradient updates.

FastICRL implements the ICRL paradigm from Reward Is Enough: LLMs Are In-Context Reinforcement Learners (Song et al., 2025). A learner LLM improves its outputs purely by reading its own history of attempts and rewards inside the context window — guided by a meta-cognitive strategist. No training, no infrastructure, just inference.

How it works

Three LLM agents collaborate in a feedback loop:

┌──────────────────────────────────────────────────┐
│                   ICRLLearner                    │
│                                                  │
│  Task ──► Learner ──► Output ──► Reward Agent    │
│             ▲                         │          │
│             │        Attempt          │          │
│             │  (task, output, score)  │          │
│             └─────────────────────────┘          │
│                          │                       │
│                  (every N episodes)              │
│                          ▼                       │
│                      Strategist                  │
│                (refines the strategy)            │
└──────────────────────────────────────────────────┘

Each agent can be backed by a different model — e.g. a cheap model for reward, a powerful one for the learner.

Installation

pip install fasticrl

Or with uv:

uv add fasticrl

Model provider extras (install whichever you use):

pip install "fasticrl[openai]"   # OpenAI
pip install "fasticrl[ollama]"   # Ollama (local models)

Requires Python ≥ 3.13.

Quick start

from fasticrl import ICRLLearner
from agno.models.openai import OpenAIChat

model = OpenAIChat(id="gpt-4o-mini")

learner = ICRLLearner(
    learner_model=model,
    reward_model=model,
    strategy_model=model,
    task_description="Write a concise, compelling product description for an e-commerce listing.",
    tasks=[
        "Wireless noise-cancelling headphones",
        "Ergonomic standing desk",
        "Portable espresso maker",
    ],
)

# Run 3 episodes, update strategy every 2 steps, show progress bar
learner.auto_learn(episodes=3, batch_size=2, cli_mode=True, strategy_update_interval=2)

# Inspect what the agent learned
print(learner.strategy)

API

ICRLLearner

ICRLLearner(
    learner_model,        # agno Model for the learner agent
    reward_model,         # agno Model for the reward agent (optional; needed for training)
    strategy_model,       # agno Model for the strategist agent (optional; needed for strategy updates)
    task_description,     # the agent's domain/identity framing (required)
    tasks,                # list of concrete task instances to cycle through
    buffer,               # optional: pre-loaded list of Attempt objects
    strategy,             # optional: pre-loaded strategy string
)

Note: task_description should carry only the agent's domain/identity framing (what kind of expert it is, what it works on). Do not include exploration or learning instructions — those are FastICRL's job, baked into its own system prompts.

Key methods

Saving and resuming

# Save
learner.to_yaml("my_agent.yaml")

# Resume later
learner = ICRLLearner.from_yaml(
    "my_agent.yaml",
    learner_model=model,
    reward_model=model,
    strategy_model=model,
)
learner.auto_learn(episodes=5)

Inference (eval mode)

Once trained, an agent can be switched to eval mode — a frozen policy that applies the learned strategy in a single LLM call per task: no reward scoring, no buffer growth, no exploration.

# Train and persist
learner.auto_learn(episodes=5)
learner.to_yaml("expert.yaml")

# Later: load for inference only — no reward/strategy models needed
expert = ICRLLearner.from_yaml("expert.yaml", learner_model=model).eval()
answer = expert.run("Compact mechanical keyboard")

To embed a trained expert as a member of an agno Team, use to_eval_agent() — it returns a plain agno Agent carrying the frozen-policy system prompt (leave model=None to let the team supply one):

expert_agent = expert.to_eval_agent(name="copywriting-expert")

Using Ollama (local models)

from agno.models.ollama import Ollama

learner = ICRLLearner(
    learner_model=Ollama(id="llama3.2"),
    reward_model=Ollama(id="llama3.2"),
    strategy_model=Ollama(id="llama3.2"),
    task_description="...",
    tasks=[...],
)

Any agno-compatible model works.

Citation

This project is based on and inspired by the following papers:

Reward Is Enough: LLMs Are In-Context Reinforcement Learners
Kefan Song, Amir Moeini, Peng Wang, Lei Gong, Rohan Chandra, Shangtong Zhang, Yanjun Qi
arXiv:2506.06303 — https://arxiv.org/abs/2506.06303

Large Language Models as Optimizers
Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, Xinyun Chen
arXiv:2309.03409 — https://arxiv.org/abs/2309.03409

Prompted Policy Search: Reinforcement Learning through Linguistic and Numerical Reasoning in LLMs
Yifan Zhou, Sachin Grover, Mohamed El Mistiri, Kamalesh Kalirathinam, Pratyush Kerhalkar, Swaroop Mishra, Neelesh Kumar, Sanket Gaurav, Oya Aran, Heni Ben Amor
NeurIPS 2025 — https://openreview.net/forum?id=95plu1Mo20

License

MIT