Journeyman

Turn any agent into a self-learning system. Your agent learns from user feedback — and every lesson must pass your evals before it goes live. Two lines of integration code.

pip install journeyman-agents        (local dev: pip install -e ".[dev]")

import journeyman as jm
jm.init()                                     # auto-instruments openai / anthropic

@jm.agent                                     # <- your agent. Any Python callable.
def my_agent(question: str, learned_context: str = "") -> str:
    # put learned_context into your prompt — that's the whole self-learning hook
    ...

my_agent("How do I reset my password?")       # traced
jm.feedback(1.0, "correct")                   # one-line learning signal

journeyman demo        # see the full flywheel on sample data (no API keys needed)
journeyman learn       # run one eval-gated learning cycle
journeyman up          # local API server (HTTP mode for other processes/languages)

What it does

The flywheel

     your agent runs                    users react
   ┌───────────────────┐            ┌─────────────────┐
   │  @jm.agent traces │──────────► │  jm.feedback()  │
   └───────────────────┘            └────────┬────────┘
             ▲                               │
             │                               ▼
   ┌─────────┴─────────┐            ┌─────────────────┐
   │ learned_context   │            │  learning cycle │  journeyman learn
   │ (lessons your     │ ◄──────────│  new lessons    │
   │  agent earned)    │   applied  │  EVAL GATE      │──► rolled back if
   └───────────────────┘   only if  └─────────────────┘    pass-rate drops
                           evals hold

The learning cycle turns feedback into candidate lessons, reconciles them with what the agent already knows, and runs your gate suite before anything goes live — regressions roll back automatically. Well-rated production answers are also mined into suggested eval cases you can accept into a suite with one API call.

Works with zero config — scales with config

• No API key? Everything runs offline with built-in heuristics. Add OPENAI_API_KEY / ANTHROPIC_API_KEY and the same pipeline upgrades itself to LLM-powered learning, LLM-as-judge, and real embeddings. Your agent's own LLM stack is untouched either way.
• No server? The SDK writes to an embedded SQLite db (.journeyman/). journeyman up reads the same file. Set JOURNEYMAN_SERVER=http://host:port to ship traces over HTTP instead (works from other processes/languages — POST /api/traces).
• No config file? Fine. journeyman init writes an optional journeyman.yaml:

project: my-bot
entrypoint: app.main:my_agent      # lets the CLI/server run your agent for evals
learning:
  gate_suite: smoke                # the eval gate
  regression_threshold: 0.05
  auto_minutes: 0                  # >0: server runs learning cycles on a timer

Eval suites

# suite.yaml  ->  journeyman eval --load suite.yaml
name: smoke
gate: true
cases:
  - input: "How do I reset my password?"
    expect_contains: "portal.acme.com"
  - input: "VPN isn't connecting on my Mac"
    expect_contains: "6.2"
  - input: "Summarize our refund policy"
    judge: "Answer must state the 30-day window and the store-credit exception."

Checks: expect_exact, expect_contains, expect_contains_any, expect_not_contains, expect_regex, judge (binary LLM-as-judge), and scorer: module:fn for custom Python.

Safety guarantees of the SDK

• Never raises into your code; internal failures degrade to no-ops (one warning).
• Never blocks your hot path — persistence happens on a background thread with a bounded queue (drops, never blocks). jm.flush() for scripts/serverless.
• KeyboardInterrupt/SystemExit always propagate; your exceptions are recorded and re-raised unchanged.
• JOURNEYMAN_DISABLED=1 is a true kill switch.
• jm.status() tells you exactly what's running and where data goes.

Other languages

The learning loop runs in the server, so other languages need only a thin client. Official SDKs, same safety contract as Python (never throw, never block, kill switch):

• TypeScript / JavaScript — sdks/typescript (Node 18+, zero dependencies)
• Go — sdks/go (standard library only)

import * as jm from "journeyman-agents";
const myAgent = jm.agent(async (q, learnedContext) => { /* your agent */ });
await myAgent("How do I reset my password?");
jm.feedback(0.0, "wrong — the portal moved");

Anything else can speak the HTTP API directly: POST /api/traces, POST /api/feedback, GET /api/recall?q=... against journeyman up. Full guide for all three paths: docs/SDKS.md.

Python API

jm.init(project=..., autolog=True, server=None, db=None)   # all optional
@jm.agent / @jm.tool / @jm.step        # decorators (bare or with args; sync or async)
jm.wrap(fn)                            # functional form
with jm.trace("name", input=...):      # for code you can't decorate
jm.feedback(score, comment, trace_id=None)
jm.recall("query")                     # the learned-context block, on demand
jm.evaluate(fn=None, suite=None)       # run a suite from Python
jm.learn()                             # run a learning cycle from Python
jm.record_llm(model, input, output, tokens_in, tokens_out)  # custom LLM clients
jm.flush(); jm.status()

How much better does it get?

A working retrieval agent — it reads documents, extracts answers, picks options — answers four public benchmarks. It gets most of them wrong, which is the point: that's the starting line. Users correct its mistakes on the original wording, one learning cycle runs, and then it faces the same questions reworded, so it can't have memorised anything.

16% → 85%, a 5.4× improvement, from 174 corrections and one learning cycle that takes 12 seconds. Both columns are the same agent on the same reworded questions; the only difference is whether it has the lessons. Reproduce: python bench/improve.py.

What happens when the feedback is wrong?

This is the question every "learns from your corrections" tool skips, and it's the one that costs money. Take an agent that already works, hand it 10 confidently wrong corrections per benchmark, run a learning cycle, and measure how much of it still answers correctly:

Every batch is tested against your eval suite before it goes live and kept only if the pass rate holds. A lookup table has no equivalent — it applies whatever it's told, immediately. Reproduce: python bench/robustness.py.

Weakest benchmark: MBPP at 82%. The gate is exactly as good as the evals you give it, and 20 cases didn't cover enough of the poisoned tasks. Full detail in docs/BENCHMARKS.md.

Development

python -m venv .venv
.venv/Scripts/python -m pip install -e ".[dev]"
.venv/Scripts/python -m pytest tests

Docs: docs/BENCHMARKS.md (what's measured and how), docs/SDKS.md (TypeScript, Go, and HTTP integration), CONTRIBUTING.md.