Journeyman

Turn any agent into a self-learning system — tracing, graph memory, evals, and an eval-gated learning loop. Two lines of integration code.

pip install journeyman-agents        (local dev: pip install -e ".[dev]")

import journeyman as jm
jm.init()                                     # auto-instruments openai / anthropic

@jm.agent                                     # <- your agent. Any Python callable.
def my_agent(question: str, learned_context: str = "") -> str:
    # put learned_context into your prompt — that's the whole self-learning hook
    ...

my_agent("How do I reset my password?")       # traced
jm.feedback(1.0, "correct")                   # one-line learning signal

journeyman demo        # see the full flywheel on sample data (no API keys needed)
journeyman learn       # run one eval-gated learning cycle
journeyman up          # local API server (HTTP mode for other processes/languages)

What it does

The flywheel

     your agent runs                    users react
   ┌───────────────────┐            ┌─────────────────┐
   │  @jm.agent traces │──────────► │  jm.feedback()  │
   └───────────────────┘            └────────┬────────┘
             ▲                               │
             │                               ▼
   ┌─────────┴─────────┐            ┌─────────────────┐
   │ learned_context   │            │  learning cycle │  journeyman learn
   │ (insights + graph │ ◄──────────│  reflect→dedup→ │
   │  + few-shots)     │   applied  │  EVAL GATE      │──► rolled back if
   └───────────────────┘   only if  └─────────────────┘    pass-rate drops
                           evals hold

The learning cycle: collects new traces → folds them into the knowledge graph → reflects on traces that carry signal (bad feedback, corrections, errors) → extracts candidate insights → dedups/merges them against what's already known (delta updates only — never wholesale rewrites) → runs your gate suite before and after activating the batch → rolls back automatically on regression. Well-rated production answers are also mined into suggested eval cases you can accept into a suite with one API call.

Works with zero config — scales with config

• No API key? Everything runs offline: deterministic extraction + hash embeddings. Add OPENAI_API_KEY / ANTHROPIC_API_KEY and the same pipeline upgrades itself to LLM extraction, LLM reflection, LLM-as-judge, and real embeddings. Your agent's own LLM stack is untouched either way.
• No server? The SDK writes to an embedded SQLite db (.journeyman/). journeyman up reads the same file. Set JOURNEYMAN_SERVER=http://host:port to ship traces over HTTP instead (works from other processes/languages — POST /api/traces).
• No config file? Fine. journeyman init writes an optional journeyman.yaml:

project: my-bot
entrypoint: app.main:my_agent      # lets the CLI/server run your agent for evals
learning:
  gate_suite: smoke                # the eval gate
  regression_threshold: 0.05
  auto_minutes: 0                  # >0: server runs learning cycles on a timer

Eval suites

# suite.yaml  ->  journeyman eval --load suite.yaml
name: smoke
gate: true
cases:
  - input: "How do I reset my password?"
    expect_contains: "portal.acme.com"
  - input: "VPN isn't connecting on my Mac"
    expect_contains: "6.2"
  - input: "Summarize our refund policy"
    judge: "Answer must state the 30-day window and the store-credit exception."

Checks: expect_exact, expect_contains, expect_contains_any, expect_not_contains, expect_regex, judge (binary LLM-as-judge), and scorer: module:fn for custom Python.

Safety guarantees of the SDK

• Never raises into your code; internal failures degrade to no-ops (one warning).
• Never blocks your hot path — persistence happens on a background thread with a bounded queue (drops, never blocks). jm.flush() for scripts/serverless.
• KeyboardInterrupt/SystemExit always propagate; your exceptions are recorded and re-raised unchanged.
• JOURNEYMAN_DISABLED=1 is a true kill switch.
• jm.status() tells you exactly what's running and where data goes.

Other languages

The learning loop runs in the server, so other languages need only a thin client. Official SDKs, same safety contract as Python (never throw, never block, kill switch):

• TypeScript / JavaScript — sdks/typescript (Node 18+, zero dependencies)
• Go — sdks/go (standard library only)

import * as jm from "journeyman-agents";
const myAgent = jm.agent(async (q, learnedContext) => { /* your agent */ });
await myAgent("How do I reset my password?");
jm.feedback(0.0, "wrong — the portal moved");

Anything else can speak the HTTP API directly: POST /api/traces, POST /api/feedback, GET /api/recall?q=... against journeyman up.

Python API

jm.init(project=..., autolog=True, server=None, db=None)   # all optional
@jm.agent / @jm.tool / @jm.step        # decorators (bare or with args; sync or async)
jm.wrap(fn)                            # functional form
with jm.trace("name", input=...):      # for code you can't decorate
jm.feedback(score, comment, trace_id=None)
jm.recall("query")                     # the learned-context block, on demand
jm.evaluate(fn=None, suite=None)       # run a suite from Python
jm.learn()                             # run a learning cycle from Python
jm.record_llm(model, input, output, tokens_in, tokens_out)  # custom LLM clients
jm.flush(); jm.status()

How much better does it get?

A working retrieval agent — it reads documents, extracts answers, picks options — answers four public benchmarks. It gets most of them wrong, which is the point: that's the starting line. Users correct its mistakes on the original wording, one learning cycle runs, and then it faces the same questions reworded, so it can't have memorised anything.

16% → 85%, a 5.4× improvement, from 174 corrections and one learning cycle that takes 12 seconds. Both columns are the same agent on the same reworded questions; the only difference is whether it has the lessons. Reproduce: python bench/improve.py.

What happens when the feedback is wrong?

This is the question every "learns from your corrections" tool skips, and it's the one that costs money. Take an agent that already works, hand it 10 confidently wrong corrections per benchmark, run a learning cycle, and measure how much of it still answers correctly:

Every batch is tested against your eval suite before it goes live and kept only if the pass rate holds. A lookup table has no equivalent — it applies whatever it's told, immediately. Reproduce: python bench/robustness.py.

Weakest benchmark: MBPP at 82%. The gate is exactly as good as the evals you give it, and 20 cases didn't cover enough of the poisoned tasks. Full detail in docs/BENCHMARKS.md.

Do the corrections stick?

Seven public benchmarks, one test each: the agent answers with no memory, every miss gets one correction on the original wording, one learning cycle runs, then it faces 40 rewordings it has never seen. Asking the same question again would only prove it memorized the test.

Three systems get the identical corrections: nothing, a keyword log (what you'd build yourself — store every correction, look it up by word overlap), and Journeyman.

The keyword log wins on raw recall, and we publish that. Looking up a correction you already have is hard to beat with plain word matching. What it can't do: it applies every correction with no gate, so one bad answer poisons the rest; it never deduplicates; and asked something nobody taught it, it returns a confident wrong answer 100% of the time (Journeyman: 69%). These are also Journeyman's floor numbers — no API key, hash embeddings.

Reproduce it in about a minute with no API keys: python bench/suite.py. Method, the rewordings, what we changed after seeing these results and what we refused to change: docs/BENCHMARKS.md.

Development

python -m venv .venv
.venv/Scripts/python -m pip install -e ".[dev]"
.venv/Scripts/python -m pytest tests

Docs: docs/DESIGN.md (architecture), docs/BENCHMARKS.md (what's measured and how), docs/RESEARCH.md (the research this design is grounded in), CONTRIBUTING.md.