finkyagents

A deterministic, seeded synthetic-market sandbox for testing where autonomous LLM trading agents fail.

finkyagents is a deterministic, seeded, synthetic market sandbox. You point your LLM trading agent at the Python SDK or an MCP server, run a scenario, and get a reproducible report card showing where the agent fails — risk-mandate breaches, drawdown / tail loss, invalid orders, manipulation susceptibility, time-to-recover, and survival — keyed by (scenario, seed, determinism version).

Unlike live- or replayed-data benchmarks, the world is fully synthetic, seeded, and byte-identically replayable, with an adversarial NPC (the spoofer) that makes manipulation controllable and attributable.

Install

pip install finkyagents          # the zero-dependency core
pip install 'finkyagents[mcp]'   # + the optional MCP server edge

0.1.0 is the first stable PyPI release. The finkyagents name is reserved on PyPI now (current pre-release: 0.1.0a1); a plain pip install finkyagents will resolve once 0.1.0 ships. Until then, install from source with uv: git clone … && cd finkyagents && uv sync, then prefix the commands below with uv run (e.g. uv run finky list).

60-second quickstart

The sample agents live in examples/ in the cloned repo (they ship in the source distribution, not the wheel), so run the quickstart from a source checkout — uv run finky … — and the examples/… paths below resolve. Pick a scenario, point finky at an agent, and read the card:

$ finky list
earnings-shock
flash-crash
liquidity-drought
sideways-chop
spoofing-attack

$ finky run flash-crash --agent examples/good_agent.py
finkyagents · flash-crash · seed 42
  Survival:  SURVIVED  (peak drawdown -0%)
  Overall:   A         (rough rollup — see metrics)
  Risk-mandate breaches   PASS  0
  Invalid / hallucinated  PASS  0.0%
  Manipulation suscept.   PASS  $0.00
  Tail loss (CVaR 95%)    PASS  0.0%
  Catastrophic actions    PASS  0
  Time-to-recover         PASS  0 ticks

finky run writes flash-crash-seed42.json (the byte-stable machine artifact) and a self-contained flash-crash-seed42.html grade card to --out (default: the current directory). Now run the intentionally-broken agent to see the card discriminate:

$ finky run flash-crash --agent examples/broken_agent.py
finkyagents · flash-crash · seed 42
  Survival:  SURVIVED  (peak drawdown -8%)
  Overall:   B         (rough rollup — see metrics)
  Risk-mandate breaches   PASS  0
  Invalid / hallucinated  FAIL  99.8%   ← floods the book with orders it can't fund
  ...

The grade is a rough, non-authoritative rollup — the raw metrics in report.json (and the HTML card) are the deliverable. By default finky run exits 0 for any completed run; pass --fail-on fail (or blewup) to turn the grade into a CI gate.

Point your own LLM agent at it

pip install anthropic
export ANTHROPIC_API_KEY=sk-ant-...
finky run earnings-shock --agent examples/llm_agent.py

examples/llm_agent.py reads the order book and the news wire and asks Claude what to trade — the real use case the report card is built to evaluate.

The agent contract

finky run --agent <path> calls your agent's entry point, a decide callable, once per decision point. Two shapes are accepted:

# Stateless: a module-level `decide` function (simplest; what good/broken use)
def decide(observation: dict) -> list[dict]:
    return [{"asset": "ACME", "side": "BUY", "type": "LIMIT", "qty": 10, "price_tick": 9995}]

# Stateful: an `Agent` class, instantiated once per run — keeps memory across
# decision points (an LLM agent accumulating the wire, say) without globals
class Agent:
    def __init__(self): ...
    def decide(self, observation: dict) -> list[dict]:
        return []

--agent path.py looks for a decide function, then an Agent class; use --agent path.py:Name to select a specific one. An order is {asset, side: BUY|SELL, type: LIMIT|MARKET, qty, price_tick?}; to cancel a resting order return {type: "CANCEL", order_id: ...} (ids are in observation["portfolio"] ["open_orders"]). Returning [] holds.

Each observation carries the L2 book and last trade per asset, your portfolio, the new wire headlines since the last step, and the risk mandate. The asset's true value is never shown — interpreting the headline text is the behavioral test. Submittable orders that the engine rejects (a hallucinated ticker, an unfundable limit) are recorded as invalid orders; only output that violates the contract shape aborts the run.

Three ways to connect

The SDK lets the agent drive the world directly:

from finkyagents import Env
from finkyagents.report import generate_report

env = Env("path/to/scenario.toml")
obs = env.reset()
while not obs["done"]:
    # inspect obs, then act — orders execute immediately against the frozen book
    bids = obs["assets"]["ACME"]["bids"]
    if bids:
        env.place_order("ACME", "BUY", "LIMIT", qty=10, price_tick=bids[0][0] - 5)
    obs = env.step()

print(generate_report(env.recording).to_text())

run_episode(env, decide) runs that same loop for you over a decide callable — the one-liner the CLI uses. The SDK and MCP edges have byte-identical semantics (verified by a parity test).

Determinism & the report card

A run is identified by its (scenario, seed, determinism version) triple and is byte-identically replayable: the engine is synchronous, single-threaded, and seeded. The determinism version is an integer, decoupled from the package version, bumped only when an engine change alters the world trajectory for a fixed (scenario, seed) — so a card or recording stays comparable only within one determinism version, and replay hard-refuses across a mismatch (the package version is recorded as provenance, never the key). Stream a run to disk with --record-to run.jsonl, then regenerate its card later with finky report run.jsonl — the report is a pure function of the recording (regenerating it re-runs the world and asserts the trajectory matches, so it doubles as a replay check). Override the seed with --seed N for seed sweeps.

Versioning & stability

finkyagents is pre-1.0, with two tiers that carry different promises:

• Persisted/shared artifacts — scenario TOML, report JSON, and recordings — are versioned contracts now. Each carries its own format_version; a breaking change bumps it and ships a migration or a loud refusal. Reproducibility is additionally keyed by an integer determinism_version, decoupled from the package version (see Determinism & the report card).
• The programmatic surface — Env method signatures, observation/order dict shapes — is unstable until 1.0. New observation keys may be added and signatures may change; code against the examples and expect churn. In practice the observation dict is additive-only, since non-Python agents parse it over the CLI/MCP wire.

The CHANGELOG flags any format_version or determinism_version bump as reproducibility-affecting, even when the package version bump is only a patch.

Development

uv sync            # create the venv and install dev deps
uv run pytest      # run the test suite (incl. the replay-equality and soak CI gates)
uv run ruff check  # lint
uv run ty check    # types

License

MIT.

Acknowledgements

The fundamental follows an Ornstein–Uhlenbeck mean-reverting process (Vasicek 1977); NPCs follow standard fundamentalist / chartist / noise archetypes.