Artemis

Run a small model on-device. Escalate only the hard tool calls to a frontier model. Let every escalation train your small model to need the big one less.

Apps that use AI pay the big, expensive model (Claude, GPT) for every tool call, even the trivial ones a tiny local model could handle. Artemis puts a small model on the device, checks its tool calls with a deterministic, schema-grounded verifier (not another LLM), and only escalates to the frontier model when the small one is actually wrong. Every escalation is auto-labeled correct/incorrect and saved as training data — so the small model gets better every night, deflects more, and the bill drops while you sleep.

small model → verify (deterministic) → pass?  ✓ done: on-device, private, free
                                     → fail?  ↑ frontier → verify → log the fix as training data
                                                                    ↓
                              nightly retrain → FITNESS GATE (promote only if it deflects
                                                more, without regressing) → ship

Why it's not clone-bait

• The verifier is deterministic. Parsed? Valid tool? Args match the JSON schema? Optional downstream oracle? Every call is labeled for free — no human, no LLM grader. That's what makes the data flywheel real.
• It compounds and it's gated. Escalations become a private training set; the fitness gate ships a retrain only when it measurably improves. Your traces and your fine-tuned model are yours.
• It works with real small models. Native OpenAI tool-calling and a prompted-JSON mode + streaming, so models that don't do native tools (most local ones, including Apple's on-device FM) still work.

Start with your own bill

Before changing any code, point Artemis at an inference log you already have. It tells you what you spent and how much of that spend ran through calls a deterministic verifier could have checked — the ceiling on what Artemis can deflect.

artemis audit mylog.jsonl                                    # ceiling, rate UNMEASURED
artemis audit mylog.jsonl --html report.html                 # shareable, self-contained
artemis audit mylog.jsonl --model http://localhost:11434/v1  # replace the ceiling with a MEASUREMENT

Runs entirely on the machine holding the log. No API key is needed for the analysis pass and nothing is uploaded — which answers "prove it on my traffic" and "I'm not sending you my prompts" with the same move. --html writes a report you can forward to whoever owns the budget, with the confidence statement and every assumption on its face.

It reads OpenAI, Anthropic, generic request/response, and Artemis's own trace format. What it will not do is flatter you: unpriceable calls are excluded rather than estimated, free-form prose is reported as not deflection-eligible, and a log spanning under a day gets no monthly projection at all.

Adopt it with one env var

artemis serve --upstream https://api.openai.com/v1 --small-model system
export OPENAI_BASE_URL=http://localhost:8787/v1     # that's the whole integration

--mode shadow forwards every call upstream unchanged and measures what the small model would have deflected — zero behavior change while you decide. Any Artemis-internal error falls through to your provider untouched.

Single tenant only. Artemis has no concept of a tenant. Every caller of one proxy writes full prompt text into one shared trace file, which is the corpus the nightly retrain consumes. Do not put one Artemis proxy in front of two parties who should not read each other's prompts. /artemis/stats and /artemis/report are unauthenticated; keep the proxy on loopback.

See it in 30 seconds

python examples/flywheel.py     # deflection 17% -> 100% after one overnight retrain
python -m artemis dashboard        # writes artemis_report.html — open it
python examples/smoke_live.py   # runs against a REAL on-device model if one's up

examples/flywheel.py output:

DAY 1 (fresh small model)   1/6 on-device (17%) | captured 5 training pairs from the misses
  ...overnight retrain, then the fitness gate...
  [PROMOTED] deflection 17% -> 100% (more deflection, no regression)
DAY 2 (retrained model)     6/6 on-device (100%)

Verified live against Apple's on-device Foundation Model: 3/3 tool calls correct, 100% on-device.

Use it with real models

from artemis import Request, TraceStore, SavingsMeter, route, ollama, local_fm, OpenAICompatibleProvider

small    = ollama("qwen3:4b")                 # local, native tool-calling
# or:    = local_fm("system")                 # Apple on-device FM (prompted mode)
frontier = OpenAICompatibleProvider("frontier", "https://api.openai.com/v1", "gpt-4o", api_key="...")

r = route(Request(system="...", messages=[{"role":"user","content":"..."}]),
          tools, small, frontier, trace=TraceStore(), savings=SavingsMeter())
print(r.handled_by, r.call)   # "small" most of the time, once it's trained

Nightly loop:

from artemis import nightly, MockTrainer  # swap MockTrainer for MLXTrainer to fine-tune for real
res = nightly(TraceStore("artemis_traces.jsonl"), small, frontier, eval_cases, MockTrainer(), keep_cases=eval_cases)
print(res.gate.summary())     # only promotes a retrain that deflects more without regressing

Layout

module	what it is
verifier.py	deterministic, schema-grounded tool-call verification (the moat seed)
router.py	the small→verify→escalate cascade
providers.py	Mock + OpenAI-compatible (native/prompted, streaming); ollama, local_fm
trace.py	captures escalations as SFT training pairs
eval.py	accuracy + deflection measurement
foundry.py	retrain + fitness gate (promote only if better)
dashboard.py	self-contained HTML savings report
agreement.py	replay oracle — AGREE / DISAGREE / ABSTAIN against your recorded answer
provenance.py	did the model read this value or invent it — works live, zero config
structural.py	deep schema (nested, $ref, format, ranges) + the checkability census
execution.py	did the world accept the call — read off the tool result already on the wire

The honest part: schema != correctness

The deterministic schema check catches malformed calls. It does not catch calls that are well-formed but wrong — verified live: a 0.5B model scored 33% and the schema check waved every wrong-but-well-formed call through (0 captured). Reproduced on this repo's own labeled fixture corpus: the schema check accepts 12 of the 25 tool-call candidates outright, 12 of which are wrong. That is the failure the oracle families below exist to route around.

route(request, tools, small, frontier, downstream=my_oracle)   # oracle = your correctness signal

Four oracle families, and what each one refuses to claim

Every one is deterministic plain code. No LLM-as-judge anywhere in the labeling path — an LLM judge costs money per call, which destroys the free-label property that is the entire point.

family	needs	works	catches	cannot see
agreement	your recorded answer (already in the log)	replay / audit	wrong-but-valid enum labels, wrong values	anything your schema does not define — it abstains
provenance	nothing	live	invented ids, placeholders, wrong recipients	computed numbers; legitimately generative text
structural	your schema	live	nested typos, bad formats, out-of-range, readOnly	semantically wrong but well-formed
execution	the tool result the app already sends back	retrospective	calls the world rejected (4xx)	valid-but-wrong calls that return 200

The distinguishing move is the third verdict. compare_calls returns AGREE, DISAGREE, or ABSTAIN, and abstention is load-bearing rather than a hedge. "Legitimately different but equally correct" is the obvious objection to scoring against a recorded answer, and the answer is not to adjudicate it more cleverly — it is to detect mechanically that your schema cannot decide, and decline. An abstention escalates (costs money, never ships a wrong answer) and writes no training label (poisons nothing). Both failure directions are routed into the safe one.

Equivalence is derived per argument from your own JSON Schema, never from meaning: enum members and numbers are adjudicable; an open string that differs, a datetime resolved from a relative expression, and an optional argument present on one side only are not.

from artemis import verified_agreement_oracle, provenance_oracle, structural_oracle

verify = verified_agreement_oracle(recorded_answer, tools)   # replay: schema AND agreement
verify = provenance_oracle(request_text, tools)              # live: did it invent that literal?
verify = structural_oracle(tools)                            # live: deep schema, zero config

Measured on the labeled fixture corpus (tests/fixtures_traffic.py, 11 records / 36 candidates)

Two error directions, named apart because they cost different things. A false flag (oracle says wrong, answer was right) escalates needlessly and writes a bad training pair — that is the one that poisons. A false accept (oracle says fine, answer was wrong) inflates the rate and ships a wrong answer.

                                        false flag   false accept   abstained   scored
structural + agreement  (the default)        0             0             6         19
shipped exact-arg match (before this)        3             0             0         25
schema check alone      (ships today)        0            12             0         25
deep structural alone                        0             7             0         25
provenance alone (live, no reference)        0             6             0         22
structured-output oracle (extraction)        0             0             0          6
numeric provenance (grounded prose)          0             0             0          5

The corpus is authored and hand-labeled. These are real counts over a stated, inspectable corpus — not an estimate of what your traffic would do. That number is what artemis audit --model exists to measure on your log.

artemis audit --model now measures agreement, not eligibility

This is the change that matters most. The headline used to be "spend that ran through calls a verifier could check" — an eligibility ceiling. It is now "spend on calls your small model answered exactly as your frontier model did" — a measurement, because in replay the reference is free: the frontier's answer is already sitting in the log you handed over.

  MEASURED AGREEMENT WITH YOUR RECORDED ANSWERS
  replayed                  : 9 calls against qwen2.5:0.5b
  agreed  (would deflect)   : 1
  disagreed (would escalate): 3
  ADJUDICATED               : 4   -> rate 25%
  ABSTAINED (undecidable)   : 5   -> coverage 44% of replayed calls
         5x  optional present on candidate only
         4x  open string, schema declares no equivalence

Real run, real model. Note what it refuses to do: abstentions are in neither the numerator nor the denominator, and the report says so on its face. Counting them as hits fabricates, counting them as misses understates, and burying them hides the coverage question.

It is AGREEMENT, not correctness, and the report says that too. Its ceiling is your frontier model's own accuracy — where that model was wrong, a small model agreeing with it is scored here as a success. A recorded answer that is empty or fails its own schema is refused the role of ground truth outright.

Is your catalog even checkable?

constraint_census answers, before anyone is promised a deflection rate, whether schema checking can work on your tools at all — a number no observability tool prints.

from artemis import constraint_census
print(constraint_census(my_tools).summary())
# 2 tools, 19 arg slots, 1.05 constraints/arg, 32% nested -> DENSE

A BARE catalog means schema verification will capture close to zero training pairs, and the zero-capture result above will reproduce exactly. Density is a lever you can pull: adding only the constraints your tool descriptions already state in prose (format: date-time on a field documented as ISO 8601, an enum copied onto a deprecated alias) is the cheapest capture improvement available.

The flywheel is real, not a mock

python examples/train_mlx.py ran a real LoRA fine-tune of a 0.5B model on captured tool-call data (16GB M1 Pro, 30 iters): val loss 6.03 → 0.29. On a held-out prompt:

base model  : As an AI language model, I don't have real-time weather data...
fine-tuned  : {"name": "weather", "arguments": {"city": "Phoenix"}}

The base model can't tool-call; after training on captured data, the same 0.5B model emits a structured call for a city it never saw. That's the whole promise, running.

Beyond tool calls: one machine, any verifier

verified_cascade is the tool-call cascade with the verifier made pluggable, so the same small→verify→escalate→capture loop works anywhere a cheap objective check exists. Proven in a second domain in examples/code_cascade.py: the small model drafts code, the tests are the oracle, failures escalate and capture (task → working code) pairs into the same trace store and retrain. That is the thesis — spend big compute only where a cheap check can't confirm the cheap answer — generalizing.

(There's also artemis/speculative.py, real token-level speculative decoding via MLX. Honest benchmark on a 16GB M1 Pro with a 0.5B→1.5B gap: 0.48x — slower. Token speculation needs a frontier-scale size gap; the action-level cascade is the better lever at this scale.)

Status

v0.7.0 — four deterministic oracle families (agreement / provenance / structural / execution) wired into the existing cascade, artemis audit --model reporting a MEASURED agreement rate with explicit abstentions, real token pricing, shareable HTML audit, OpenAI-compatible proxy, fail-open cascade + kill switch, fitness-gated retrain, real MLX (mlx_lm lora) fine-tuning run end-to-end. Real-model verified: Apple on-device FM, Ollama qwen2.5:0.5b (live replay through the full audit path), a real LoRA fine-tune. 230 tests, stdlib-only core.

Next: the honest gap is that abstention rate and capture rate are measured on an authored corpus, not on customer traffic — point artemis audit --model at a real log and count what actually abstains. Then: the retrospective consequence-label channel (a human correcting the model's output hours later is a free, perfect training label, and it is the cheapest unbuilt thing here).

MIT © Lunar Labs