evenkeel

Keeps a LangChain agent on an even keel when things fail: failures are classified and handled by kind, bad output is retried with a hint, spend is tracked in dollars with a budget, and failures that need a person pause the run for approval.

Ported from keel, a durable execution engine for AI agents. An independent project, not affiliated with LangChain. Experimental: 0.x, expect changes.

What it adds to LangChain's built-in middleware

Use it alongside the built-ins: ModelFallbackMiddleware for switching models, HumanInTheLoopMiddleware for approvals fixed in advance.

Install

pip install evenkeel          # rule-based classification
pip install "evenkeel[jev]"   # adds TypeSafe's Jev classifier

Use

from langchain.agents import create_agent
from langgraph.checkpoint.memory import InMemorySaver
from evenkeel import EvenKeelMiddleware, ModelPrice, PriceTable

agent = create_agent(
    "openai:gpt-4o-mini",
    tools=[...],
    middleware=[
        EvenKeelMiddleware(
            budget_usd=0.50,
            prices=PriceTable({"gpt-4o-mini": ModelPrice(input_per_mtok=0.15, output_per_mtok=0.60)}),
        )
    ],
    checkpointer=InMemorySaver(),  # needed for pauses
)

With Jev (needs TYPESAFE_API_KEY):

from typesafe_sdk import TypeSafeClient
from evenkeel import EvenKeelMiddleware, JevClassifier

EvenKeelMiddleware(classifier=JevClassifier(TypeSafeClient()))

How failures are handled

Retries and escalations stop at max_attempts (default 4). Each failure is recorded (error, kind, confidence, which classifier decided, action): in the agent's state under evenkeel_attempts when the call ends in a result, and on the raised error as error.evenkeel_attempts when it finally fails.

Some failures never reach the middleware, because LangChain turns them into messages for the model itself: a call to a tool that does not exist, and tool arguments that fail the tool's schema.

Pauses and how to resume

A failure that needs a person:

from langgraph.types import Command

agent.invoke(Command(resume={"decision": "retry", "hint": "approved by finance"}), config)
agent.invoke(Command(resume={"decision": "fail"}), config)
agent.invoke(Command(resume={"decision": "skip", "message": "Refunded by hand."}), config)  # tool calls only

A retry with a hint on a tool call sends the reviewer's words to the model, which can call the tool again; a tool cannot take a hint itself. A call that still needs a person after one approved retry fails. An answer that is not one of these values does not raise: the run pauses again, with an error field in the payload saying what was wrong.

Over budget:

agent.invoke(Command(resume={"approve": True, "budget_usd": 1.00}), config)
agent.invoke(Command(resume={"approve": False}), config)  # ends the run

The budget applies to each run: every new invoke on a thread starts with zero spend and the configured budget.

Limitations

• Budgets can overshoot by one call. The check runs before a model call, but a call's cost is known only after it. keel has the same limitation.
• No built-in prices. You supply them; a model without a price, or a response without token usage, is an error rather than a silent $0. Dated or versioned names (gpt-4o-mini-2024-07-18, claude-3-5-sonnet-20241022, gpt-4-0613, ...-latest) use the price of their base name; any other variant (gpt-4o-mini against gpt-4, gpt-4-turbo) needs its own entry.
• On resume after an escalation, the failing call runs once more before the decision applies, because LangGraph re-runs the paused node. For a tool with side effects, make it idempotent or use skip.
• Classification is only as good as the classifier. Rules only recognise explicit signals (status codes, network errors, evenkeel's error classes): on keel's 75 error messages from public SDK issues they pick the right kind 29 times and the right action 39 times. With Jev the results are below.

Classifier accuracy

Measured with evals/jev_eval.py on keel's labelled error sets: 75 error messages from public SDK issues and 60 sampled from 13 other repositories. Jev pinned to jev-1.13.0, one run on 2026-10-14. "Right action" counts a verdict as right when the default policy does the same thing as for the label (bad_input and fatal both fail).

Caveats: the labels are one person's (keel's author, who also wrote the classifier prompt), most messages come from development rather than production, and Jev's answers vary by about one case between runs. On the public set Jev answered 66 cases and the rules 9 (Jev below 0.5 confidence), of which 7 were wrong. keel's TypeScript cascade scored 46 to 47 and 59 to 60 on the same set (its results); the difference is in how often Jev falls below the confidence floor, and its cause has not been established.

Develop

uv sync --all-extras
uv run pytest