Skip to main content

Memory Decision Eval

A small offline evaluation inspired by the NORA paper Remember, Verify, or Ask? It scores the decision an agent declares and the tool calls recorded by its runner. The eight synthetic cases cover durable memory, one-time use, verification of changing facts, and clarification of ambiguous intent.

Run the demo

From a checkout with Python 3.10 or later, run python -m memory_decision_eval. This uses fabricated predictions so you can inspect what happens when the declared decision and the observed tool trace disagree. The example needs neither model access nor an API key. Add --json for per-case results.

Metric Fabricated demo
Declared decisions correct 7 / 8
Observed tool traces correct 6 / 8
Both correct 5 / 8
Cases with an unexpected durable write 1

Score your agent's traces

  1. Export the case inputs with python -m memory_decision_eval --cases > cases.jsonl.
  2. Pass original_interaction and reuse_context to your agent. Use the case ID for joining records. Keep benchmark IDs and reference labels outside the model input. Record the declared decision and normalize the memory-related calls from the runner's trace to the tool names below. Use the target namespace shown by the case export to normalize actual call targets.
  3. Write one UTF-8 JSONL row per case. Run python -m memory_decision_eval --predictions predictions.jsonl --json.

A prediction row has a case_id, a decision, and an array named tool_calls. Here is one complete row for a verification case.

{"case_id":"delivery/stale","decision":"verify","tool_calls":[{"name":"verify_source","target":"cedar.delivery_window"}]}
Declared decision Reference trace
persist One write_memory call for the intended memory key
ephemeral An empty array of memory-related calls
verify One verify_source call for the current fact
clarify One ask_user call for the ambiguous intent or reference

Tool scoring compares the observed sequence of names and targets with the reference sequence. Wrong targets, missing calls, and extra memory-related calls all affect the trace score. An unexpected durable write is a case where the trace includes write_memory and the reference decision is different from persist. Case IDs must be unique and all eight cases must be present. Blank lines are allowed.

These fixtures are a smoke evaluation for scope and tool selection. The counts apply to these eight cases. Storage, tool execution success, and downstream task quality are outside the scoring contract. The original paper's results use its own larger benchmark and evaluation conditions.

Install and test

Install from the repository with python -m pip install .. The GitHub release also provides a wheel. After installation, memory-decision-eval is available as a shell command. Python integrations can import CASES, Prediction, ToolCall, and evaluate from memory_decision_eval.

Run the tests with python -m unittest discover -s tests -v.

Research provenance

The implementation and all eight cases were written for this repository. The four decision types and the motivation for scoring declared decisions and tool traces separately come from the NORA paper by Baichuan Li, Junyi Yao, and Zihao Zheng. The dataset for this release consists of synthetic examples authored for the smoke evaluation.

Metadata

Release files for memory-decision-eval 0.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for memory-decision-eval 0.1.1
File Size Uploaded Provenance Actions
memory_decision_eval-0.1.1.tar.gz 7.7 kB With provenance Details for memory_decision_eval-0.1.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for memory-decision-eval 0.1.1
File Interpreter ABI Platform Actions
memory_decision_eval-0.1.1-py3-none-any.whl Python 3 none any Details for memory_decision_eval-0.1.1-py3-none-any.whl

Total release size: 15.6 kB

Release files / memory_decision_eval-0.1.1.tar.gz

Download URL memory_decision_eval-0.1.1.tar.gz
Size 7.7 kB
Tags Source
SHA-256 checksum
How to use checksums
b026440c92de7d5a3c123b1738fad45e000f70cadc2cfbca6a29580725ab65ed
BLAKE2b-256 checksum
How to use checksums
3a4c923d7bd6530101cbb82b83abdbb674b9b00470ad6f8cdc9aec310df49b1a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.

Transparency log

Release files / memory_decision_eval-0.1.1-py3-none-any.whl

Download URL memory_decision_eval-0.1.1-py3-none-any.whl
Size 7.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
cb9c53a26462a11fe5df5268943f965a4f3714da3cdc0b3e2b55a2f6676c599a
BLAKE2b-256 checksum
How to use checksums
49ebb8b3818087e33f0df7ca5672899713fcb753eec54ac6d0fec1c4a81c73e1
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.

Transparency log

Security for memory-decision-eval 0.1.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for memory_decision_eval-0.1.1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.1 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page