Memory Decision Eval
A small offline evaluation inspired by the NORA paper Remember, Verify, or Ask? It scores the decision an agent declares and the tool calls recorded by its runner. The eight synthetic cases cover durable memory, one-time use, verification of changing facts, and clarification of ambiguous intent.
Run the demo
From a checkout with Python 3.10 or later, run python -m memory_decision_eval.
This uses fabricated predictions so you can inspect what happens when the
declared decision and the observed tool trace disagree. The example needs
neither model access nor an API key. Add --json for per-case results.
| Metric | Fabricated demo |
|---|---|
| Declared decisions correct | 7 / 8 |
| Observed tool traces correct | 6 / 8 |
| Both correct | 5 / 8 |
| Cases with an unexpected durable write | 1 |
Score your agent's traces
- Export the case inputs with
python -m memory_decision_eval --cases > cases.jsonl. - Pass
original_interactionandreuse_contextto your agent. Use the case ID for joining records. Keep benchmark IDs and reference labels outside the model input. Record the declared decision and normalize the memory-related calls from the runner's trace to the tool names below. Use the target namespace shown by the case export to normalize actual call targets. - Write one UTF-8 JSONL row per case. Run
python -m memory_decision_eval --predictions predictions.jsonl --json.
A prediction row has a case_id, a decision, and an array named tool_calls.
Here is one complete row for a verification case.
{"case_id":"delivery/stale","decision":"verify","tool_calls":[{"name":"verify_source","target":"cedar.delivery_window"}]}
| Declared decision | Reference trace |
|---|---|
persist |
One write_memory call for the intended memory key |
ephemeral |
An empty array of memory-related calls |
verify |
One verify_source call for the current fact |
clarify |
One ask_user call for the ambiguous intent or reference |
Tool scoring compares the observed sequence of names and targets with the
reference sequence. Wrong targets, missing calls, and extra memory-related
calls all affect the trace score. An unexpected durable write is a case
where the trace includes write_memory and the reference decision is
different from persist. Case IDs must be unique and all eight cases must
be present. Blank lines are allowed.
These fixtures are a smoke evaluation for scope and tool selection. The counts apply to these eight cases. Storage, tool execution success, and downstream task quality are outside the scoring contract. The original paper's results use its own larger benchmark and evaluation conditions.
Install and test
Install from the repository with python -m pip install .. The GitHub
release also provides a wheel. After installation, memory-decision-eval
is available as a shell command. Python integrations can import CASES,
Prediction, ToolCall, and evaluate from memory_decision_eval.
Run the tests with python -m unittest discover -s tests -v.
Research provenance
The implementation and all eight cases were written for this repository. The four decision types and the motivation for scoring declared decisions and tool traces separately come from the NORA paper by Baichuan Li, Junyi Yao, and Zihao Zheng. The dataset for this release consists of synthetic examples authored for the smoke evaluation.
Metadata
Release files for memory-decision-eval 0.1.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| memory_decision_eval-0.1.1.tar.gz | 7.7 kB | With provenance | Details for memory_decision_eval-0.1.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| memory_decision_eval-0.1.1-py3-none-any.whl | Python 3 | none | any | Details for memory_decision_eval-0.1.1-py3-none-any.whl |
Total release size: 15.6 kB
Release files / memory_decision_eval-0.1.1.tar.gz
| Download URL | memory_decision_eval-0.1.1.tar.gz |
|---|---|
| Size | 7.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
b026440c92de7d5a3c123b1738fad45e000f70cadc2cfbca6a29580725ab65ed
|
|
BLAKE2b-256 checksum How to use checksums |
3a4c923d7bd6530101cbb82b83abdbb674b9b00470ad6f8cdc9aec310df49b1a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.
Transparency logRelease files / memory_decision_eval-0.1.1-py3-none-any.whl
| Download URL | memory_decision_eval-0.1.1-py3-none-any.whl |
|---|---|
| Size | 7.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
cb9c53a26462a11fe5df5268943f965a4f3714da3cdc0b3e2b55a2f6676c599a
|
|
BLAKE2b-256 checksum How to use checksums |
49ebb8b3818087e33f0df7ca5672899713fcb753eec54ac6d0fec1c4a81c73e1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.
Transparency logSecurity for memory-decision-eval 0.1.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for memory_decision_eval-0.1.1.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.