forgetlayer 0.1.0

Provable deletion for AI agent memory: cascade-delete a memory across raw records, embeddings, summaries and caches, then independently verify it's gone and emit a signed receipt.

ForgetLayer — the attack that proves "delete" is a lie

You told your AI agent to forget the user's SSN. Here it is, 30 seconds later, pulled straight out of "deleted" memory.

Memory-enabled agents (Mem0, Zep, LangGraph memory + a vector DB) store user data across sessions. When a user exercises their GDPR right to erasure, apps call the store's delete() — and believe the data is gone.

It isn't. delete() drops the raw record and its embedding, but leaves the secret alive in derived summaries, cached answers, and duplicate records. This repo proves it, then shows the cascade that actually removes it.

Install

pip install forgetlayer            # core: numpy only, keyless
pip install "forgetlayer[receipts]"  # + Ed25519 signed receipts (cryptography)
pip install "forgetlayer[mem0]"      # + the live Mem0 adapter (mem0ai, fastembed, groq)

From a source checkout use pip install . (add the same [receipts]/[mem0] extras) or pip install -e . for development. The core stays numpy-only so the offline demo never needs a key.

Run it (zero API keys, ~10 seconds)

python -m forgetlayer.demo

Output:

=== After native delete(m1) - today ===
  [RECOVERED] summary   <- 'Alice: ... SSN on file: 123-45-6789.'
  [RECOVERED] cache     <- 'Your SSN on file is 123-45-6789.'
  [RECOVERED] neighbor  <- record 'm2': 'SSN same as before: 123-45-6789.'
  Forgetting Residue Score: 1.0   (1.0 = delete was a lie)

=== After ForgetLayer cascade - actually gone ===
  [    clean] summary
  [    clean] cache
  [    clean] neighbor
  Forgetting Residue Score: 0.0

What this proves (and what it does not)

Proves (honestly): the residue channels are real and store-agnostic. Deleting a record by id does not touch a summary already written from it, an answer already cached, or a duplicate in another record. This is true of every memory stack.

Does NOT claim:

• This is not a live Mem0/Pinecone run — that needs your API keys. The store here faithfully models the four tiers real stacks use; the residue mechanisms are identical. Real-store adapter is the next step (see below).
• The embedding is a deterministic offline hashing embedder (attack/embed.py), chosen so the demo is reproducible with no keys. Swapping in OpenAI embeddings does not change which channels leak.
• A residue score of 0.0 means "unrecoverable through the covered channels," not "provably non-existent everywhere." Side channels (LLM provider logs, app logs, analytics, backups) are out of scope and must be named on any real receipt.

Layout

forgetlayer/attack/embed.py     deterministic offline embedding
forgetlayer/attack/store.py     4-tier memory store: raw, vectors, summaries, cache
forgetlayer/attack/attacks.py   recovery attacks + Forgetting Residue Score
forgetlayer/attack/providers.py real Gemini embeddings + Groq LLM (real_demo)
forgetlayer/adapters/mem0.py    live Mem0 cascade (delete + history scrub)
forgetlayer/receipt.py          Ed25519-signed, hash-chained receipts
forgetlayer/cli.py              the `forget` command
forgetlayer/demo.py             the canary before/after
forgetlayer/tests/              CI proof: native delete leaks, cascade doesn't

python -m pytest forgetlayer/tests/ -q   # 14 passed

Verifier CLI + signed receipts

The independent verifier and the receipt are the point of the project — a store grading its own homework convinces no auditor. All keyless, against the demo store:

python -m forgetlayer verify                    # attack before/after cascade; prints residue
python -m forgetlayer delete --owner user:alice # cascade the owner, emit a signed receipt
python -m forgetlayer report                    # re-walk the audit chain, verify every signature

After pip install, these are also available as the forget console command (forget verify, forget delete --owner ..., forget report) — same behavior, no python -m prefix. delete/report need the [receipts] extra (cryptography) for Ed25519 signing; verify does not. Receipts and the signing key live under .forgetlayer/ (override with --home or FORGETLAYER_HOME).

A receipt is evidence of a deletion process, not proof of universal non-existence. It names the stores it covers, enumerates the side channels it does not (LLM provider logs, app logs, analytics, backups), and its residue score is measured over the covered channels only:

{
  "owner": "user:alice",
  "covered_stores": ["offline-demo-store (model)"],
  "out_of_scope": ["LLM provider logs ...", "application logs", "... backups ..."],
  "deleted": {"memories": 3, "summaries": 1, "cache_entries": 1},
  "closure_hash": "sha256:...",
  "residue_score": 0.0,
  "residue_scope": "covered channels only",
  "claim": "... evidence of a deletion process, not proof of universal non-existence.",
  "chain_prev": "sha256:...",
  "sig_alg": "ed25519",
  "signature": "..."
}

The audit log is hash-chained (Certificate-Transparency style): editing, deleting, or reordering any receipt breaks the chain and every signature after it — forget report exits non-zero and prints TAMPERED.

Real-store run (free keys)

Same attack, but with real Gemini embeddings and a Groq-written summary/answer — so the leaked artifacts are genuinely model-derived, not hand-authored.

export GEMINI_API_KEY=...   # free: https://aistudio.google.com/apikey
export GROQ_API_KEY=...     # free: https://console.groq.com/keys
python -m forgetlayer.real_demo

Groq serves the LLM (gemini-embedding-001 serves embeddings; Groq has no embeddings API). It runs both halves against the same real store:

REAL-STORE residue: native delete = 0.667  |  ForgetLayer cascade = 0.0
VERIFIED on real embeddings + real LLM: native delete leaked the SSN; the cascade removed it.

Native delete() leaks the SSN through the Groq-written CRM profile and a duplicate record surfaced by Gemini semantic search; the cascade removes both. The residue check is format-normalized, so writing the SSN as 123 45 6789 still counts as a leak — the verifier cannot be fooled into under-counting. Next up: a Mem0 + Pinecone adapter so it runs against the exact stack teams deploy.

Live Mem0 (the real store, not a model)

The above runs against a faithful model of a memory stack. This runs against real mem0ai — real local Qdrant vectors, real fastembed embeddings, real Mem0 history DB — and it corrected an assumption the model got wrong:

• On real Mem0, the LLM consolidates facts, so duplicate turns collapse into one memory. After Memory.delete(id), get_all and semantic search are clean — the model's "duplicate record" and "summary/cache" channels don't apply the same way.
• The real residue channel is Mem0's history DB. delete() drops the vector but the memory text survives verbatim in Mem0's SQLite history table (the ADD row's new_memory and the DELETE row's old_memory). So a user exercises erasure and their SSN is still sitting in history.db.

The cascade for Mem0 is therefore: Memory.delete() + scrub the history rows.

pip install mem0ai fastembed groq            # + cryptography for the receipt
export GROQ_API_KEY=...                       # infer=True: real LLM consolidation
python -m forgetlayer.mem0_demo
# no LLM budget? storage-only mode, no key needed:
FORGETLAYER_MEM0_INFER=0 python -m forgetlayer.mem0_demo

LIVE MEM0 residue: native delete = 1.0  |  ForgetLayer cascade = 0.0
  receipt: fr_... status=complete (covers mem0: qdrant vector store, mem0: history.db)
VERIFIED on live Mem0: native delete left the SSN in the history DB; the cascade removed it.

Honest scope limit found by experiment: Mem0's history table has no owner/user_id column — rows are keyed by memory_id. So owner-scoped history scrub is reliable only for ids ForgetLayer captures at delete time; history rows for memories deleted before ForgetLayer was present can't be owner-attributed and are out of scope. The receipt covers Mem0's own stores only (vector store + history.db), never the LLM provider that did consolidation, app logs, or backups. The integration test (tests/test_mem0_adapter.py) runs this keyless in CI.

Status

The claim — and the fix — hold on live Mem0 (its history DB is the real residue channel), with a signed receipt and a keyless integration test. Next adapters in evidence-ordered priority: pgvector, then Pinecone/Qdrant, then Zep.