meme-core

Memory-Compute Pareto analysis core — memory strategies (with the guard's 2×2 decomposition) and solver-based Best-of-N for small language models

What is this?

meme-core extracts the reusable parts of the MEME project as a standalone Python library:

1. Memory strategies, including the guard's 2×2 (guard)

Structured memory hurts pure-retrieval tasks: entity-grouped memory (M3) costs 10–50 percentage points in every model we tested (median −30pp on TriviaQA-lite, 5/5 models negative, sign test p = 0.0625).

The fix is to stop telling the model to trust a block that was derived from the question. But "the guard" as usually implemented changes two things at once, so it cannot say which change does the work. This package therefore ships the full 2×2:

	block wording (Using ONLY the facts above)	question wording (the QUESTION is authoritative)
no fallback	M2 / M3 / M4	M2W / M3W / M4W (wording-only)
with fallback	M2G / M3G / M4G (prose fallback)	M2Q / M3Q / M4Q

What the experiments behind this package found (full numbers in the paper):

• On retrieval tasks the two factors are substitutes: the prose fallback alone restores the whole loss in 14 of 14 cells that have one, the question-authoritative wording alone in 8 of the 14.
• On constraint tasks the fallback is inert (it never fires when there is material to structure) and the question-authoritative wording is harmful (−10 to −40pp) — there the block is the material.

So the rule is task-dependent, not universal: if the task has nothing to structure, use prose; if it has everything to structure, keep the block authoritative.

2. Solver-based Best-of-N (solver_bon)

Best-of-N without a verifier has no selection criterion and degenerates to a random sample. Using a constraint solver as the reward model makes BoN deterministic and free of extra LLM calls on constraint-satisfaction tasks.

Key insight (MEME v3.9+):

• For tasks with verified unique solutions (ZebraLogic), the constraint solver is the ground truth
• No LLM call needed for verification — deterministic, reproducible
• On our leak-free hard set (20 problems), solver-selected Best-of-N lifts the 3B model from 40% to 85% (+45pp, McNemar p = 0.022) — the only significant positive cell in the whole study, and the only intervention whose sign we can predict before running it

meme_core.answer.extract_answer is the answer-extraction rule used for every number we report, so re-scoring stored samples with this package reproduces the paper's accuracies.

Installation

pip install meme-core

Or from source:

pip install -e .

Quick Start

The 2×2: fallback × wording

from meme_core.guard import GuardedMemory, Problem

problem = Problem(
    id=1,
    question="Who is the engineer?",
    expected="Alice",
    constraints=[
        "Alice is the engineer",
        "Bob is the doctor",
    ],
    structure_kind="constraints",
)

# block wording, no fallback
print(GuardedMemory(strategy="M3").render(problem))

# question wording, NO fallback  (the "wording-only" corner of the 2x2)
print(GuardedMemory(strategy="M3W").render(problem))

# question wording + prose fallback
print(GuardedMemory(strategy="M3Q").render(problem))

When structure_kind="none", the guarded strategies fall back to prose:

problem = Problem(
    id=2,
    question="What is the capital of France?",
    expected="Paris",
    constraints=[],
    structure_kind="none",
)

# M3G -> plain prose (no scaffold, no authority claim)
print(GuardedMemory(strategy="M3G").render(problem))

# M3W -> still renders the (empty) block, because the wording-only corner
#       must NOT get the fallback
print(GuardedMemory(strategy="M3W").render(problem))

Best-of-N with the constraint solver

from meme_core.solver_bon import best_of_n, best_of_n_from_texts
from meme_core.guard import Problem

problem = Problem(
    id=1,
    question="Who is the engineer?",
    expected="lawyer",
    aliases=["advocate", "attorney"],
)

# From extracted candidate answers ...
winner, scores = best_of_n(["doctor", "lawyer", "engineer", "lawyer"], problem, use_solver=True)
print(winner)                      # "lawyer"
print([s.confidence for s in scores])  # [0, 100, 0, 100]

# ... or straight from the raw model responses
texts = [
    "Let me think. The engineer is Bob.\nAnswer: Bob",
    "Alice holds that post.\nAnswer: lawyer",
]
winner, scores = best_of_n_from_texts(texts, problem, use_solver=True)
print(winner)                      # "lawyer"

Memory Strategies

Strategy	Fallback	Authority wording	Description	Use case
M0	—	—	Raw prose question (baseline)	Control / pure retrieval
M1	—	—	Full context + enumerate constraints	General purpose
M2	—	block	Structured bullet constraints	Compression
M3	—	block	Entity-grouped facts (auto-extracted)	Entity-centric reasoning
M4	—	block	Knowledge-graph triples	Relationship reasoning
M2W/M3W/M4W	no	question	Wording-only: the block is always rendered	Isolating the wording factor
M2G/M3G/M4G	yes	block	Guarded, block-authoritative	Constraint tasks (recommended)
M2Q/M3Q/M4Q	yes	question	Guarded, question-authoritative	Retrieval tasks (recommended)
B0	—	—	Long-context baseline (repeat the context)	Context-length baseline
B1	—	block	RAG baseline (top-k retrieved items)	Retrieval baseline

B1 needs the retrieved items, which is where your own retrieval layer comes in:

prompt = GuardedMemory(strategy="B1").render(
    problem, retrieved=["...item 1...", "...item 2..."], k=2, n=10
)

The structure_kind Field

Problems declare what auxiliary material they provide, which determines guard behavior:

structure_kind	Example Task	Guard Behavior
constraints	ZebraLogic	Organize constraints into structured view
quantities	GSM8K-Hard	Organize numeric givens
options	ARC	Organize answer choices
facts	HotpotQA	Organize facts for chaining
none	TriviaQA, HumanEval	Fall back to prose (no scaffold)

Architecture

meme_core/
├── __init__.py           # Package entry point
├── answer.py             # extract_answer — the answer-extraction rule we score with
├── guard/
│   ├── __init__.py       # Guard module exports
│   ├── core.py           # GuardedMemory class, render_prompt, Problem
│   └── templates.py      # Prompt templates
└── solver_bon/
    ├── __init__.py       # Solver BoN exports
    └── core.py           # ConstraintSolver, best_of_n, normalize_for_comparison

Examples

python3 examples/guard_factorial.py          # render the 2x2 for one problem
python3 examples/gemma4_26b_improvement.py   # stored gemma4:26b Best-of-N numbers

examples/gemma4_26b_improvement.py prints stored results from the MEME project (no model calls). Note the task label: the constraint numbers for gemma4:26b are on ZebraLogic-easy (10 problems, data_sha 252183ea9177), not on the 20-problem hard set — the hard set was re-generated after the leakage audit and gemma4:26b has not been re-run on it.

Key Results (from the MEME project)

All numbers below are single cells or sign-replication counts, quoted with their uncertainty. A 10-problem cell cannot resolve a 10pp difference (that is one problem), so read these as effect sizes, not as verdicts.

Retrieval: structured memory hurts, in every model

Task	M0	M3 (unguarded)	M3G	M3Q
TriviaQA (qwen3:1.7b)	100%	60%	100%	100%
HotpotQA (qwen3:1.7b)	90%	60%	90%	90%
GSM8K-Hard (qwen3:1.7b)	70%	70%	80%	70%
ARC (qwen3:1.7b)	90%	100%	90%	100%

Cross-family, TriviaQA M3 → M3G: qwen3:1.7b +40pp, qwen3:8b +30pp, llama3.2:3b +50pp, phi3.5 +30pp — the penalty is present in 5/5 models (median −30pp, sign test p = 0.0625).

Constraint tasks: the wording must not be relaxed

On the leak-free hard set (H1–H20, 20 problems) the question-authoritative variants are worse than the block-authoritative ones in six of the seven directly comparable pairs (median −12.5pp: −40, −30, −15, −10, −10, −10pp). The four significant cells in the whole study are all of this kind (−35 to −40pp, p < 0.05).

Compute: verifier-based Best-of-N replicates, memory does not

Across 27 model×family Best-of-N cells: 16 positive, 11 ties, 0 negative. On the hard set: Qwen3-1.7B 75%→95%, Qwen3-8B 85%→100%, Llama-3.2-3B 40%→85%, Phi-3.5 45%→70% (median +22.5pp). The same four samples spent on a majority vote instead are worth a median of +7.5pp and are negative in one model. The difference is the selection rule, not the sampling.

Citation

If you use meme-core in your research, please cite:

@misc{meme2026,
  title  = {Memory is the Cheap Axis: A Paired, Audited Re-analysis of
            Test-Time Memory and Compute for Small Language Models},
  author = {Abel},
  year   = {2026},
  note   = {Preprint, https://github.com/abell12134/meme-llm}
}

References

• MEME Project (meme-llm) — Full research project with data, code, and results
• Paper (arXiv) — Full methodology and results (link added on upload)

License

Apache 2.0 — see LICENSE for details.