Skip to main content

dspy-wsp

Weight-space programming as a DSPy optimizer.

WeightSpace improves a DSPy program by having a frontier LM, the writer, edit the student model's weights directly. The writer reads the weights in a sandboxed Python kernel. It runs your program on practice examples through a budgeted forward-pass tool, works out what goes wrong inside the network, and saves targeted edits. It uses no gradients and no training.

The method comes from Sparks of Weight-Space Programming by Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab (MIT CSAIL, 2026). In their Experiment 8, a frontier model edited 0.016% of Qwen3.5-0.8B's weights. The edit raised MATH-500 from 7.9% to 48.1% by removing a direction that drove the model's looping. This package rebuilds that harness around DSPy programs and metrics.

On the Typed Decisions benchmark, WeightSpace raised the test accuracy of three small decision models by 13 to 16 points, and the gain compounds with dspy.experimental.ReAnchor to reach 17 to 18 points.

Bar chart of test accuracy on the Typed Decisions benchmark for d1-omni-600M, NanoJev and d1-2B: unoptimized, with ReAnchor, with WeightSpace, and with both.

This is an early release. We are releasing it outside the core DSPy library so that more people can try it and tell us what breaks. Please report problems and results, and tell us which models you would like us to support, in GitHub issues or on the DSPy Discord.

Contents

Install

pip install dspy-wsp

The writer's code runs in a sandbox (see Safety):

  • macOS uses the built-in sandbox-exec. It needs no setup.
  • Linux uses bubblewrap: apt install bubblewrap. On Ubuntu 24.04, AppArmor blocks the user namespaces that bubblewrap needs. Allow them with sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0.

The base install runs every supported decision model, with MLX on Apple silicon and PyTorch elsewhere. Two extras add more:

  • pip install "dspy-wsp[vllm]" adds the vLLM backend for text-generating LLMs on NVIDIA GPUs.
  • pip install "dspy-wsp[examples]" adds the datasets for the benchmark scripts in examples/.

Use

Before you start:

  • Requirements: Python 3.10 or later, and an API key for the writer. The example below reads OPENAI_API_KEY.
  • Time and cost: a 25-pass run on a decision model took between 78 and 167 minutes in our tests, and it cost between $5.42 and $7.63 in writer calls. Set max_passes=3 for a first run that checks your setup.
  • Writer: use gpt-6-astra at maximum reasoning effort. It wrote better edits than gpt-6.1-sol on two of our three models (see Results).

WeightSpace works like the other DSPy optimizers. It takes your metric and your module, and compile returns the improved module. Here the student is d1-omni-600M, a decision model that runs on a laptop:

import json
from typing import Literal

import dspy
from dspy_wsp import WeightSpace, load_decision_backend


class Triage(dspy.Signature):
    """Triage a support ticket."""

    ticket: str = dspy.InputField()
    urgent: bool = dspy.OutputField(desc="Does this ticket need a reply today?")
    team: Literal["billing", "technical", "account"] = dspy.OutputField(desc="Which team should handle it?")


def metric(example, pred, trace=None):
    return ((pred.urgent == example.urgent) + (pred.team == example.team)) / 2


def load(path):
    """One labeled ticket per line: {"ticket": "...", "urgent": true, "team": "billing"}"""
    return [dspy.Example(**json.loads(line)).with_inputs("ticket") for line in open(path)]


train, test = load("tickets_train.jsonl"), load("tickets_test.jsonl")

backend = load_decision_backend("d1")   # d1-omni-600M at the tested revision
optimizer = WeightSpace(
    metric=metric,
    writer_lm=dspy.LM("openai/gpt-6-astra", model_type="responses", reasoning_effort="max"),
    backend=backend,
    max_passes=25,
    holdout=0.25,
    run_dir="runs/triage",
)
compiled = optimizer.compile(dspy.Predict(Triage), trainset=train, valset=test)

compiled(ticket="I was charged twice this month.")   # runs on the edited weights
print(compiled.wsp_patch.summary())
print(compiled.wsp_report["validation"]["paired"])

To try a run without data of your own, clone this repository and run the benchmark script's short mode:

pip install -e ".[examples]"
OPENAI_API_KEY=... python examples/typed_decisions.py --smoke

How compile uses your data:

  • trainset becomes the writer's practice set.
  • holdout moves a quarter of those examples to a validation set. The writer can score edits on it but never sees its examples. An integer moves that many examples.
  • valset stays hidden from the writer. The optimizer uses it once at the end, and it compares the original and edited weights on the same examples with the same seeds.
  • Metrics can return a number, a bool, or dspy.Prediction(score=..., feedback=...) as GEPA metrics do. The writer reads any feedback text in the records.

A decision model reads bool outputs as yes or no questions and Literal outputs as choices. Each output needs a question in its desc. The Choice, Noul and Score types in dspy.experimental add criteria for each option and ordered score levels, and they return each option's probability.

Results

We edited three decision models on the Typed Decisions benchmark with gpt-6-astra at maximum reasoning effort as the writer. The writer had one conversation with 25 forward passes for each model. The chart at the top of this page scores the answers each program returns on 400 held-out test cases. Each gain is the paired gain over the unoptimized model in points. The standard errors are between 1.1 and 1.3 points.

  • ReAnchor: dspy.experimental.ReAnchor fits how the program turns probabilities into answers, i.e., its Noul thresholds, Score cut points and Choice weights. It changes no weights, so it stacks on a WeightSpace edit.
  • The writer matters: with gpt-6.1-sol as the writer, the WeightSpace edits alone gained 5.8, 12.1 and −0.6 points on the same three models.
  • The paper's rule: in these runs, the writer could recompute the decision head in its kernel and fit edits there. The paper's writer could run the network only through the forward-pass tool. We ran gpt-6-astra on d1-omni-600M again under the paper's rule. Its edit gained 2.8 ± 0.7 points alone, and it added nothing to ReAnchor. Pass simulate_head=False to use the paper's rule.

Our unoptimized scores differ from the benchmark's public leaderboard. The leaderboard lists d1-2B at 63.7%, and our chart shows 60.6%. Two things differ:

  • Request format: the leaderboard sends each case's own state and questions to the model. We send the requests that a DSPy program builds from its signature.
  • Measure: the leaderboard scores the model's most likely option. We score the answer the program returns, after its thresholds and cut points. On the most likely option, our unoptimized d1-2B scores 62.2%.

The leaderboard does not list d1-omni-600M or NanoJev. An edited model has practiced on the benchmark's train split, so it belongs with the leaderboard's fitted models and not with its zero-shot table.

The results folder's README has the full table with each gain's standard error, the gpt-6.1-sol comparison, both accuracy measures, and every run and edit.

How a run works

The writer works in turns. Each turn it explains its next step and sends exactly one code block, marked python or forward:

  • A python block runs in a persistent kernel with no GPU and, under the sandbox, no network. The kernel predefines NAMES, SHAPES, load(name), load_original(name), save_edit(name, tensor), drop_edit(name), edited() and tok. Gradients are disabled.
  • A forward block sends a JSON request to the forward-pass tool, the only way to run the model:
    • generate runs your program on up to 512 practice items with the original weights, the edited weights, or "both". Every answer is scored with your metric. "both" uses the same seed for each item and reports the paired gain with its standard error. The records keep every LM call's full prompt and completion. Pooled activations are optional.
    • teacher_forced reads up to 1,024 texts and returns log-probs, top-k predictions, hidden states, MLP activations or attention probabilities as .npy files.

Each forward pass costs one unit of budget, whatever its size. A refused request costs nothing. The session continues until the writer has used every pass. Then it leaves its chosen edit in work/edited/ and writes a final answer.

The harness follows the paper's rules:

  • Guardrails: save_edit refuses edits to the embeddings, the output layer and the final norm. It also refuses edits that only rescale a whole tensor. The host checks every edit again before each pass.
  • Limits: the writer may run Python up to 300 times, for up to 15 minutes and 10,000 characters of output each time. Our 25-pass runs used between 33 and 43 Python runs.
  • Explanations: a block sent without an explanation is returned unrun, at most twice in a row.
  • Handoffs: every 40 turns the conversation restarts with the task, the writer's notes.md and its current edit.

Every turn is written to disk. Pass resume=True with the same run_dir to continue an interrupted run.

By default, a run is one long conversation. For decision models, search=GEPASearch(...) runs GEPA's population search instead. The writer still works in the Python kernel with the same tools. The search keeps a population of candidate edits and runs in rounds:

  1. GEPA picks a parent from its Pareto frontier over the validation examples.
  2. GEPA scores the parent on a minibatch of practice examples.
  3. The writer gets a short session that starts from the parent's edit and notes, with the minibatch's feedback and a small budget of forward passes. The edit it leaves becomes the child.
  4. GEPA scores the child on the same minibatch. If the child scores higher, GEPA scores it on the validation set and adds it to the population.
from dspy_wsp import GEPASearch, WeightSpace

optimizer = WeightSpace(metric=metric, writer_lm=writer, backend=backend, max_passes=51, holdout=300,
                        search=GEPASearch(min_rounds=10, minibatch_size=8, passes_per_proposal=2))
  • max_passes counts every pass: the writer's passes and each of GEPA's scorings. A round costs at most 3 + passes_per_proposal passes.
  • WeightSpace refuses a max_passes that can't pay for min_rounds rounds, and a max_code_runs below min_rounds * code_runs_per_proposal (30 by default).
  • A parent that scores perfectly on its minibatch can't be beaten there, so GEPA skips that round's writer session. The parent's scoring still costs one pass.
  • A writer call that times out, drops its connection or runs past the model's output limit is retried twice. If it still fails, or every example fails to run, the search stops. The run then finishes with the best candidate so far, and the report's search.error says why.
  • Writers can't run validation passes in a round, because GEPA uses the validation set to select candidates.
  • Each round is written to rounds.jsonl and progress.md when it ends. Every candidate's edit and notes are kept in candidates/.
  • Pass resume=True with the same run_dir to continue a stopped search. It picks up GEPA's saved state, the round log and the pass count, and it reuses the seed's saved validation scores.

The example scripts take --search gepa --rounds 10. To compare the two searches, give the conversation the same --passes.

What you get back

runs/triage/
  task_prompt.txt      what the writer was told
  turns/tNNN/          each turn's prompt, reply and token usage
  transcript.jsonl     every Python run and forward pass
  state.json           the turn count, budget and writer cost, for resume
  work/                the writer's folder: notes.md, edited/, gen_k.jsonl, fwd_k/, practice/
  final_answer.md      the writer's account of its edit and the evidence
  final_check/         the valset comparison (never shown to the writer)
  patch/               manifest.json + edit.npz
  report.json          the budget used, the final check and the patch summary

A GEPA search keeps each round's turns under rounds/. It also writes candidates/, rounds.jsonl, progress.md and gepa_state/.

compiled.save("program.json") records the base model, its settings and the patch location. program.load("program.json", allow_unsafe_lm_state=True) rebuilds the edited model; DSPy asks for that flag before it imports a custom engine.

The patch stores only the changed entries, plus the sha256 of every base and edited tensor. Applying it checks both hashes, so anyone with the public base model can rebuild the edited weights bit for bit.

To deploy an edited LLM, write a full checkpoint and serve it like any other model. This example applies the patch from a run in runs/qwen-math:

dspy-wsp apply ~/.cache/huggingface/.../Qwen3.5-0.8B runs/qwen-math/patch qwen-edited/
vllm serve qwen-edited/
program.set_lm(dspy.LM("openai/qwen-edited", api_base="http://localhost:8000/v1"))

Safety

The writer's code runs on your machine. With sandbox="auto" (the default), the kernel runs in a sandbox: bubblewrap on Linux and sandbox-exec on macOS. The sandbox is an allowlist:

  • Read: the system's libraries, the Python install and its packages, this package, and the model's files. Your home folder, your keys and your other projects are outside it.
  • Write: the run's work folder and a scratch folder.
  • Network: none.

Everything the kernel prints goes to the writer's API, so the writer's provider sees the model's weights and your practice examples as the writer reads them. The host reads the writer's notes and edits from the work folder, and it refuses links there.

If no sandbox is available, WeightSpace(...) raises an error before it loads anything. If your Python install keeps libraries outside its own prefix (e.g. /nix/store), name those folders in sandbox_readonly=[...]. sandbox="none" runs the writer's code with your permissions and network; use it only on a machine you're willing to hand to the writer.

Decision models

A System One decision model reads a state and typed questions (Choice, Noul and Score) and returns a probability for every option. It generates no text. WeightSpace supports three families, each with our own loader in MLX and in PyTorch:

Backend Models Alias Network
D1Backend Liquid AI's d1-omni models, text path "d1" (d1-omni-600M) An encoder with a decision head
NanoJevBackend NanoJev models "nanojev" A Qwen3 backbone with a decision head
D1LMBackend Liquid AI's d1 models built on LFM3 "d1-2b" (d1-2B) A language model that answers at each question's answer position

D1LMBackend has no separate head, and its output layer is the protected token embedding, so the writer's edits land in the backbone. It answers Choice, Noul and Score questions and refuses d1-2B's other question types.

We have tested these three checkpoints so far. If you run another open decision model, please tell us how it went, and tell us which models you want supported, in GitHub issues or on the DSPy Discord.

Load a model. load_decision_backend takes an alias, a Hugging Face repo id or a local folder. It reads the checkpoint's config to pick the family, and it refuses other architectures. The aliases open the checkpoints we tested, at pinned revisions.

from dspy_wsp import load_decision_backend

backend = load_decision_backend("nanojev")                      # C-Tianyu/NanoJev at the tested revision
backend = load_decision_backend("someone/nanojev-finetune")     # any checkpoint of a supported family
backend = load_decision_backend("/path/to/checkpoint", framework="torch")

Check a new checkpoint. Run the check before you spend writer calls on a model we have not tested:

dspy-wsp check someone/nanojev-finetune

The check confirms four things. check_decision_model runs the same check from Python.

  • The model loads, and its tensors map to our network.
  • Its answers to sample Choice, Noul and Score questions are valid distributions.
  • A request gets the same answer alone and in a batch.
  • An edit to one tensor moves the answers, and restoring the tensor restores them.

Write the program. The backend runs a local /v1/systemone server, and a DSPy program calls it through dspy.LM on DSPy's TypeSafe provider, as it would call TypeSafe's hosted Jev. Give the program's signatures bool and Literal outputs, or Choice, Noul and Score outputs from dspy.experimental.

What the writer gets. A decision model changes the writer's tools:

  • decide passes run the program on practice examples and score it. A decide pass on the validation set returns only aggregate scores.
  • read passes return each option's logit and the decision head's inputs.
  • The writer may recompute the head from those reads and derive edits in closed form. simulate_head=False forbids this.
  • The writer may edit any tensor except the token embedding and the final norm. In our runs, every writer edited the decision head, or d1-2B's last block.

Save the edited model. write_checkpoint copies the model's folder with the edited tensors substituted and checked against their hashes. The copy keeps the base folder's layout, so the model's own code and load_decision_backend both open it.

from dspy_wsp import WeightSpace, load_decision_backend

backend = load_decision_backend("d1", framework="mlx")
optimizer = WeightSpace(metric=metric, writer_lm=dspy.LM("openai/gpt-6-astra", model_type="responses", reasoning_effort="max"),
                        backend=backend, max_passes=25, holdout=300, holdout_key=lambda ex: ex.workflow)
compiled = optimizer.compile(program, trainset=train, valset=test)
compiled.wsp_patch.write_checkpoint(backend.weights_dir, "edited-d1/")

Benchmark scripts. examples/typed_decisions.py runs the Typed Decisions benchmark, and examples/procedural_decisions.py runs three tasks from Procedural Typed Decisions. Both take --model with an alias, repo id or folder. Both use gpt-6-astra as the writer unless you pass --writer sol. They save the edited checkpoint in the run folder's checkpoint/.

Generating LLMs

WeightSpace also edits text-generating LLMs, as the paper did. A pass on an LLM generates many long answers, so we recommend VLLMBackend on a GPU machine. HFBackend and MLXBackend suit small experiments.

A backend holds the student model, swaps edits in and out, and runs every forward pass.

  • VLLMBackend runs vLLM on NVIDIA GPUs for runs at the paper's scale. vLLM schedules every concurrent LM call, so the forward-pass tool keeps a whole pass in flight (concurrency=512). Teacher-forced reads go to an HFBackend on the same GPU, because vLLM cannot return hidden states or MLP activations. Both models hold the same edits. To swap edits, the backend waits for running requests to finish, reloads the checkpoint in place with the edited tensors substituted, and clears vLLM's prefix cache. gpu_memory_utilization (default 0.6) sets vLLM's share of GPU memory; the Hugging Face model uses the rest. batch_invariant=True makes each answer depend only on its prompt and seed, at some cost in speed.
from dspy_wsp import VLLMBackend

backend = VLLMBackend("Qwen/Qwen3.5-0.8B", revision="<commit>", max_tokens=32_768, presence_penalty=1.5,
                      chat_template_kwargs={"enable_thinking": True})
  • HFBackend runs transformers in-process on CUDA, MPS or CPU. It generates one answer at a time, so it suits small models and short answers.
  • MLXBackend runs mlx-lm on Apple silicon. It decodes up to batch_size answers together, and the forward-pass tool runs that many programs at once to fill the batch. Each answer samples from its own seed. It also supports the presence penalty in Qwen's recommended sampling settings. On an M2 Max it generates about 650 tokens per second across a batch of 32 answers from Qwen3.5-0.8B, which is several times faster than HFBackend on the same machine.
from dspy_wsp import MLXBackend

backend = MLXBackend("Qwen/Qwen3.5-0.8B", revision="<commit>", batch_size=64, max_tokens=32_768,
                     presence_penalty=1.5, chat_template_kwargs={"enable_thinking": True})

MLXBackend reads attention probabilities only from softmax-attention blocks. In Qwen3.5 those are blocks 3, 7, 11, 15, 19 and 23; the other blocks use linear attention.

Replicate the paper

The paper's Experiment 8 edited Qwen3.5-0.8B to stop its looping on MATH-500. Two scripts rebuild that setup:

  • examples/math500_qwen_vllm.py runs it on one GPU: 50 passes of up to 512 answers, and a final check on all 500 MATH-500 problems with 4 samples each.
  • examples/math500_qwen_mlx.py runs a smaller version on a Mac. Its --baseline N flag scores the unedited model without a writer.

This path is a bigger step than the decision models, and we have tested it less. A full run at the paper's scale takes many GPU hours and a long writer session. results/2026-11-04-math500-qwen-vllm-1h/ documents a one-hour proof-of-concept run on eight A100 GPUs, with the checks we ran before it. We have not yet repeated the paper's full result. If you run this setup, please share what you find in GitHub issues or on the DSPy Discord.

Differences from the paper

  • The task is your DSPy program and metric, not fixed math benchmarks. Generation passes run the program; the records show every LM call.
  • The paper's writer was gpt-6-astra at maximum reasoning effort, driven through the Codex CLI with every built-in tool switched off. Here the writer is any dspy.LM. The writer matters: on our decision model runs, gpt-6-astra wrote better edits than gpt-6.1-sol on two of three models.
  • For decision models, the writer may recompute the decision head in its kernel. simulate_head=False keeps the paper's rule, and the gain then drops sharply (see Results).
  • activations takes separate layers (hidden-state indices) and mlp_layers (block indices) lists, because one list served both in the paper.
  • fallback_if_worse=True returns the original weights when the final check shows a loss. This is an addition to the paper's protocol, and it is off by default.
  • Patches store changed entries or whole tensors. The paper's release also has a low-rank form, which this version doesn't produce yet.

Limits and roadmap

  • HFBackend generates one sequence at a time in-process. That's fine for small models and short answers, but too slow at the paper's scale. There, one pass produced 512 answers of up to 32k tokens, and a pass took up to 30 minutes on a server. MLXBackend batches on a Mac, but a 64-problem paired pass at 32k tokens still takes one to two hours.
  • A run is long and costly. The paper's 50-pass run took 158 turns and 20 hours. Budget for the writer's tokens accordingly.
  • Editing removes obstacles; it doesn't add abilities. The paper's Gemma 3 270M follow-up barely moved. That model didn't loop, so the writer had no failure mechanism to repair.

Credits and license

Code: MIT.

  • The method and the harness come from Sparks of Weight-Space Programming by Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab.
  • The writer's task prompt is adapted from the Experiment 8 prompt released by Takehi et al. in lm_understand-logs under CC BY 4.0.
  • Haoze "Hector" He's weight editing design for llm-honing is earlier work on editing decision models, including NanoJev, and releasing the edits as checked checkpoints. See his post.
  • Michael Isaac contributed the GEPA population search. It uses Lakshya Agrawal's GEPA library.
@misc{takehi2026weightspace,
  title        = {{Sparks of Weight-Space Programming: LLMs can natively read \& write neural network weights}},
  author       = {Takehi, Rikiya and Teoh, Jayden and Zhang, Alex and Khattab, Omar},
  year         = {2026},
  month        = nov,
  howpublished = {\url{https://rikiyat.github.io/blog/weight-space-programming/}},
  note         = {Blog post}
}

Metadata

Release files for dspy-wsp 0.1.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for dspy-wsp 0.1.2
File Size Uploaded Provenance Actions
dspy_wsp-0.1.2.tar.gz 163.0 kB Without provenance Details for dspy_wsp-0.1.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for dspy-wsp 0.1.2
File Interpreter ABI Platform Actions
dspy_wsp-0.1.2-py3-none-any.whl Python 3 none any Details for dspy_wsp-0.1.2-py3-none-any.whl

Total release size: 306.4 kB

Release files / dspy_wsp-0.1.2.tar.gz

Download URL dspy_wsp-0.1.2.tar.gz
Size 163.0 kB
Tags Source
SHA-256 checksum
How to use checksums
338bb184d5d8d3d04fd3aed97f96b07caca0fdb53ffbf555b19a889d676d709f
BLAKE2b-256 checksum
How to use checksums
b60451845b668ea506fb522d61b4fc106e9997dc51006647c903eeee05a4540e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.4

Release files / dspy_wsp-0.1.2-py3-none-any.whl

Download URL dspy_wsp-0.1.2-py3-none-any.whl
Size 143.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
0f883a39b6f2791572248cc5a2393d6f86868ad7fd65589f8e27c2fed03bb77f
BLAKE2b-256 checksum
How to use checksums
5ff64e62bb99ea8a2f6a87dd133dc3febbf0c394aa1b3721e54fc4d2f6b1511d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.4

Security for dspy-wsp 0.1.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page