Weightsmith

Don't retrain. Reforge.

Weightsmith is a programming language for weight-space programming. You load a network's parameters, inspect and edit them directly without gradient descent, and then verify the result against behaviours you declared up front.

The editing can be done by fixed strategies, by your own code, or by a language model working under a budget.

🔨 Example

Claude gets a broken model, a budget and a contract in this program, and practises on one set of probes while Weightsmith judges the repair on another.

Here is an abridged copy of Claude's report and the verdicts:

editor's account (a hypothesis): The broken task was count, and I've fixed it.
  The cause: two units in layer 0's feed-forward block, neurons 185 and 206, had weights
  about 10 times larger than any other neuron there.
  The fix: I zeroed both neurons completely, their input, gate and output weights.

improve  count_test     verified   improved by +0.1905, at least 0.1
preserve opposite_test  verified   changed by +0.0000; degradation within 0.07
preserve first_test     verified   changed by +0.0312; degradation within 0.02
require  finite         verified   no NaN or infinite parameters
Changed parameters: 384 of 131264 (0.2925%) in 3 tensor(s)

The original model is untouched, since the run writes an exact, reversible patch and a transcript of every exchange. The same repair with a scripted editor runs offline:

💡 Why a language

• Programs state intent (operations, behaviours, requirements) and leave execution to a backend, so the compiler can check them against tensor manifests without loading a tensor library.
• Every effect, from loading and probing to mutating, calling a language model and saving, must be allowed by a policy with budgets; the compiler rejects programs that overstep it, and the runtime stops before a budget runs out.
• Findings say how well supported they are (observed, experimentally_supported, ...), and verdicts are verified, failed, inconclusive or skipped.
• A run works on a copy of the model and emits a content-addressed patch that you can apply or revert.

📦 Installation

Weightsmith needs Python 3.12 or later and installs with uv:

git clone https://github.com/JGalego/weightsmith.git && cd weightsmith
uv sync                      # installs CPU-only PyTorch from the PyTorch index
uv run weightsmith --version

Extras: hf runs Hugging Face models, anthropic and openai enable language-model editors, lsp adds the language server. uv sync installs them all; checking programs needs none of them.

🚀 Quick start

uv run weightsmith fixtures                         # build the synthetic models (under a minute, CPU)
uv run weightsmith check examples/repair_mlp.wsm    # static checks only, no tensors loaded
uv run weightsmith run   examples/repair_mlp.wsm --out out/repair
uv run weightsmith diff  examples/fixtures/tiny_mlp_defective out/repair/model
uv run weightsmith apply examples/fixtures/tiny_mlp_defective out/repair/patch --out out/applied

🧪 Examples

These run on synthetic models with planted defects, built by weightsmith fixtures.

Find and repair

• inspect_transformer.wsm: structural finders on raw Transformer weights catch a NaN, a duplicate tensor and a dead row.
• repair_mlp.wsm: finds the outlying rows and repairs them, then backs each finding with intervention evidence before verifying the result.
• repair_tiny_llama.wsm: finds amplified MLP neurons in a Llama-style model by role; count goes from 0.70 to 1.00.
• prune_mlp.wsm: prunes by magnitude and zeroes a slice while preserving both behaviours.

Merge and transplant

• merge_mlps.wsm: interpolates with a fine-tuned variant and transplants a layer.
• merge_unrelated.wsm: runs the same merge with an unrelated model, and verification fails as it should.
• head_surgery.wsm: zeroes an attention head and transplants a key/value head from a donor.

Language models as editors

• repair_with_editor.wsm: a scripted editor that runs offline and replays exactly with --replay.
• repair_with_claude.wsm: gives the same task to Claude, with only "one task is broken" as a hint, through the Anthropic API.

✨ Replicating Sparks of weight-space programming

examples/wsp/ reruns the experiments from the article with its own networks and prompts and with GPT-6 Astra as the editor. uv run python examples/wsp/fetch.py downloads them from the published logs (CC BY 4.0) and checks that every prompt rebuilds byte for byte.

👀 Language at a glance

The full definition is in docs/semantics.md.

💻 Command line

run exits with 0 when everything is verified, 1 for an invalid program, 2 for a runtime error or budget stop, 3 when a contract item failed and 4 when one was inconclusive.

⚖️ How it compares

• Fine-tuning and LoRA change weights by gradient descent on data. Weightsmith changes them directly and checks the result against a contract.
• mergekit merges checkpoints with methods such as SLERP and TIES. Weightsmith's compose also merges, but it does so under a policy and fails verification when a merge breaks a declared behaviour.
• TransformerLens and similar libraries study activations and circuits. Weightsmith's finders are cruder weight statistics, but its edits are verified and shipped as patches.
• weightspace explores language-model-driven weight editing in Python. Weightsmith doesn't depend on it.

⚠️ Known limitations

• Runnable models are sequential arch.json networks or Hugging Face causal language models. Roles exist for the GPT-2 and Llama families; other models are addressed by parameter name.
• Finders locate statistical anomalies in weights, not capabilities, circuits or features. Asking for "the reasoning circuit" is a compile error.
• Behaviours are probe suites, Kinetix rollouts, or MATH-500 and GSM8K on Hugging Face Jobs.
• The python tool is not a security sandbox, because editor code runs as you, though without credentials and within a time limit.
• Verdicts hold only for the declared probes and come without confidence intervals.
• Patches store replacement values, so a dense patch is about twice the size of the tensors it changes.
• Merging unrelated networks fails without permutation alignment, which Weightsmith doesn't do.

See docs/vision.md for where the language is headed and CONTRIBUTING.md to work on it.

📜 Licence

Apache-2.0. See LICENSE.