---

Armsmith profiles an AI repo on Arm, diagnoses why it is slow on aarch64 with a 13-rule anti-pattern pack, drafts fixes, and renders a PR in which every fix has passed a reproduce-benchmark gate — median-of-N, MAD noise bands, output-hash equality. The LLM plans; the silicon decides. In-band deltas are reported as no change, never as wins, and dropped fixes are reported, never hidden.

(PR rendering is dry-run today: it prints exactly what would ship and makes no network call. Posting is on the roadmap and marked in code rather than implied here.)

Scan any repo for aarch64 anti-patterns without installing anything:

uvx armsmith scan .          # or: pipx run armsmith scan .   ·   pip install armsmith

Then record a real bundle from your own machine and run the full 13-rule diagnosis on it — no fixtures of ours involved:

armsmith record . --out ./armsmith-bundle --python .venv/bin/python
armsmith diagnose --replay ./armsmith-bundle

To reproduce the full gate — baseline, 13-rule scan, keep/drop verdicts, signed report — clone and run the replay bundle:

git clone https://github.com/edycutjong/armsmith && cd armsmith
python3 -m venv .venv && source .venv/bin/activate && pip install -e '.[dev]'
python -m pytest -q                                       # 410 passing, offline
armsmith diagnose --replay fixtures/replays/scenario_ragserve   # 4 kept · 2 dropped

No Arm hardware, no network, no API key. Full walkthrough in Getting Started · every gate at once with make all · extending the rule pack: CONTRIBUTING.

💡 The Problem & Solution

The Problem

Moving an AI workload from x86 to Arm is supposed to be a cost win, and usually it is — but when a repo runs slowly on aarch64, nobody can tell you why. The failure modes are boring and invisible: an amd64-pinned base image quietly running under QEMU, NumPy on reference BLAS, a GGUF quantization that misses the ISA's repack path, a build with no -mcpu so the dot-product unit never gets used.

The deeper problem is what happens next. An LLM is very willing to tell you it made your code 30% faster. Benchmarks are noisy, "improvements" inside the noise band get reported as wins, and a fix that changes your output is still a fix if nobody checked. Performance claims are the easiest thing in software to fake, including by accident.

The Solution

Armsmith is an agent that is not allowed to claim its own results. It scans, it drafts fixes, and then every fix has to survive a reproduce gate that the agent does not control:

armsmith diagnose ./repo
   ├─ host fingerprint (lscpu → dotprod/i8mm/SVE/SVE2/BF16/SME routing)
   ├─ 13-rule scan (static AST/Dockerfile/CI + recorded runtime probes)
   ├─ planner orders fixes (deterministic fallback; Claude tool-use = TODO(S1))
   ├─ REPRODUCE GATE  ── keep only: outside noise band AND output-hash equal
   └─ signed report (ed25519 + sha256) ─→ PR body with evidence table (dry-run)

A fix is kept only if it beats the measured noise band and produces byte-identical output. Fixes that fail are reported with reasons, never dropped silently. The report embeds the raw samples, so armsmith verify recomputes every statistic and every verdict independently — you never have to trust the number that was printed at you.

What is measured, and what is replayed

Status: hardware-free core + one live Arm leg. 410 pytest tests, all green, at 100% line coverage. The rule pack, the planner and the diagnose loop run against replay bundles, and a bundle is one of two things, always labeled:

bundle	manifest	where it comes from
the fixtures in this repo	"synthetic": true	hand-authored shapes for offline tests — measured on nothing
what armsmith record writes	"synthetic": false	observed on your host, or copied verbatim from your own instrument output

Every loader refuses a bundle that declares neither. Provenance and transport are tracked separately on purpose: a recorded bundle is replayed but entirely real, so its report carries "mode": "replay" with "synthetic": false, and stamping it synthetic would understate a genuine measurement exactly as badly as the reverse would overstate one.

One further path produces hardware numbers in-process rather than from a bundle, armsmith bench-live (see below) — "mode": "live", "synthetic": false. Every number in this repo is one of these, and says which. The remaining live instruments (perf/PMU, Performix, llama-bench, hyperfine, cosign-in-CI, the Claude planner loop, PR posting) land at S1 and are marked TODO(S1) in code.

Recording a bundle for your own repo

armsmith diagnose needs a bundle. armsmith record writes one from the machine you run it on, so the probe rules work on your code rather than only on our fixtures:

armsmith record . --out ./armsmith-bundle --python .venv/bin/python

It captures what the host can honestly answer — lscpu, transparent-hugepage state, and the BLAS that numpy.show_config() reports for the interpreter you point --python at (that flag matters: R3 is a claim about the venv that serves your model, and armsmith's own does not even depend on numpy). For the probes that only exist as output from a real instrument, hand it the artifact you already have and it is copied in unmodified:

armsmith record . --out ./b \
  --build-log build.log      # → R2      --pip-log pip-install.log   # → R8
  --cmake-cache CMakeCache.txt  # → R10   --gguf model.gguf           # → R5
  --perf perf.txt            # → R9      --ort-session session.json  # → R7
  --llama-bench lb.json --hyperfine hf.json   # → R13 (needs both)

Three rules the honesty contract will not let it fill in:

• env and proc_maps are never captured, so R6 never runs from a recorded bundle and R11 stays half-fed. A bundle is something you publish; an environment block carries CI tokens and a maps dump carries host paths. This is refused in code, not by convention, and a test asserts the files are absent.
• Anything not observed is omitted, not guessed. The rules that needed it report skipped with the probe named, and record prints exactly which rules your bundle can and cannot answer before you run diagnose.

🏗️ Architecture & Tech Stack

The 13-Rule Pack

Every rule ships as a YAML descriptor (src/armsmith/rules/packs/) with a detector, a deterministic fix generator, a citation URL, and positive/negative fixtures. Expected-gain ranges are estimates from the citations used only for planning order — results come exclusively from the gate.

id	anti-pattern	detector
R1	amd64-pinned image → QEMU emulation	static (Dockerfile/compose)
R2	native build without -mcpu/-march	probe (build log + lscpu)
R3	NumPy on reference BLAS	probe (numpy.show_config())
R4	silent float64 coercion	static (Python AST)
R5	GGUF quant mismatched to ISA repack path	probe (GGUF header + lscpu)
R6	threads × workers > vCPUs	probe (recorded env)
R7	ONNX Runtime session defaults	probe (SessionOptions record)
R8	pip sdist fallback for perf-critical wheels	probe (pip log)
R9	tokenizer/preprocess memcpy storm	probe (perf report)
R10	llama.cpp built without KleidiAI	probe (CMake cache)
R11	THP/allocator untuned for big-model RSS	probe (sysfs + maps)
R12	CI publishes amd64-only images	static (workflow YAML)
R13	serving overhead dominates kernel time	probe (llama-bench × hyperfine)

Precision on a repo we've never seen

A linter that cries wolf gets uninstalled, so R4 is measured against a real target rather than its own fixtures. Pointed at a fresh clone of huggingface/text-generation-inference:

	flagged	false positives
naive "no dtype= → float64"	5	4
shipped R4	1	0 proven wrong

The four that vanished were integer permutation arrays in the Marlin GPTQ path (layers/marlin/gptq.py:461,463, layers/marlin/util.py:134,136). numpy infers dtype from the data — np.array([0, 2, 4]) is int64, and np.full(n, 0) is int64 too — so those were never float64, and the fix R4 would have proposed (pin dtype=np.float32) would have silently turned an index array into floats. R4 now reasons per constructor: zeros/ones/empty/linspace are float64 whatever you pass them and are always reported; array/full are reported only when the payload isn't a provable integer literal.

The one surviving hit, utils/segments.py:17, is np.array(adapter_indices) — a non-literal argument. Armsmith cannot prove that one statically, so it reports it: under-reporting a real float64 coercion on an inference path costs more than one honest question. That is the deliberate bias, and test_r4_does_not_flag_an_integer_permutation_array pins the regression.

R13 is the two-instrument triangulation rule: llama-bench timings exclude tokenization + sampling, so Armsmith reconstructs kernel time from llama-bench samples and compares it with hyperfine end-to-end wall time — >15% divergence means the pipeline, not the kernels, is the bottleneck. Both instruments' self-reported stats are cross-checked against their own raw samples first; disagreement makes the rule refuse to diagnose.

Tech Stack

layer	choice	why
CLI	Typer + Rich	subcommand surface + the evidence tables judges actually read
Statistics	pure stdlib, zero deps (armsmith.benchstats)	the math that accepts or rejects a claim must be auditable at a glance
Rule descriptors	PyYAML	a 14th rule is one YAML file + one detector, no core changes
Report signing	cryptography (ed25519)	tamper-evident reports; verify re-derives every statistic
Report schema	jsonschema (draft 2020-12)	public contract, CI-validated — build your own viewer against it
Live Arm bench	GCC + binutils objdump on aarch64	compile A/B from one source, then count SDOT in the disassembly
CI	GitHub Actions — ubuntu-24.04-arm · ubuntu-22.04-arm · ubuntu-latest × Py 3.11/3.12	native arm64 legs, free, no hardware to rent
Quality	pytest + pytest-cov · ruff · mypy · CodeQL · TruffleHog	410 tests, 100% line coverage

🏆 Arm Integration (Cloud AI Track)

Armsmith is not an app that happens to run on Arm — Arm is the subject matter. The Arm-specific surfaces it actually uses:

• Native arm64 CI runners — ubuntu-24.04-arm / ubuntu-22.04-arm, four green jobs per push, plus a dedicated live-bench job that takes a real measurement on Neoverse-N2.
• ISA feature routing — lscpu flags parsed into dotprod / i8mm / SVE / SVE2 / BF16 / SME, and rules gate their advice on what the target CPU actually has.
• -mcpu / -march flag matrix (R2) — the rule with a live, measured A/B behind it (below).
• Arm dot-product & int8-matmul ISA — armsmith.witness counts SDOT/UDOT/SMMLA/USMMLA in real disassembly, so a kernel claim is proven at the instruction level.
• GGUF + KleidiAI paths (R5, R10) — a real GGUF header parser checks whether the chosen quantization can reach the ISA's repack path; R10 checks whether llama.cpp was built with KleidiAI.
• Arm Learning Path citations — every rule carries a real upstream URL; armsmith rules export renders them as 13 migration cards.

Armsmith is arch-clean Python and installs identically on aarch64 (AWS Graviton c7g/c8g, Ampere, Axion, or a GitHub ubuntu-24.04-arm runner):

sudo apt-get update && sudo apt-get install -y python3-venv  # (perf, hyperfine, llama.cpp = live-mode, S1)
git clone https://github.com/edycutjong/armsmith && cd armsmith
python3 -m venv .venv && source .venv/bin/activate
pip install -e '.[dev]'
python -m pytest -q                                                    # 410 passing on aarch64
armsmith doctor --offline --replay fixtures/replays/scenario_ragserve  # shows dotprod/i8mm/SVE routing
armsmith diagnose --replay fixtures/replays/scenario_ragserve          # identical loop, native arm64
armsmith bench-live --require-witness                                  # the real measurement, on your silicon

The offline suite proves the package is arch-clean on real Arm silicon, and bench-live takes a genuine measurement on it. Full live capture — driving perf/hyperfine/llama-bench against your workload and recording a real before/after — is the S1 path (armsmith diagnose <repo> --target ssh://…, LiveProbe over SSH), marked TODO(S1) in code; Armsmith never fabricates a hardware number.

The drop-in CI twin runs the same gate on an Arm runner:

# .github/workflows/perf-gate.yml
jobs:
  arm-perf-gate:
    runs-on: ubuntu-24.04-arm  # free native-arm64 hosted runner
    steps:
      - uses: actions/checkout@v4
      - uses: edycutjong/armsmith@v1  # composite action — see action.yml
        with:
          replay: fixtures/replays/scenario_ragserve

📊 Engineering Rigor

_{Built with}

_{Quality gates}

metric	value
Tests	410 passing, 100% line coverage
CI jobs per push	8 — incl. 5 native arm64 (4 test legs + 1 live bench)
Rules	13, each with a citation + positive/negative fixtures
Live Arm speedup (measured)	7.4× / −86.5%, outside a ±0.24% noise band
ISA witness	SDOT 0 → 1 in the hot symbol, from real disassembly
Report integrity	ed25519 signature + sha256 content hash + full statistic recompute
Security	CodeQL 0 alerts · TruffleHog (full history) · Dependabot · pip-audit

Measured on Real Arm Silicon

Everything else here proves the loop is honest. This is where it stops being a replay.

armsmith bench-live compiles bench/int8_dot.c — an int8×int8→int32 dot product, the quantized-inference inner loop in miniature — twice from one source, differing only in the -march flag that rule R2 exists to flag. It then measures both builds on the machine it is running on. It refuses to run on anything that is not aarch64.

Latest run, on a GitHub-hosted ubuntu-24.04-arm runner (ci.yml → job Live Arm reproduce gate), host Neoverse-N2, gcc 13.3.0:

	baseline = -O3 -march=armv8-a	fix_R2 = -O3 -march=armv8.2-a+dotprod
SDOT in dot_i8	0	1
median kernel_s	0.059975 s	0.008123 s
p95	0.060079 s	0.008200 s
Δ median	—	−0.051852 s (−86.5%, a 7.4× speedup)
noise band (k=3)	—	±0.000144 s
output hash equal	—	✅ identical
gate verdict	—	keep

Read that first row before the timings. ARMv8.0 has no dot-product instruction, so the baseline cannot contain one; enabling the ISA level lets GCC's vectorizer emit SDOT and the whole accumulate collapses into it. The stopwatch says 7.4×, but the disassembly says why, and a disassembly is not a benchmark you can argue with.

The measurement is not special-cased anywhere: the samples go through the same benchstats → gate → signed-report path as every replay bundle, under the same refuse-to-claim-inside-the-noise-band rule. A run that fails to beat its own noise is reported no_change and dropped — and that outcome is a success for the tool, not a bug.

armsmith bench-live --require-witness       # on any aarch64 box; writes a signed report-live.json
armsmith verify report-live.json            # hash + ed25519 + schema + recompute-from-samples

CI runs exactly those two commands on every push and uploads the signed report as a build artifact, so the numbers above are re-derivable by anyone with the repo and an Arm runner — including you.

The Trust Chain

1. Statistics engine (armsmith.benchstats) — median-of-N, MAD noise bands (k·√(smad_a²+smad_b²), k=3), p50/p95 by documented linear interpolation, ABAB interleave planning, and a hard refuse-to-claim-inside-band rule.
2. Reproduce gate (armsmith.gate) — drop on hash mismatch, drop on any out-of-band regression, drop when nothing clears the band. Reasons are machine-readable and shipped.
3. Tamper-evident reports (armsmith.report) — raw samples embedded next to every claimed statistic; canonical-JSON sha256 content addressing; ed25519 signature; armsmith verify recomputes every statistic and gate verdict from the embedded samples. Editing a number without re-running the math is detectable. Schema: schema/report.schema.json.
4. ISA witness (armsmith.witness) — counts SDOT/UDOT/SMMLA/USMMLA in disassembly before/after: wall-clock can be argued with; emitted instructions cannot.
5. PR evidence (armsmith.evidence, armsmith.ghpr) — the | metric | before | after | Δ | noise band | PMU Δ | table, the drop log, and the judge-facing cosign verify-blob command line. PR module is dry-run only here: it renders exactly what would be posted and never touches the network.

Honesty Notes

• Replay bundles are synthetic shapes, generated by scripts/make_fixtures.py and labeled in every manifest.json; loaders refuse unlabeled measurement data, reports carry mode: "replay" + synthetic: true, and every rendered artifact shows a replay banner.
• armsmith doctor refuses to run without --offline + a recorded fixture: this development machine is never fingerprinted as if it were a target.
• The planner cannot claim results; only the gate can, and armsmith verify re-checks the gate.
• Live and replay never mix. bench-live reports carry mode: "live" + synthetic: false; every other report carries mode: "replay" + synthetic: true. bench-live raises rather than run on a non-aarch64 host, so there is no code path that yields an Arm number off Arm silicon — and a unit test asserts exactly that.
• LiveProbe refuses the env and proc_maps probes even though it could trivially serve them: a report is a published artifact, and a CI environment block contains tokens. Every probe it cannot answer honestly raises instead of guessing.
• The live 7.4× is one microbenchmark on one runner, not a claim about your model. It is there to prove the gate works on real silicon; the honest way to get your number is to run it on yours.
• The benchstats module is shared with the Assayer project (declared in both repos).

🚀 Getting Started

Prerequisites

• Python 3.11 or 3.12
• No Arm hardware and no network beyond pip — the whole judge surface runs on an x86 laptop
• (optional) an aarch64 box + gcc/objdump if you want to take the live measurement yourself

Installation

To use it — armsmith on PyPI, no clone:

uvx armsmith scan .        # zero-install, one command
pipx install armsmith      # or keep it on your PATH
pip install armsmith       # or into a venv you manage

To reproduce the gate or hack on it — the replay bundles and the test suite live in the repo, so this path needs the clone:

git clone https://github.com/edycutjong/armsmith && cd armsmith
python3 -m venv .venv && source .venv/bin/activate
pip install -e '.[dev]'

Judge Quickstart — Zero Hardware (~2 min)

This is the primary "runnable by a judge" surface, because most judges have no Graviton box. All commands exit 0; the tamper step at the end goes red on purpose.

python -m pytest -q                                            # 410 passing, fully offline
armsmith scan fixtures/replays/scenario_ragserve               # static R1/R4/R12 on a real dir, zero hardware
armsmith diagnose --replay fixtures/replays/scenario_ragserve  # full reproduce gate (4 kept, 2 dropped)
armsmith witness fixtures/witness/objdump_before.txt fixtures/witness/objdump_after.txt  # ISA proof: 0→4 dotprod
armsmith verify fixtures/replays/scenario_ragserve/report.json # -> VERIFY OK (recomputes every stat)
armsmith ci --replay fixtures/replays/scenario_ragserve        # -> CI GATE PASSED (exit-code CI twin)
python scripts/verify_offline.py                               # -> ALL CHECKS PASSED — honest & offline

The 20-second trust proof (whole pitch, zero hardware): open fixtures/replays/scenario_ragserve/report.json, change one digit in any samples array, re-run armsmith verify …/report.json → red VERIFY FAILED. You never trust the printed number; the arithmetic is independently re-derivable and tamper-evident.

Other surfaces: armsmith rules list · armsmith rules explain R13 (fix + Arm Learning Path) · armsmith rules export --format md (writes the 13 migration-template cards to docs/migration-templates/) · armsmith doctor --offline --replay fixtures/replays/scenario_ragserve (host/ISA fingerprint) · armsmith pr fixtures/replays/scenario_ragserve/report.json (renders the bot PR — dry-run).

🧪 Testing & CI

The replay harness is hardware-free and runs in under a second locally; the live Arm leg needs aarch64 and refuses to run anywhere else:

.venv/bin/pip install -e '.[dev]'

.venv/bin/python -m pytest -q               # 410 tests, 100% line coverage
.venv/bin/ruff check .                      # lint gate (clean)
.venv/bin/mypy src                          # types — advisory, not a gate
.venv/bin/python scripts/verify_offline.py  # scan → gate → sign → verify, end-to-end

.venv/bin/armsmith bench-live --require-witness   # aarch64 only — the real measurement

CI (.github/workflows/ci.yml) runs that exact suite on a native-arm64 + x86 matrix — ubuntu-24.04-arm, ubuntu-22.04-arm, and ubuntu-latest × Python 3.11 / 3.12 — plus the offline end-to-end loop and a JSON-Schema check on schema/report.schema.json. Because every test is replay/fixture-based, the arm64 legs need zero Arm-specific setup and prove the package is arch-clean. A separate live-bench job then runs armsmith bench-live --require-witness on ubuntu-24.04-arm, verifies the signed report it produces, and uploads it as an artifact — that job is where the numbers in Measured on Real Arm Silicon come from.

layer	tool	status
unit + replay suite	pytest (410 tests, 100% cov)	✅ green, offline
lint	ruff	✅ gate
types	mypy	✅ advisory (continue-on-error)
end-to-end loop	verify_offline.py	✅ scan → gate → sign → verify
report schema	jsonschema (draft 2020-12)	✅ validated in CI
SAST	CodeQL (language: python)	✅ codeql.yml
secret scanning	TruffleHog (--only-verified, full history)	✅ CI security gate
dependency updates	Dependabot (pip + actions)	✅ dependabot.yml
dependency audit	pip-audit	✅ advisory
live Arm bench	armsmith bench-live on ubuntu-24.04-arm	✅ real measurement + ISA witness, signed & verified in CI
live-hardware instruments	hyperfine / llama-bench / Performix / cosign	TODO(S1) — not wired yet

Everything above is real today. The live Arm row is a genuine measurement taken on a native arm64 runner; the last row is the honestly-deferred remainder — no CI job claims a measurement it did not take.

📁 Project Structure

src/armsmith/          benchstats · probes · fingerprint · gguf · rules/ (packs + 13 detectors)
                       gate · report · keys · evidence · witness · ghpr · planner/ · diagnose · cli
                       livebench (the live Arm A/B: compile → witness → measure → gate)
bench/int8_dot.c       the live workload — one source, compiled two ways (rule R2)
schema/                report.schema.json (draft 2020-12, CI-validated)
fixtures/              hosts/ · rules/rXX_{pos,neg}/ · replays/scenario_ragserve/ · witness/
scripts/               make_fixtures.py (fixture provenance) · verify_offline.py
tests/                 410 tests (goldens, pos/neg per rule, gate, signing, CLI, e2e, live bench)
site/                  landing page + pitch deck (deployed straight from this repo)
docs/assets/           brand + hero assets (see ASSETS pipeline)
docs/migration-templates/  13 x86→Arm migration cards (armsmith rules export)
action.yml             composite GitHub Action — drop-in arm64 perf-regression gate

🧩 Reuse & Extend

Every artifact is reusable standalone of the CLI — this is the "could it be taken further / reused" DX clause and the rubric's reusable-artifacts Impact:

• 13 x86→Arm migration templates — armsmith rules export --format md renders one card per rule (anti-pattern · fix · expected gain · upstream citation · Arm Learning Path) into docs/migration-templates/. Reusable on any repo.
• Add a 14th rule without touching the engine — one YAML descriptor into src/armsmith/rules/packs/, one detector, one import line; the loader validates and wires it in (see CONTRIBUTING for the exact detect() signature).
• Public signed-report schema — schema/report.schema.json (draft 2020-12, CI-validated). Build your own viewer/CI gate against it.
• Importable methodology modules — from armsmith.benchstats import compare (median-of-N/MAD/ noise-band), armsmith.gate, armsmith.report, armsmith.witness — no CLI required.
• Drop-in Arm CI gate — uses: edycutjong/armsmith@v1 on runs-on: ubuntu-24.04-arm (see action.yml); the Marketplace listing is publish-pending, never claimed as live.
• Installable in one command — armsmith on PyPI: uvx armsmith scan . runs the aarch64 anti-pattern scan on any repo with nothing to clone and nothing to configure. Published from CI by Trusted Publishing (OIDC, no API token in the repo), with sdist + wheel attached to every GitHub Release.

🗺️ Roadmap

• 13-rule pack with citations, fixtures, and deterministic fix generators
• Reproduce gate — noise bands, output-hash equality, machine-readable drop reasons
• Tamper-evident signed reports + verify statistic recompute
• ISA witness (SDOT/UDOT/SMMLA/USMMLA) driven against real binaries
• Native arm64 CI + live measured A/B on Neoverse-N2
• armsmith record — live capture on the local host writes a real, replayable bundle
• LiveProbe over ssh:// (record a bundle for a remote Arm box from your laptop)
• Live instruments: perf/PMU, hyperfine, llama-bench, Arm Performix CLI ingestion
• Claude planner tool-use loop (contract already pinned in planner/interface.py)
• Real PR posting (armsmith pr is dry-run only today) + cosign keyless attestation in CI
• Arm MCP Server handshake → query_arm_mcp container-validation cross-check

📽️ Demo Materials

• Demo video (3 min): youtu.be/vq15rK1iCww — the reproduce gate dropping two of its own fixes on camera, the ISA witness, the tamper test, and the arm64 CI run. Scenes drawn from the replay bundle carry a [replay] badge on screen throughout.
• Live site: armsmith.edycu.dev — deployed straight from site/ in this repo, so the page you see is the source you can read.
• Pitch deck: armsmith.edycu.dev/deck.html
• Signed live report: downloadable from any CI run as the armsmith-live-report-arm64 artifact.

📄 License

MIT — see LICENSE.

🙏 Acknowledgments

• Arm Learning Paths and upstream Arm/llama.cpp/ONNX Runtime documentation — every rule in the pack cites a real source rather than folk wisdom.
• GitHub arm64 hosted runners, which made a genuine Neoverse-N2 measurement possible with no hardware to rent.
• The benchstats module is shared with the Assayer project (declared in both repos).