cibuildmp

Build MicroPython native C extensions for every target they support, from one declarative config -- on CI and on your own machine. cibuildwheel, for MicroPython.

Covers natmod (dynamically loadable native .mpy modules, built against py/dynruntime.mk) and usermod (USER_C_MODULES, compiled straight into a port's own firmware build) C extensions.

This repository supersedes ballistics-lab/micropython-native-ci. Every composite action that lived there now lives here, unchanged in behaviour but on new paths -- see Migrating below. The old repository is deprecated and will be archived once its consumers have repinned; nothing new lands there.

Two layers

cibuildmp, the CLI. One cibuildmp.toml in a module's own repo describes its whole target matrix. The tool resolves it into build identifiers, fetches MicroPython, builds mpy-cross, provisions each target's cross toolchain, and runs the build -- the same way locally as on a runner. This is the direction the project is going; see docs/BACKLOG.md for the design decisions and what is implemented so far.

$ cibuildmp --dry-run
cibuildmp: 10 target(s) against MicroPython v1.28.0
  [ 1/10] mpy6.3-natmod-x86            CROSS=(host)                 make -C natmod ARCH=x86 dist
  [ 2/10] mpy6.3-natmod-x64            CROSS=(host)                 make -C natmod ARCH=x64 dist
  ...

Drop --dry-run and it builds for real: each target lands in its own output-dir/<identifier>/ directory (mpyhouse/mpy6.3-natmod-x64/, …) alongside a package.json once version is set — see examples/template and examples/wasm2mpy (native source is WebAssembly, compiled through wasm2c — the natmod contract doesn't care what produced the C).

The composite actions. The original building blocks, still the way every usermod target is built today. cibuildmp is absorbing them one at a time (natmod first); until it does, they remain fully supported.

Why this exists

MicroPython itself already defines two standard, unrelated build mechanisms for a native C extension:

• natmod -- natmod/Makefile includes MicroPython's own py/dynruntime.mk and is parameterised by ARCH=. It produces a runtime-loadable .mpy per target architecture. See MicroPython's own examples/natmod/.
• usermod -- usermod/micropython.cmake + usermod/micropython.mk, pointed at via USER_C_MODULES= on a port's own build. Compiled into the firmware image itself. See MicroPython's docs/develop/cmodules.rst.

ballistics-lab/micropython-bclibc, o-murphy/micropython-wasm3 and o-murphy/a7p each already follow that same natmod/ + usermod/ layout -- that part was never the problem. What diverged was the CI around it: each repo's GitHub Actions workflow was hand-copied into the next and then evolved independently, so the same ~10-architecture build matrix, the same toolchain-install steps and the same real-ARM-Linux test trick ended up as three separate, slowly drifting copies (different actions/checkout versions, different path filters, one bug fixed in one repo and not the other two).

This repo is the shared home for the parts that are genuinely identical across all three -- so a fix or an improvement lands once, and every consuming repo picks it up deliberately by bumping the tag it's pinned to, instead of by hand-patching three YAML files that have already started to disagree with each other.

The composite actions solved that for the steps. They could not solve it for everything around them: the arch matrix itself still had to be spelled out in each repo, a composite action structurally cannot choose its own runs-on:, artifact globs stayed caller-side, and none of it could be run on a laptop. That is what cibuildmp is for.

Migrating

Action paths change repo, nothing else. Behaviour, inputs and outputs are identical:

- uses: ballistics-lab/micropython-native-ci/.github/actions/build-natmod@v0.2.0
+ uses: ballistics-lab/cibuildmp/.github/actions/build-natmod@v0.3.0

Pin a tag, as before -- not @main, not a commit SHA.

What's here

Composite actions (.github/actions/)

Every table below is the action's complete input surface -- if it isn't listed here, the action doesn't accept it. MPY_DIR in a "Requires" line means fetch-micropython or clone-micropython (this repo's own) must have already run in the same job; a composite action step can't set an env var that steps before it will see, only ones after.

fetch-micropython

Downloads and extracts a MicroPython release tarball, exports MPY_DIR. Use for a plain natmod build or a unix-port build; the tarball already vendors every port's lib/ submodules, so no submodule init is needed. Not usable on a Windows runner outside MSYS2 -- it shells out to wget, which plain Git Bash doesn't have (see build-usermod-windows below for why the Windows actions never call it either).

No outputs; exports MPY_DIR to $GITHUB_ENV as a side effect.

clone-micropython

Shallow git-clones a MicroPython release branch instead of fetching a tarball, with a chosen set of submodules initialised, and exports MPY_DIR. Use this when the build needs a submodule the release tarball doesn't vendor (lib/pico-sdk for an rp2 firmware build, for instance) -- or, as a7p and now bclibc/wasm3's own webassembly/rp2040/windows-adjacent jobs use it, any time the caller needs MPY_DIR set without dragging in fetch-micropython's wget dependency.

No outputs; exports MPY_DIR to $GITHUB_ENV as a side effect.

build-natmod

Installs whatever toolchain a single dynruntime.mk ARCH needs (plain apt package, the xtensa-lx106 tarball, or esp-idf -- dispatched per arch, matching dynruntime.mk's own CROSS choices), builds mpy-cross, then runs make ARCH=<arch> dist in the natmod directory.

Requires: MPY_DIR (see above) and the calling repo already checked out, submodules included if the natmod Makefile needs any.

No outputs.

build-usermod-unix

The unix-port cross-compile matrix for a USER_C_MODULES usermod: x64, x86 (32-bit), aarch64, armhf, or mipsel. Installs the arch's toolchain (apt package, qemu-user-static for the emulated ones, a from-source libffi for the statically-linked ones), builds mpy-cross, then runs the port build.

Requires: MPY_DIR and checkout, same as build-natmod. The caller's own matrix still has to choose runs-on: per arch (ubuntu-24.04-arm for aarch64/armhf -- both execute natively there, not under an emulator; ubuntu-latest for the rest, mipsel included -- it stays under qemu-user-static, since GitHub has no mips runner) -- a composite action can't pick its own runner.

build-usermod-windows

The ports/windows half of the same usermod build, run inside an MSYS2 shell (every step is shell: msys2 {0}): builds mpy-cross then the port itself, including the four CLANGARM64-only overrides every consuming repo's Windows row needed (LDFLAGS_ARCH/COMPILER_TARGET because CLANGARM64 links via clang+lld rather than GNU ld/gcc, STRIP=""/SIZE="true" because that toolchain ships neither binary).

Deliberately narrower than build-usermod-unix: fetching MicroPython and setting up MSYS2 both stay the caller's own job. Requires:

• MPY_DIR, exported to a POSIX-style path (no backslashes -- MSYS2 bash's own escape character eats them on any unquoted command line built from a native D:\a\... value, a real failure documented in every caller this was extracted from). Neither fetch-micropython nor clone-micropython is safe to use for this on a Windows runner as-is: the former shells out to wget, which plain Git Bash doesn't have, and the latter's own $GITHUB_WORKSPACE-derived MPY_DIR is the native backslash form. Every caller this was extracted from sets MPY_DIR itself with an inline curl+$(pwd) step instead.
• msys2/setup-msys2 already run in the calling job, with the target msystem. This action's own steps can't do it for themselves -- they're composite-action steps, so their shell: is fixed at msys2 {0} regardless of what ran before them in the calling job, and that shell wrapper only exists on PATH once setup-msys2 has put it there.

Every path input defaults to a $(pwd)-relative value, never an absolute one, for the same backslash reason MPY_DIR has to be POSIX-style.

build-usermod-webassembly

The ports/webassembly usermod build: installs emsdk, builds mpy-cross, then runs the port build under it, producing a micropython.mjs + micropython.wasm pair.

Requires: MPY_DIR and checkout, same as build-usermod-unix. Combining FROZEN_MANIFEST with the port's own default (variants/<variant>/manifest.py) is deliberately not done here -- every one of usermod/manifest.py's own try/except tricks in the three repos this was extracted from only ever probed $(PORT_DIR)/boards/manifest.py, which doesn't exist for this port (it has variants/, not boards/) -- so passing that file straight through as FROZEN_MANIFEST silently dropped the variant's own default (for pyscript: asyncio, backed by a custom JS-runtime scheduler, plus a require() list of 24 stdlib/utility modules). That was a real gap, not a stylistic one -- the .mjs/.wasm these jobs upload is a build artifact real code can import against, not just a test fixture, and tests/-only coverage never exercises it. Every consuming repo now writes its own combined manifest first and passes that as frozen_manifest.

build-usermod-rp2040

The ports/rp2 usermod build: installs the arm-none-eabi + CMake toolchain, builds mpy-cross, then runs the port build under it, producing a firmware.uf2.

Requires: MPY_DIR and checkout, same as build-usermod-unix. Plain fetch-micropython is sufficient here, no clone-micropython + submodules needed: ports/rp2/CMakeLists.txt redirects PICO_TINYUSB_PATH/PICO_LWIP_PATH/PICO_BTSTACK_PATH/ PICO_CYW43_DRIVER_PATH at ${MICROPY_DIR}/lib/<name> -- MicroPython's own top-level submodules, which the release tarball already vendors -- rather than at pico-sdk's own nested vendored copies, so pico-sdk's internal submodule tree is never actually touched by this build.

build-usermod-armv7m

The ports/qemu usermod build: installs the arm-none-eabi toolchain, builds mpy-cross, then runs the port build under it, producing a firmware.elf.

QEMU itself is deliberately not installed here -- it's a runtime emulator for testing the resulting firmware.elf, not a build dependency, same split build-usermod-rp2040 uses for the rp2040py emulator. Install qemu-system-arm (and whatever your own test harness needs, e.g. pyserial) as a caller-side step, alongside your own run_qemu.py-equivalent.

Requires: MPY_DIR and checkout, same as build-usermod-unix.

build-usermod-esp32

The ports/esp32 usermod build: installs ESP-IDF, builds mpy-cross, then runs the port build under it, producing micropython.bin/firmware.bin. Dumps IDF's own build logs and re-runs ninja -v on failure -- idf.py's own console output swallows the actual compiler diagnostic on a failing build, printing only "ninja failed with exit code 1".

No caching yet -- every consumer's original recipe had none either, so this preserves behavior exactly rather than mixing an extraction with a new capability. A real follow-up, not forgotten: ESP-IDF's own --recursive clone is the heaviest single step across every action in this repo.

No build_dir input, unlike the sibling actions: a real CI failure showed that passing BUILD= explicitly on the make command line -- regardless of the value, even the port's own default -- makes esp32's internal CMake-driven mpy-cross sub-build (a separate copy from the top-level one this action already pre-builds) pick up FROZEN_MANIFEST through MAKEFLAGS and fail with undefined reference to mp_qstr_frozen_const_pool. The port's own build-$(BOARD) default is always used instead.

Requires: MPY_DIR and checkout, same as build-usermod-unix.

Usage example

jobs:
  build:
    strategy:
      fail-fast: false
      matrix:
        arch: [x64, x86, armv6m, armv7m, armv7emsp, armv7emdp, rv32imc, rv64imc, xtensa, xtensawin]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
        with:
          submodules: recursive

      - uses: ballistics-lab/cibuildmp/.github/actions/fetch-micropython@v0.3.0
        with:
          mpy_tag: v1.28.0

      - uses: ballistics-lab/cibuildmp/.github/actions/build-natmod@v0.3.0
        with:
          arch: ${{ matrix.arch }}
          # natmod_dir: natmod              # default; a7p passes micropython/natmod
          # pre_build_command: make fetch-nanopb   # a7p-only, runs before `make dist`

      - uses: actions/upload-artifact@v7
        with:
          name: my-module-${{ matrix.arch }}
          path: natmod/build/${{ matrix.arch }}*/
          if-no-files-found: error

That replaces roughly 70 lines of per-arch toolchain-install boilerplate (apt package selection, the xtensa tarball fetch, the esp-idf install + the esp-idf-venv-shadows-system-pip fix) with one uses: block per matrix leg, identical across every consuming repo.

Artifact upload is left to the caller on purpose -- artifact names and the exact glob under natmod/build/ differ per repo/module and aren't part of the shared contract.

Conventions this repo assumes

A module following the natmod/usermod layout looks like:

natmod/
  Makefile              # includes py/dynruntime.mk, dispatches on ARCH=
usermod/
  micropython.cmake
  micropython.mk
  manifest.py

build-natmod only assumes natmod/Makefile (or whatever natmod_dir points at) accepts ARCH= and MPY_DIR= and has a dist target that drops the built .mpy under build/<arch>*/. Nothing here assumes a specific module name, precision scheme, or test framework -- those stay in the consuming repo.

One more requirement for the cibuildmp CLI specifically (not build-natmod, which gives every arch its own job and checkout): scope dynruntime.mk's BUILD variable by $(ARCH) -- BUILD = .obj/$(ARCH) before the include, kept outside build/ so it does not collide with the dist output the CLI globs for (see examples/template/natmod/Makefile). cibuildmp with no --only runs every target sequentially in one natmod/ tree (D9), and dynruntime.mk defaults BUILD ?= build unscoped, so without this a second ARCH= in the same invocation finds the previous arch's own object files "up to date" (same path, source unchanged) and skips rebuilding -- the merged $(MOD).mpy silently stays the first arch's binary. cibuildmp catches this itself (the header-arch verification that is its auditwheel equivalent fails loudly instead), but scoping BUILD avoids paying for the failed build at all.

If the module also uses rv32imc's arch-flags (D15), BUILD needs $(ARCH_FLAGS) folded in too -- BUILD = .obj/$(ARCH)$(if $(ARCH_FLAGS),+$(ARCH_FLAGS)). Same bug, second axis: rv32imc's own object file does not depend on ARCH_FLAGS at all, so building several arch-flags variants back to back in one invocation (arch-flags = ["", "zba", "zba,zcmp"]) reuses the first variant's cached .o/.mpy for every later one just as silently, even though $(ARCH) never changed. Found the same way as the $(ARCH) case: by actually running the whole variant list, not by inspection.

None of this cares what produced the .c files SRC lists -- examples/wasm2mpy compiles WebAssembly to C via wasm2c in a Makefile rule before the same dynruntime.mk flow takes over, and needs nothing from cibuildmp beyond module-dir = "." and an extra-make-args entry for its own APP= variable.

Roadmap

cibuildmp is the roadmap. docs/BACKLOG.md is the plan of record: the decisions taken (and why), what is implemented, and what is deliberately deferred.

Where it stands: target selection, MicroPython and mpy-cross provisioning, cross-toolchain resolution, and the natmod build itself (running make, collecting the .mpy, verifying its header) are done. There is no separate publish step -- cibuildmp writes each identifier's own package.json as part of the normal build once version is set, the same way cibuildwheel has no publish step either. Adopted in all three consuming repos' natmod workflows and verified green on real CI, arch by arch (micropython-bclibc, a7p, micropython-wasm3) -- not just --dry-run. usermod is next: it will vendor board data from mpbuild rather than depend on the package, and test runners stay deferred.

Until usermod lands and is verified against real CI in a consuming repo the same way natmod now is, the composite actions below stay the supported path for it -- natmod's own composite actions (fetch-micropython, build-natmod) are still here too, unchanged, for anything that hasn't repinned to the cibuildmp CLI yet.

Versioning

Pin consumers to a tag, not @main and not a commit SHA. Bumping the tag a consumer references is a deliberate, visible edit in that repo, same as bumping any other CI dependency -- a change here never silently changes what three other repos' builds do.

v0.3.0 is the first tag where the cibuildmp CLI actually builds a module, not just plans it -- v0.3.0a1 (this repository's first tag) shipped the composite actions and the CLI's target-resolution half only. Both continue micropython-native-ci's version line rather than restarting it: the composite actions are v0.2.0's, moved. Consumers pinned to micropython-native-ci@v0.2.0 keep working until they repin -- that repository is deprecated, not deleted.

Older tags, on the old repository: v0.2.0 added build-usermod-windows, -webassembly, -rp2040, -armv7m and -esp32, and dropped the -arch name suffix (build-natmod-arch → build-natmod). v0.1.0 had only fetch-micropython, clone-micropython, build-natmod-arch and build-usermod-unix-arch.

The cibuildmp package and the actions share one version. The package is not on PyPI yet, so both actions install it from their own checkout (uv tool install ${{ github.action_path }}) -- which means the tool that runs is exactly the ref you pinned, with no index to keep in sync.