Tinkerbox for Apple Neural Engine

Run models on Apple's Neural Engine through Core AI. Critical inference always runs on the Neural Engine, the CPU does only supplementary work, and the GPU stays idle.

Requires Apple Silicon (M4 or newer), macOS 27, and Python 3.12 or 3.13. Earlier generations (M1–M3) are not supported due to Neural Engine hardware and offline compiler constraints.

Quickstart

Install the wheel from the GitHub release (Xcode is not required):

python -m pip install ./tinkerbox_ane-*.whl

import tinkerbox_ane

with tinkerbox_ane.load("d1-3b") as model:
    result = model.system_one(
        "I was charged twice this month.",
        {"refund": {"type": "noul", "instructions": "Is this a refund request?"}},
    )
    print(result)

tinkerbox_ane.load(model_id) loads any model in the catalog and returns the API for its task. It downloads the Core AI bundle for this Mac to ~/Library/Caches/tinkerbox-ane/models/<model>/<revision>/; pass a directory as the second argument to select a local AOT-compiled bundle. Each model package also has its own load(), such as tinkerbox_ane.d1.load().

Decision models expose the System One API: system_one, system_one_batch, probabilities, and probabilities_batch for binary, choice, and score queries, plus count_tokens. Several questions about one input share its computation. Inputs, limits, and execution details are in each model's notes.

Tinkerbox app

A standalone desktop application is in app. To build and launch it from source:

uv run --no-project --python 3.12 python app/run.py

Pass --build-only to create artifacts/swiftui/Tinkerbox.app and the release ZIP without launching. The prebuilt app manages its own Python environment and dependencies via an embedded uv binary.

Architecture & Staging

The Swift host manages persistent state and dynamically allocates attention cache buffers. Initial execution compiles and specializes the AOT asset for the local ANE hardware, after which Core AI reuses its persistent cache.

Weight staging uses an APFS copy-on-write workaround to eliminate redundant disk writes during specialization.

Adding a Model

Model-independent infrastructure is located at the package top level and is shared across models:

A model package (such as tinkerbox_ane.d1) provides:

• Neural Graph & Export: Fused FP16 graph (model.py) and Core AI export (export.py).
• Runtime: Tokenization, prompt formatting, readout (runtime.py), and optional tree planning (tree.py) or vision processing (vision.py).
• Package Interface: Exposes load(asset=None), bundle KIND, and compile/publish CLI wrappers.

To register a model, add an entry to src/tinkerbox_ane/models.json specifying id, name, publisher, summary, task, repo, revision, and min_version. Add notes in docs/models/<id>.md.

Building from Source

Building requires Xcode 27 / Swift 6.4 and uv:

uv sync --locked
uv build

The wheel in dist/ bundles the native tinkerbox-ane-host executable and links against macOS system frameworks. Source and editable installs compile the host during installation. Compilation verifies source identity, function manifests, and full ANE placement.

Each model's notes describe its export, compile, and publishing commands. Reusable Core AI compilation and Hugging Face publishing utilities are provided in tinkerbox_ane.coreai and tinkerbox_ane.publish. To use a compiled bundle locally before publishing:

uv run python -m tinkerbox_ane.install <model> <bundle directory>

Tests

uv run python -m tinkerbox_ane.d1.guard artifacts/validation/smoke -- uv run pytest -q