Skip to main content

PhotonRT

PhotonRT is a CPU-first native inference runtime for the Photon image-captioning model and its MobileCLIP-S1 image encoder.

Current pipeline

image (preprocessed CHW float32)
        |
        v
MobileCLIP-S1 native CPU encoder
        |
       512-D
        |
        v
Photon projector
        |
      8 x 256
        |
        v
Photon decoder (6 layers)
        |
        v
caption

The Photon decoder is implemented in C++ with a KV-cache generation path. MobileCLIP-S1 is implemented as a reparameterized FastViT/MCI-1 CPU graph. The upstream MobileCLIP repository defines S1 as a 256-pixel image model with a 512-D embedding and MCI-1 image encoder; see the official Apple implementation for model details.

Build (Windows + Visual Studio generator)

cmake -S . -B build
cmake --build build --config Release --parallel
ctest --test-dir build -C Release --output-on-failure

Convert Photon checkpoint

python tools/convert_photon.py \
  --checkpoint "C:/path/to/nano_ep55.pt" \
  --tokenizer "C:/path/to/mobilecap_tokenizer.json" \
  --model-out photon.f32.photon \
  --tokenizer-out photon.tokenizer

Convert MobileCLIP-S1

Use Apple's official MobileCLIP Python package for conversion:

python tools/convert_mobileclip.py \
  --checkpoint "C:/path/to/mobileclip_s1.pt" \
  --output mobileclip-s1.f32.mclip

The converter writes reparameterized inference weights and does not bundle the original Apple checkpoint.

Prepare an exact-reference MobileCLIP input

python tools/preprocess_mobileclip.py \
  --image "C:/path/to/image.jpg" \
  --output image_chw.f32

Run native MobileCLIP

./build/Release/mobileclip-embedding.exe \
  mobileclip-s1.f32.mclip \
  image_chw.f32

Compare native MobileCLIP against PyTorch

python tools/compare_mobileclip.py \
  --checkpoint "C:/path/to/mobileclip_s1.pt" \
  --image-tensor image_chw.f32 \
  --native-embedding mobileclip_embedding.f32

Run end-to-end C++ image captioning

./build/Release/image-caption.exe \
  mobileclip-s1.f32.mclip \
  photon.f32.photon \
  photon.tokenizer \
  image_chw.f32

Important release note

PhotonRT can distribute its own runtime code under its chosen open-source license. Apple's MobileCLIP source code is MIT, while the pretrained MobileCLIP model weights are governed by Apple's model terms; do not bundle Apple's weights into a release without checking those terms.

Status

  • Photon C++ CPU reference runtime: working
  • real Photon checkpoint conversion: working
  • Photon generation: working
  • native MobileCLIP-S1 CPU graph: implemented
  • PyTorch/native MobileCLIP parity: next validation target
  • optimized SIMD kernels: next
  • quantization: next
  • CUDA kernels: next
  • JPEG/PNG native image decoding: next
  • Python bindings: next

Metadata

Release files for photonrt 0.2.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for photonrt 0.2.1
File Size Uploaded Provenance Actions
photonrt-0.2.1.tar.gz 6.4 MB Without provenance Details for photonrt-0.2.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for photonrt 0.2.1
File Interpreter ABI Platform Actions
photonrt-0.2.1-cp310-cp310-win_amd64.whl CPython 3.10 CPython 3.10 Windows x86-64 Details for photonrt-0.2.1-cp310-cp310-win_amd64.whl

Total release size: 12.6 MB

Release files / photonrt-0.2.1.tar.gz

Download URL photonrt-0.2.1.tar.gz
Size 6.4 MB
Tags Source
SHA-256 checksum
How to use checksums
49f694d3b3e19fd9343ad34e0df8fdc42f173e5e5c7a1a7dda0f2b1752f545f7
BLAKE2b-256 checksum
How to use checksums
7c1b6f249d568b4afaf57b208344faf0a56b378aca5ada87a988441a3285daf3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.10.20

Release files / photonrt-0.2.1-cp310-cp310-win_amd64.whl

Download URL photonrt-0.2.1-cp310-cp310-win_amd64.whl
Size 6.2 MB
Tags CPython 3.10 Windows x86-64
SHA-256 checksum
How to use checksums
18a450f652e4c2eec045af4e69e0f2f26648fc11d21a7652032aec0c26aa98a4
BLAKE2b-256 checksum
How to use checksums
56730776e76a32fc55db9c8e4bb58642cb9cbcd349d788dc1d92c98a04c6b30a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.10.20

Security for photonrt 0.2.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.2.1 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page