tensorcamera

Point your phone at something and see your own model's prediction, live.

This is the server half. The other half is the Tensor Camera app (iOS), which streams camera frames to whatever machine your model is on and shows what comes back.

pip install tensorcamera

Two minutes, no code

pip install 'tensorcamera[hf]'
tensorcamera serve --model objects

That downloads a model from Hugging Face, starts a server, and prints a QR code. Scan it with the app. tensorcamera presets lists the other built-in models, and any Hub id works too.

Your own model

from tensorcamera import TensorCamera

cam = TensorCamera(model="my-classifier", labels=["dog", "cat"])

@cam.on_frame
def predict(frame):
    probs = my_model(frame.resized(224)[None] / 255.0)[0]
    index = probs.argmax()
    return {"label": cam.labels[index], "confidence": float(probs[index])}

cam.serve()

serve() prints a QR code and listens on port 8080. Phone and computer need to be on the same Wi-Fi network.

The handler

Return whatever is convenient:

Return a label and a confidence if you can. Display text is for humans; label is what lets the app fire rules — play a sound, speak, vibrate, POST a webhook — when something is detected. A bare string leaves the app display-only.

The frame

@cam.on_frame
def predict(frame):
    frame.array        # numpy uint8, HWC, RGB — what Keras and torch expect
    frame.float01      # numpy float32, HWC, RGB, 0..1
    frame.resized(224) # numpy uint8, 224x224x3
    frame.jpeg         # raw JPEG bytes, undecoded
    frame.width, frame.height, frame.seq, frame.ts

Decoding is lazy. If your handler never touches the pixels, no decode happens.

Talking back to the phone

cam.send_action({"type": "sound", "asset": "bark.mp3"})
cam.send_action({"type": "speak", "text": "dog detected"})
cam.send_action({"type": "haptic", "style": "success"})

The app ignores anything it does not support, and refuses action types it did not advertise. It also rate-limits inbound actions, so a 10 fps stream cannot fire a sound ten times a second — but prefer device-side rules for that, since they keep working when the connection drops.

Keeping up

If inference is slower than the frame rate, predictions drift behind reality. Tell the phone how fast you actually are:

cam = TensorCamera(max_fps=3)

The app throttles to that budget. The server logs a warning the first time a handler overruns it.

Options

TensorCamera(
    model="my-classifier",   # shown in the app's diagnostics
    labels=["dog", "cat"],   # populates the rule editor's label picker (≤40)
    max_fps=10,              # frame budget asked of the phone
    quality=0.4,             # requested JPEG quality
    width=640, height=480,   # requested frame size
    token="shared-secret",   # require this from clients; carried in the QR code
)

cam.serve(host="0.0.0.0", port=8080, show_qr=True, name="Workshop MacBook")

Older app versions

The app on the App Store today predates this protocol. It sends raw float32 tensors on an untyped channel with no handshake, and expects a plain string back. This server detects that and handles it — including the newer app's legacy mode, which sends JPEG on the same channel. Both work, no flags.

You lose structured predictions and rules against an old client, because the v0 wire format has nowhere to put a label. Everything else works.

Protocol

The wire format is specified in PROTOCOL.md. You should not need it — that is the point of this package — but it is there if you want to write a server in another language. The app talks to anything that speaks it; this package is one implementation, not the contract.

Licence

MIT.