tensorcamera

Point your phone at something and see your own model's prediction, live.

This is the server half. The other half is the Tensor Camera app (iOS), which streams camera frames to whatever machine your model is on and shows what comes back.

pip install tensorcamera

Two minutes, no code

pip install 'tensorcamera[hf]'
tensorcamera serve --model objects

That downloads a model from Hugging Face, starts a server, and prints a QR code:

  Tensor Camera server on ws://192.168.2.118:8080
  model: google/vit-base-patch16-224

  Scan this with the Tensor Camera app:

   ▄▄▄▄▄▄▄ ▄ ▄▄▄ ▄▄▄▄ ▄▄ ▄▄▄  ▄  ▄▄▄▄▄▄▄
   █ ▄▄▄ █  ▄ ▄ ▀▄█▀█▀▀ █▀▀█  ▄█ █ ▄▄▄ █
   █ ███ █ █▄▀ ▀██▄▀██ ▄ ▄█▀██▄█ █ ███ █
   █▄▄▄▄▄█ █▀█▀█ ▄▀▄▀▄▀█ ▄ ▄▀▄ ▄ █▄▄▄▄▄█
                    ( … )

  Won't scan? Open the image instead:  /tmp/tensorcamera-qr.png
  ...or enter the address by hand:      ws://192.168.2.118:8080

  Phone and computer must be on the same Wi-Fi network.

Scan it and you are connected. The code carries the address and, if you set one, the auth token — so there is no IP address to type on a phone keyboard.

A PNG is always written alongside the terminal version, because block-character QR codes depend on the font's line spacing and do not survive every terminal or screenshot. If the printed one will not scan, open the file.

tensorcamera presets lists the other built-in models, and any Hugging Face Hub id works too.

Ports

The default is 8080, which is a crowded neighbourhood. If it is taken, the server steps to the next free port and says what was in the way:

  port 8080 is busy (held by node (pid 66527)) — trying 8081
  using port 8081 instead of 8080

The port is bound before the QR code prints, so the code always encodes a port that is genuinely being served — it can never advertise a server that failed to start. Pass --strict-port (or auto_port=False) to fail instead of stepping, which is what you want under a process manager.

Your own model

from tensorcamera import TensorCamera

cam = TensorCamera(model="my-classifier", labels=["dog", "cat"])

@cam.on_frame
def predict(frame):
    probs = my_model(frame.resized(224)[None] / 255.0)[0]
    index = probs.argmax()
    return {"label": cam.labels[index], "confidence": float(probs[index])}

cam.serve()

serve() prints a QR code and listens on port 8080. Phone and computer need to be on the same Wi-Fi network.

The handler

Return whatever is convenient:

Return a label and a confidence if you can. Display text is for humans; label is what lets the app fire rules — play a sound, speak, vibrate, POST a webhook — when something is detected. A bare string leaves the app display-only.

The frame

@cam.on_frame
def predict(frame):
    frame.array        # numpy uint8, HWC, RGB — what Keras and torch expect
    frame.float01      # numpy float32, HWC, RGB, 0..1
    frame.resized(224) # numpy uint8, 224x224x3
    frame.jpeg         # raw JPEG bytes, undecoded
    frame.width, frame.height, frame.seq, frame.ts

Decoding is lazy. If your handler never touches the pixels, no decode happens.

Talking back to the phone

cam.send_action({"type": "sound", "asset": "bark.mp3"})
cam.send_action({"type": "speak", "text": "dog detected"})
cam.send_action({"type": "haptic", "style": "success"})

The app ignores anything it does not support, and refuses action types it did not advertise. It also rate-limits inbound actions, so a 10 fps stream cannot fire a sound ten times a second — but prefer device-side rules for that, since they keep working when the connection drops.

Keeping up

If inference is slower than the frame rate, predictions drift behind reality. Tell the phone how fast you actually are:

cam = TensorCamera(max_fps=3)

The app throttles to that budget. The server logs a warning the first time a handler overruns it.

Options

TensorCamera(
    model="my-classifier",   # shown in the app's diagnostics
    labels=["dog", "cat"],   # populates the rule editor's label picker (≤40)
    max_fps=10,              # frame budget asked of the phone
    quality=0.4,             # requested JPEG quality
    width=640, height=480,   # requested frame size
    token="shared-secret",   # require this from clients; carried in the QR code
)

cam.serve(
    host="0.0.0.0",
    port=8080,           # steps to the next free port if taken
    show_qr=True,
    name="Workshop MacBook",
    auto_port=True,      # False to fail on a busy port instead
    qr_png="qr.png",     # where to write the scannable image
)

Older app versions

The app on the App Store today predates this protocol. It sends raw float32 tensors on an untyped channel with no handshake, and expects a plain string back. This server detects that and handles it — including the newer app's legacy mode, which sends JPEG on the same channel. Both work, no flags.

You lose structured predictions and rules against an old client, because the v0 wire format has nowhere to put a label. Everything else works.

Protocol

The wire format is specified in PROTOCOL.md. You should not need it — that is the point of this package — but it is there if you want to write a server in another language. The app talks to anything that speaks it; this package is one implementation, not the contract.

Licence

MIT.