Skip to main content

PhoneSensor for machinevisiontoolbox. Get camera and IMU data from a camera remotely with Python

Project description

machinevision-toolbox-python.phone-sensor

This package provides a PhoneSensor class which hosts a webapp through which you can easily retrieve camera and orientation data (if supported) from a smart-phone for example.

For best results, use with the latest possible version of Chrome on Android.

Quick start

Install with pip:

pip install -U --force-reinstall machinevision-toolbox-python.phone-sensor

Example Code:

from phone_sensor import PhoneSensor
from matplotlib import pyplot as plt
import numpy as np

# Hosts a webserver in a background thread.
# And display a QR code link to the app
with PhoneSensor(qrcode=True) as phone:
    # wait for button press to snap a photo
    bgr, time = phone.grab(button=True)
    # get device orientation as a Quaternion
    imu_data = phone.imu()

    plt.subplot(1, 2, 1)
    # img is bgr (opencv style), matplotlib uses RGB - so flip
    rgb = np.flip(bgr, axis=2)
    plt.imshow(rgb)
    plt.title(f"t = {time}")

    plt.subplot(1, 2, 2)
    plt.bar(['x', 'y', 'z', 'w'], imu_data.quaternion)
    plt.title(f"t = {imu_data.unix_timestamp}")
    plt.show()

which will output:

example

The above code will create an ASCII QRcode in your terminal. Scanning it with a smartphone will take you to a webapp hosted by the `PhoneSensor`. Due to browser API security concerns, the app must be served over HTTPS, which in this case requires a self-signed certificate. Therefore you must accept the certificate before accessing the app (don't be scared!). For example, on Firefox Android: drawing

Once done, you should be met with the main UI, which looks like:

UI screenshot

The output img is a width x height x 3 BGR np.ndarray like you'd expect from machinevisiontoolbox or opencv-python.

Documentation

PhoneSensor()

class phone_sensor.PhoneSensor(): ...

def PhoneSensor.__init__(self, *, qrcode=False, host='0.0.0.0', port=8000,
                         logger=logging.getLogger('mvt.phone_sensor'), log_level=logging.WARN,
                         proxy_client_from=None)
  • Parameters

    • qrcode (bool) – True to output a QRCode in the terminal window that points to the server accessible via LAN, defaults to False

    • host (str) – Which hostname/ip to host the server on, defaults to “0.0.0.0”

    • port (int) – Which port to host the server on, defaults to 8000

    • logger (Logger) – A standard logging.Logger for debugging and general log information, defaults to logging.getLogger(‘mvt.phone_sensor’)

    • log_level (int) – Log level for the aforementioned logger, defaults to logging.WARN

    • proxy_client_from (Optional[str]) – A separate host from which to proxy the web client, defaults to None. Mainly for development purposes, using a hot-reloaded webpack server for the client rather than the one shipped with your pip install


PhoneSensor.close()

def PhoneSensor.close(self)

Close the server and relinquish control of the port. Use of PhoneSensor as a context manager is preferred to this where suitable. May be called automatically by the garbage collector.


PhoneSensor.grab()

def PhoneSensor.grab(self, cam='back', *, resolution=(640, 480), button=False, wait=None,
                     encoding='webp', quality=90) -> Tuple[np.ndarray, float]

Grab an image from the first/currently connected webapp client

  • Parameters

    • cam (Literal[‘front’, ‘back’]) – Default camera to use, defaults to ‘back’. Most smartphones have a ‘front’ (the side with the touchscreen) and a ‘back’ camera. This may be temporarily overridden through the menu on the client.

    • resolution (Tuple[int, int]) – The desired resolution (width, height) of the photo, defaults to (640, 480). Choosing lower values will increase performance, allowing you to take more photos in quicker succession. Note this is not necessarily respected - It’s up to the browser’s implementation which resolution it chooses, with this value being the ‘ideal’. For example, if you ask for (999999, 480) the browser might choose (640, 480) instead.

    • button (bool) – True to wait for button press, defaults to False.

    • wait (Optional[float]) – Minimum amount of time to wait since previous photo before taking a new one, defaults to None. Incompatible with the button arg.

    • encoding (Literal[‘jpeg’, ‘png’, ‘webp’, ‘bmp’]) – The encoding mimetype for the image, defaults to ‘webp’. In order of most to least performance, the recommended options are: [‘webp’, ‘jpeg’, ‘png’, ‘bmp’]. ‘webp’ and ‘jpeg’ are lossy compressions, so they will have differing compression artifacts. ‘png’ and ‘bmp’ are lossless. ‘bmp’ is essentially “no encoding” so you may use this if network is not a bottleneck (which it typically is). Otherwise ‘png’ is also lossless.

    • quality (int) – The quality (within (0, 100]) at which to encode the image, defaults to 90. Lower may slightly increase performance at the cost of image quality, however, the effect is typically insignificant. Does nothing for lossless encodings such as ‘png’.

  • Raises

    PhoneSensor.ClientDisconnect – If the device disconnects from the app after receiving the command.

  • Return type

    Tuple[ndarray, float]

  • Returns

    An (img, timestamp) tuple, where img is a numpy.ndarray in the format you would expect from OpenCV (h x w x bgr) and timestamp is a unix timestamp from the client device (seconds since epoch)


PhoneSensor.imu()

def PhoneSensor.imu(self, wait=None) -> ImuDataFrame

Retrieve orientation and motion data from a capable device.

  • Parameters

    wait (Optional[float]) – Minimum amount of time to wait since previous reading before taking a new one, defaults to None.

  • Raises

    • PhoneSensor.ClientDisconnect – If the device disconnects from the app after receiving the command.

    • PhoneSensor.DataUnavailable – if the device is incapable of providing the data (eg. desktop pc), or if the browser disallows it, either due to app permissions or if it does not support the features.

  • Return type

    PhoneSensor.ImuDataFrame

  • Returns

    An ImuDataFrame, with the orientation and IMU sensor data.


PhoneSensor.ImuDataFrame

class PhoneSensor.ImuDataFrame:
    unix_timestamp: float
    quaternion: Tuple[float, float, float, float] # (x, y, z, w)
    # if supported by the browser (rare)
    accelerometer: Optional[Tuple[float, float, float]] # (x, y, z)
    gyroscope: Optional[Tuple[float, float, float]] # (x, y, z)
    magnetometer: Optional[Tuple[float, float, float]] # (x, y, z)

A collection of sensor readings taken from the phone and the time at which it was recorded. Includes raw accelerometer, magnetometer and gyroscope tuples if supported by the browser (generally only new versions of Android Chrome).

Contributing

PRs welcome! The stack is Python3.6 and Typescript4.1 & CreateReactApp4.0

Development Install

First, clone this repo and install dependencies

npm install
pip install -e .

Running in Development Mode

  1. Run the dev-server (and keep it running):
npm start
  1. Create a PhoneSensor in proxy mode:
>>> from phone_sensor import PhoneSensor
>>> phone = PhoneSensor(qrcode=True, proxy_client_from="localhost:3000")
>>> phone.grab(button=True)

or just run the python examples/devmode.py

Then click the link in terminal to test the app

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

Built Distribution

File details

Details for the file machinevision-toolbox-python.phone-sensor-1.0.4.tar.gz.

File metadata

  • Download URL: machinevision-toolbox-python.phone-sensor-1.0.4.tar.gz
  • Upload date:
  • Size: 11.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/54.1.1 requests-toolbelt/0.9.1 tqdm/4.59.0 CPython/3.9.2

File hashes

Hashes for machinevision-toolbox-python.phone-sensor-1.0.4.tar.gz
Algorithm Hash digest
SHA256 1e13e52bc47e4bf03be076bf1e67e2ddc3a3b6519512c466f01871594a72103b
MD5 f3386f51926b9db63a82075e57232b51
BLAKE2b-256 717a51d868f52d80115123f4d677c2cb4bccabc8bb7997a19fc79cf29adbbb0a

See more details on using hashes here.

File details

Details for the file machinevision_toolbox_python.phone_sensor-1.0.4-py3-none-any.whl.

File metadata

File hashes

Hashes for machinevision_toolbox_python.phone_sensor-1.0.4-py3-none-any.whl
Algorithm Hash digest
SHA256 d602ae1518206ab30562676fd2698204057caa65657460fb0efa804a8e876d21
MD5 e302c1ffcf0615aa97d89155b9cc718d
BLAKE2b-256 560b72ad568f4244d9f2509dc38a90f328e2a647b4f9ba77b674b2513f81009c

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page