openmmla-vision 0.1.0

Vision module for the OpenMMLA platform.

OpenMMLA Vision

Video module of the mBox multimodal learning analytic system. For more details, please refer to mbox System design.

Uber Server Setup

Before setting up the video base, you need to set up a server hosting the InfluxDB, Redis, Mosquitto, and Nginx services. Please refer to mbox-uber module.

Video Base & Server Setup

Downloading and Setting up the mbox-video analysis system is accomplished in two steps:
(1) Clone the repository from GitHub to your local home directory.
(2) Install the required dependencies.

1. Clone the repository from GitHub

   git clone https://github.com/ucph-ccs/mbox-video.git
   

2. Install the required dependencies with conda environment

   • Conda

     # For Raspberry Pi
     wget "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"
     bash Miniforge3-$(uname)-$(uname -m).sh
     
     # For Mac and Linux
     wget "https://repo.anaconda.com/miniconda/Miniconda3-latest-$(uname)-$(uname -m).sh"
     bash Miniconda3-latest-$(uname)-$(uname -m).sh
     

   • Video Base

     # For video base
     conda create -c conda-forge -n video-base python=3.10.12 -y
     conda activate video-base
     cd mbox-video
     pip install openmmla-vision
     

Usage

After setting up the uber server and installation of dependencies, you can run the real-time video analysis by following the instructions:

1. Stream video from your camera(s) to the server (e.g., Raspberry Pi, MacBook, etc.)
   If you have multiple cameras and each wired to one machine, you can either run the video streaming on each machine or stream all cameras to a centralized RTMP server by following the guidelines in raspi_rtmp_streaming.md.

2. Calibrate camera's intrinsic parameters

   1. Go to ./camera_calib/pattern, print out the chessboard image and stick it on a flat surface.
   2. Capture the chessboard image with your camera and calibrate the camera by running ./calib_camera.sh bash script.

3. Synchronize multi-cameras' coordinate systems

   If more than one camera is to be used at the same time, you need to calculate the transformation matrix between the main and the alternative camera. The transformation matrix is used to convert the alternative camera's coordinate system to the main camera's coordinate system.

   1. Centralized mode
      If your cameras are streaming to a centralized RTMP server or serially wired to a single machine, then run

      # :param -d the number of cameras needed to sync, default to 2. 
      # :param -s the number of camera sync manager, default to 1.
      # e.g.
      ./sync_camera.sh -d 2 -s 1
      

   2. Distributed Mode
      If your cameras are not streamed to a centralized RTMP server and serially wired to a single machine, run camera detector on each your camera hosting machine, and camera sync manager on your synchronizing machine.

      # on camera hosting machine, e.g. Raspberry Pi
      ./sync_camera.sh -d 1 -s 0
      # on synchronizing machine, e.g. MacBook
      ./sync_camera.sh -d 0 -s 1
      

4. Run real-time video analysis system

   1. Centralized mode
      If your cameras are streaming to a centralized RTMP server or serially wired to a single machine, then run

      # :param -b the number of video base needed to run, default to 1. 
      # :param -s the number of video base synchronizer needed to run, default to 1. 
      # :param -v the number of visualizer need to run, default to 1. 
      # :param -g whether to display the graphic window, default to true. 
      # :param -r whether to record the video frames in image for clip baking, default to false.
      # :param -v whether to store the real time visualizations with visualizer, default to false.
      # e.g.
      ./run.sh
      

   2. Distributed Mode
      If your cameras are not streamed to a centralized RTMP server and serially wired to a single machine, run video base on each camera hosting machine and run the synchronizer and visualizer on your synchronizing machine.

      # on camera hosting machine, e.g. Raspberry Pi
      ./run.sh -b 1 -s 0 -v 0 -g false
      # on synchronizing machine, e.g. MacBook
      ./run.sh -b 0 -s 1 -v 1
      

FAQ

Citation

If you use this code in your research, please cite the following paper:

@inproceedings{inproceedings,
author = {Li, Zaibei and Jensen, Martin and Nolte, Alexander and Spikol, Daniel},
year = {2024},
month = {03},
pages = {785-791},
title = {Field report for Platform mBox: Designing an Open MMLA Platform},
doi = {10.1145/3636555.3636872}
}

References

• apriltags

License

This project is licensed under the MIT License - see the LICENSE file for details.