Python implemention of the TensorFlow BodyPix model.
Project description
TensorFlow BodyPix (TF BodyPix)
A Python implementation of body-pix.
Goals of this project is:
- Python library, making it easy to integrate the BodyPix model
- CLI with limited functionality, mostly for demonstration purpose
Prerequisits
- Python 3.7+
Install
Install with all dependencies:
pip install tf-bodypix[all]
Install with minimal or no dependencies:
pip install tf-bodypix
Extras are provided to make it easier to provide or exclude dependencies when using this project as a library:
extra name | description |
---|---|
tf | TensorFlow (required). But you may use your own build. |
tfjs | TensorFlow JS Model support, using tfjs-graph-converter |
tflite | tflite-runtime |
image | Image loading via Pillow, required by the CLI. |
video | Video support via OpenCV |
webcam | Webcam support via OpenCV and pyfakewebcam |
all | All of the libraries (except tflite-runtime ) |
Python API
from pathlib import Path
import tensorflow as tf
from tf_bodypix.api import download_model, load_model, BodyPixModelPaths
# setup input and output paths
output_path = Path('./data/example-output')
output_path.mkdir(parents=True, exist_ok=True)
input_url = (
'https://www.dropbox.com/s/7tsaqgdp149d8aj/serious-black-businesswoman-sitting-at-desk-in-office-5669603.jpg?dl=1'
)
local_input_path = tf.keras.utils.get_file(origin=input_url)
# load model (once)
bodypix_model = load_model(download_model(
BodyPixModelPaths.MOBILENET_FLOAT_50_STRIDE_16
))
# get prediction result
image = tf.keras.preprocessing.image.load_img(local_input_path)
image_array = tf.keras.preprocessing.image.img_to_array(image)
result = bodypix_model.predict_single(image_array)
# simple mask
mask = result.get_mask(threshold=0.75)
tf.keras.preprocessing.image.save_img(
f'{output_path}/output-mask.jpg',
mask
)
# colored mask (separate colour for each body part)
colored_mask = result.get_colored_part_mask(mask)
tf.keras.preprocessing.image.save_img(
f'{output_path}/output-colored-mask.jpg',
colored_mask
)
# poses
from tf_bodypix.draw import draw_poses # utility function using OpenCV
poses = result.get_poses()
image_with_poses = draw_poses(
image_array.copy(), # create a copy to ensure we are not modifing the source image
poses,
keypoints_color=(255, 100, 100),
skeleton_color=(100, 100, 255)
)
tf.keras.preprocessing.image.save_img(
f'{output_path}/output-poses.jpg',
image_with_poses
)
CLI
CLI Help
python -m tf_bodypix --help
or
python -m tf_bodypix <sub command> --help
List Available Models
python -m tf_bodypix list-models
The result will be a list of all of the bodypix
TensorFlow JS models available in the tfjs-models bucket.
Those URLs can be passed as the --model-path
arguments below, or to the download_model
method of the Python API.
The CLI will download and cache the model from the provided path. If no --model-path
is provided, it will use a default model (mobilenet).
To list TensorFlow Lite models instead:
python -m tf_bodypix list-tflite-models
Inputs and Outputs
Most commands will work with inputs (source) and outputs.
The source path can be specified via the --source
parameter.
The following inputs are supported:
type | description |
---|---|
image | Static image (e.g. .png ) |
video | Video (e.g. .mp4 ) |
webcam | Linux Webcam (/dev/videoN or webcam:0 ) |
If the source path points to an external file (e.g. https://
), then it will be downloaded and locally cached.
The output path can be specified via --output
, unless --show-output
is used.
The following outpus are supported:
type | description |
---|---|
image_writer | Write to a static image (e.g. .png ) |
v4l2 | Linux Virtual Webcam (/dev/videoN ) |
window | Display a window (by using --show-output ) |
Example commands
Creating a simple body mask
python -m tf_bodypix \
draw-mask \
--source \
"https://www.dropbox.com/s/7tsaqgdp149d8aj/serious-black-businesswoman-sitting-at-desk-in-office-5669603.jpg?dl=1" \
--show-output \
--threshold=0.75
Image Source: Serious black businesswoman sitting at desk in office
Add the mask over the original image using --mask-alpha
python -m tf_bodypix \
draw-mask \
--source \
"https://www.dropbox.com/s/7tsaqgdp149d8aj/serious-black-businesswoman-sitting-at-desk-in-office-5669603.jpg?dl=1" \
--show-output \
--threshold=0.75 \
--mask-alpha=0.5
Image Source: Serious black businesswoman sitting at desk in office
Colorize the body mask depending on the body part
python -m tf_bodypix \
draw-mask \
--source \
"https://www.dropbox.com/s/7tsaqgdp149d8aj/serious-black-businesswoman-sitting-at-desk-in-office-5669603.jpg?dl=1" \
--show-output \
--threshold=0.75 \
--mask-alpha=0.5 \
--colored
Image Source: Serious black businesswoman sitting at desk in office
Additionally select the body parts
python -m tf_bodypix \
draw-mask \
--source \
"https://www.dropbox.com/s/7tsaqgdp149d8aj/serious-black-businesswoman-sitting-at-desk-in-office-5669603.jpg?dl=1" \
--show-output \
--threshold=0.75 \
--mask-alpha=0.5 \
--parts left_face right_face \
--colored
Image Source: Serious black businesswoman sitting at desk in office
Add mask overlay to a video
python -m tf_bodypix \
draw-mask \
--source \
"https://www.dropbox.com/s/s7jga3f0dreavlb/video-of-a-man-laughing-and-happy-1608393-360p.mp4?dl=1" \
--show-output \
--threshold=0.75 \
--mask-alpha=0.5 \
--colored
Video Source: Video Of A Man Laughing And Happy
Add pose overlay to a video
python -m tf_bodypix \
draw-pose \
--source \
"https://www.dropbox.com/s/pv5v8dkpj5wung7/an-old-man-doing-a-tai-chi-exercise-2882799-360p.mp4?dl=1" \
--show-output \
--threshold=0.75
Blur background of a video
python -m tf_bodypix \
blur-background \
--source \
"https://www.dropbox.com/s/s7jga3f0dreavlb/video-of-a-man-laughing-and-happy-1608393-360p.mp4?dl=1" \
--show-output \
--threshold=0.75 \
--mask-blur=5 \
--background-blur=20
Video Source: Video Of A Man Laughing And Happy
Replace the background of a video
python -m tf_bodypix \
replace-background \
--source \
"https://www.dropbox.com/s/s7jga3f0dreavlb/video-of-a-man-laughing-and-happy-1608393-360p.mp4?dl=1" \
--background \
"https://www.dropbox.com/s/b22ss59j6pp83zy/brown-landscape-under-grey-sky-3244513.jpg?dl=1" \
--show-output \
--threshold=0.75 \
--mask-blur=5
Video Source: Video Of A Man Laughing And Happy
Background: Brown Landscape Under Grey Sky
Capture Webcam and adding mask overlay
python -m tf_bodypix \
draw-mask \
--source webcam:0 \
--show-output \
--threshold=0.75 \
--mask-alpha=0.5 \
--colored
Capture Webcam and adding mask overlay, writing to v4l2loopback device
(replace /dev/videoN
with the actual virtual video device)
python -m tf_bodypix \
draw-mask \
--source webcam:0 \
--output /dev/videoN \
--threshold=0.75 \
--mask-alpha=0.5 \
--colored
Capture Webcam and blur background, writing to v4l2loopback device
(replace /dev/videoN
with the actual virtual video device)
python -m tf_bodypix \
blur-background \
--source webcam:0 \
--background-blur 20 \
--output /dev/videoN \
--threshold=0.75
Capture Webcam and replace background, writing to v4l2loopback device
(replace /dev/videoN
with the actual virtual video device)
python -m tf_bodypix \
replace-background \
--source webcam:0 \
--background \
"https://www.dropbox.com/s/b22ss59j6pp83zy/brown-landscape-under-grey-sky-3244513.jpg?dl=1" \
--threshold=0.75 \
--output /dev/videoN
Background: Brown Landscape Under Grey Sky
TensorFlow Lite Model support (experimental)
The model path may also point to a TensorFlow Lite model (.tflite
extension). Whether that actually improves performance may depend on the platform and available hardware.
You could convert one of the available TensorFlow JS models to TensorFlow Lite using the following command:
python -m tf_bodypix \
convert-to-tflite \
--model-path \
"https://storage.googleapis.com/tfjs-models/savedmodel/bodypix/mobilenet/float/075/model-stride16.json" \
--optimize \
--quantization-type=float16 \
--output-model-file "./mobilenet-float-multiplier-075-stride16-float16.tflite"
The above command is provided for convenience. You may use alternative methods depending on your preference and requirements.
Relevant links:
TensorFlow Lite Runtime support (experimental)
This project can also be used with tflite-runtime instead of full TensorFlow (e.g. by using the tflite
extra).
However, TensorFlow Lite converter would require full TensorFlow.
In order to avoid it, one needs to use a TensorFlow Lite model (see previous section).
Docker Usage
You could also use the Docker image if you prefer.
The entrypoint will by default delegate to the CLI, except for python
or bash
commands.
# pull latest image (you may also use tags)
docker pull de4code/tf-bodypix
# mount real and virtual webcam devices on linux
docker run --rm \
--device /dev/video0 \
--device /dev/video2 \
de4code/tf-bodypix \
blur-background \
--source /dev/video0 \
--output /dev/video2 \
--background-blur 20 \
--threshold=0.75
# mount x11 display on linux
docker run --rm \
--net=host \
--volume /tmp/.X11-unix:/tmp/.X11-unix \
--volume ${HOME}/.Xauthority:/root/.Xauthority \
--env DISPLAY \
de4code/tf-bodypix \
replace-background \
--source \
"https://www.dropbox.com/s/s7jga3f0dreavlb/video-of-a-man-laughing-and-happy-1608393-360p.mp4?dl=1" \
--background \
"https://www.dropbox.com/s/b22ss59j6pp83zy/brown-landscape-under-grey-sky-3244513.jpg?dl=1" \
--show-output \
--threshold=0.75 \
--mask-blur=5
Example Media
Here are a few example media files you could try.
Images:
- Serious black businesswoman sitting at desk in office (Source)
- Woman Wearing Gray Notch Lapel Suit Jacket (Source)
- Smiling Woman Standing In Front Of A Colorful Flag (Source)
- Man and Woman Smiling Inside Building (Source)
- Two Woman in Black Sits on Chair Near Table (Source)
- Female barista in beanie and apron resting chin on had (Source)
- Smiling Woman Holding White Android Smartphone While Sitting Front of Table (Source)
- Woman Having Coffee and Rice Bowl (Source)
- Woman Smiling While Holding a Coffee Cup (Source)
Videos:
- Video Of A Man Laughing And Happy (Source)
- A Group Of People In A Business Meeting (Source)
- An Old Man Doing A Tai Chi Exercise (Source)
Background:
Experimental Downstream Projects
- Layered Vision is an experimental project using the
tf-bodypix
Python API.
Acknowledgements
- Original TensorFlow JS Implementation of BodyPix
- Linux-Fake-Background-Webcam, an implementation of the blog post describing using the TensorFlow JS implementation with Python via a Socket API.
- tfjs-to-tf for providing an easy way to convert TensorFlow JS models
- virtual_webcam_background for a great pure Python implementation
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for tf_bodypix-0.4.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b97fd758e3dd656d0080d913770e3702d98449a9bda60cd4c698e7b157c27c7d |
|
MD5 | e28d2807fc32b9f50d28bba0c687d3f8 |
|
BLAKE2b-256 | ed65a96fe53b4c201cf9eb430eb8ffb88c81e6322a59ec862459552ad717b412 |