ffmpegcv 0.3.0

No project description provided

FFMPEGCV is an alternative to OPENCV for video read and write.

The ffmpegcv provide Video Reader and Video Witer with ffmpeg backbone, which are faster and powerful than cv2.

• The ffmpegcv is api compatible to open-cv.
• The ffmpegcv can use GPU accelerate encoding and decoding*.
• The ffmpegcv support much more video codecs v.s. open-cv.
• The ffmpegcv support RGB & BGR format as you like.
• The ffmpegcv can support ROI operations.You can crop, resize and pad the ROI.

In all, ffmpegcv is just similar to opencv api. But is faster* and with more codecs.

Basic example

Read a video by CPU, and rewrite it by GPU.

vidin = ffmpegcv.VideoCapture(vfile_in)
vidout = ffmpegcv.VideoWriterNV(vfile_out, 'h264', vidin.fps)

with vidin, vidout:
    for frame in vidin:
        cv2.imshow('image', frame)
        vidout.write(frame)

Read the camera by device name.

cap = ffmpegcv.VideoCaptureCAM("Integrated Camera")

Install

You need to download ffmpeg before you can use ffmpegcv

conda install ffmpeg

pip install ffmpegcv

GPU Accelation

• Support NVIDIA card only.
• Perfect in the Windows. That ffmpeg supports NVIDIA acceleration just by conda install.
• Struggle in the Linux. That ffmpeg didn't orginally support NVIDIA accelerate. Please re-compile the ffmpeg by yourself. See the link
• Works in the Google Colab notebook without pain (no need to recompile ffmpeg).
• Infeasible in the MacOS. That ffmpeg didn't supports NVIDIA at all.

* The ffmegcv GPU reader is a bit slower than CPU reader, but much faster when use ROI operations (crop, resize, pad).

Codecs

Codecs	OpenCV-reader	ffmpegcv-cpu-r	gpu-r	OpenCV-writer	ffmpegcv-cpu-w	gpu-w
h264	√	√	√	×	√	√
h265 (hevc)	not sure	√	√	×	√	√
mjpeg	√	√	×	√	√	×
mpeg	√	√	×	√	√	×
others	not sure	ffmpeg -decoders	×	not sure	ffmpeg -encoders	×

Benchmark

On the way...

Video Reader

---

The ffmpegcv is just similar to opencv in api.

# open cv
import cv2
cap = cv2.VideoCapture(file)
while True:
    ret, frame = cap.read()
    if not ret:
        break
    pass

# ffmpegcv
import ffmpegcv
cap = ffmpegcv.VideoCapture(file)
while True:
    ret, frame = cap.read()
    if not ret:
        break
    pass
cap.release()

# alternative
cap = ffmpegcv.VideoCapture(file)
nframe = len(cap)
for frame in cap:
    pass
cap.release()

# more pythonic, recommand
with ffmpegcv.VideoCapture(file) as cap:
    nframe = len(cap)
    for iframe, frame in enumerate(cap):
        if iframe>100: break
        pass

Use GPU to accelerate decoding. It depends on the video codes. h264_nvcuvid, hevc_nvcuvid ....

cap_cpu = ffmpegcv.VideoCapture(file)
cap_gpu = ffmpegcv.VideoCapture(file, codec='h264_cuvid') #NVIDIA GPU0
cap_gpu0 = ffmpegcv.VideoCaptureNV(file)         #NVIDIA GPU0
cap_gpu1 = ffmpegcv.VideoCaptureNV(file, gpu=1)  #NVIDIA GPU1

Use rgb24 instead of bgr24

cap = ffmpegcv.VideoCapture(file, pix_fmt='rgb24')
ret, frame = cap.read()
plt.imshow(frame)

ROI Operations

You can crop, resize and pad the video. These ROI operation is ffmpegcv-GPU > ffmpegcv-CPU >> opencv.

Crop video, which will be much faster than read the whole canvas.

cap = ffmpegcv.VideoCapture(file, crop_xywh=(0, 0, 640, 480))

Resize the video to the given size.

cap = ffmpegcv.VideoCapture(file, resize=(640, 480))

Resize and keep the aspect ratio with black border padding.

cap = ffmpegcv.VideoCapture(file, resize=(640, 480), resize_keepratio=True)

Crop and then resize the video.

cap = ffmpegcv.VideoCapture(file, crop_xywh=(0, 0, 640, 480), resize=(512, 512))

Video Writer

---

# cv2
out = cv2.VideoWriter('outpy.avi',
                       cv2.VideoWriter_fourcc('M','J','P','G'), 
                       10, 
                       (w, h))
out.write(frame1)
out.write(frame2)
out.release()

# ffmpegcv, default codec is 'h264' in cpu 'h265' in gpu.
# frameSize is decided by the size of the first frame
out = ffmpegcv.VideoWriter('outpy.mp4', None, 10)
out.write(frame1)
out.write(frame2)
out.release()

# more pythonic
with ffmpegcv.VideoWriter('outpy.mp4', None, 10) as out:
    out.write(frame1)
    out.write(frame2)

Use GPU to accelerate encoding. Such as h264_nvenc, hevc_nvenc.

out_cpu = ffmpegcv.VideoWriter('outpy.mp4', None, 10)
out_gpu0 = ffmpegcv.VideoWriterNV('outpy.mp4', 'h264', 10)        #NVIDIA GPU0
out_gpu1 = ffmpegcv.VideoWriterNV('outpy.mp4', 'hevc', 10, gpu=1) #NVIDIA GPU1

Input image is rgb24 instead of bgr24

out = ffmpegcv.VideoWriter('outpy.mp4', None, 10, pix_fmt='rgb24')
out.write(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB))

Video Reader and Writer

---

import ffmpegcv
vfile_in = 'A.mp4'
vfile_out = 'A_h264.mp4'
vidin = ffmpegcv.VideoCapture(vfile_in)
vidout = ffmpegcv.VideoWriter(vfile_out, None, vidin.fps)

with vidin, vidout:
    for frame in vidin:
        vidout.write(frame)

Camera Reader

---

*Experimental feature. The ffmpegcv offers Camera reader. Which is consistent with VideoFiler reader.

• The VideoCaptureCAM aims to support ROI operations. The Opencv will be general fascinating than ffmpegcv in camera read. I recommand the opencv in most camera reading case.
• The ffmpegcv can use name to retrieve the camera device. Use ffmpegcv.VideoCaptureCAM("Integrated Camera") is readable than cv2.VideoCaptureCAM(0).
• The VideoCaptureCAM will be laggy and dropping frames if your post-process takes long time. The VideoCaptureCAM will buffer the recent frames.
• The VideoCaptureCAM is continously working on background even if you didn't read it. Please release it in time.
• Works perfect in Windows, not-perfect in Linux and macOS.

import cv2
cap = cv2.VideoCapture(0)
while True:
    ret, frame = cap.read()
    cv2.imshow('frame', frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break
cap.release()

# ffmpegcv, in Windows&Linux
import ffmpegcv
cap = ffmpegcv.VideoCaptureCAM(0)
while True:
    ret, frame = cap.read()
    cv2.imshow('frame', frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break
cap.release()

# ffmpegcv use by camera name, in Windows&Linux
cap = ffmpegcv.VideoCaptureCAM("Integrated Camera")

# ffmpegcv use camera path if multiple cameras conflict
cap = ffmpegcv.VideoCaptureCAM('@device_pnp_\\\\?\\usb#vid_2304&'
    'pid_oot#media#0001#{65e8773d-8f56-11d0-a3b9-00a0c9223196}'
    '\\global')

# ffmpegcv use camera with ROI operations
cap = ffmpegcv.VideoCaptureCAM("Integrated Camera", crop_xywh=(0, 0, 640, 480), resize=(512, 512), resize_keepratio=True)

list all camera devices

from ffmpegcv.ffmpeg_reader_camera import query_camera_devices

devices = query_camera_devices()
print(devices)

{0: ('Integrated Camera', '@device_pnp_\\?\usb#vid_2304&pid_oot#media#0001#{65e8773d-8f56-11d0-a3b9-00a0c9223196}\global'),
1: ('OBS Virtual Camera', '@device_sw_{860BB310-5D01-11D0-BD3B-00A0C911CE86}\{A3FCE0F5-3493-419F-958A-ABA1250EC20B}')}

Set the camera resolution, fps, vcodec/pixel-format

from ffmpegcv.ffmpeg_reader_camera import query_camera_options

options = query_camera_options(0)  # or query_camera_options("Integrated Camera") 
print(options)
cap = ffmpegcv.VideoCaptureCAM(0, **options[-1])

[{'camcodec': 'mjpeg', 'campix_fmt': None, 'camsize_wh': (1280, 720), 'camfps': 60.0002}, {'camcodec': 'mjpeg', 'campix_fmt': None, 'camsize_wh': (640, 480), 'camfps': 60.0002}, {'camcodec': 'mjpeg', 'campix_fmt': None, 'camsize_wh': (1920, 1080), 'camfps': 60.0002}, {'camcodec': None, 'campix_fmt': 'yuyv422', 'camsize_wh': (1280, 720), 'camfps': 10}, {'camcodec': None, 'campix_fmt': 'yuyv422', 'camsize_wh': (640, 480), 'camfps': 30}, {'camcodec': None, 'campix_fmt': 'yuyv422', 'camsize_wh': (1920, 1080), 'camfps': 5}]

Known issues

1. The VideoCaptureCAM didn't give a smooth experience in macOS. You must specify all the camera parameters. And the query_camera_options woun't give any suggestion. That's because the ffmpeg cannot list device options using mac native avfoundation.

# The macOS requires full argument.
cap = ffmpegcv.VideoCaptureCAM('FaceTime HD Camera', camsize_wh=(1280,720), camfps=30, campix_fmt='nv12')

2. The VideoCaptureCAM cann't list the FPS in linux. Because the ffmpeg cound't query the device's FPS using linux native v4l2 module. However, it's just OK to let the FPS blank.