Gstreamer duplicate camera stream very slow on vim3

rahul_s · December 23, 2021, 7:00am

What we have done :

We have duplicated the camera stream. One of it is being streamed on ANPR frontend desktop. While the second is being consumed by ANPR backend to detect number plates. This script runs

on khadas startup and duplicated the camera.

#!/bin/bash
sudo modprobe v4l2loopback devices=3 video_nr=10 card_label="OBS Cam" exclusive_caps=1
gst-launch-1.0 v4l2src device=/dev/video0  ! tee name=t ! queue !  v4l2sink device=/dev/video2 t. ! queue ! v4l2sink device=/dev/video3

Then this runs on frontend :-

    ` gst-launch-1.0 v4l2src device=/dev/video2 ! videoconvert ! videoscale ! video/x-raw, width=848, height=480, framerate=30/1 ! appsink drop=1`

While this on backend

  `   cv::VideoCapture cap("v4l2src device=/dev/video3 ! videoconvert ! videoscale ! video/x-raw, width=640, height=480, framerate=30/1 ! appsink drop=1", cv::CAP_GSTREAMER);`

This is my camera supported resolution

khadas@Khadas:~$ v4l2-ctl --list-formats-ext -d /dev/video0
ioctl: VIDIOC_ENUM_FMT
    Type: Video Capture

    [0]: 'MJPG' (Motion-JPEG, compressed)
        Size: Discrete 1920x1080
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.040s (25.000 fps)
            Interval: Discrete 0.050s (20.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
        Size: Discrete 160x120
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.040s (25.000 fps)
            Interval: Discrete 0.050s (20.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
        Size: Discrete 176x144
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.040s (25.000 fps)
            Interval: Discrete 0.050s (20.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
        Size: Discrete 320x240
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.040s (25.000 fps)
            Interval: Discrete 0.050s (20.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
        Size: Discrete 352x288
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.040s (25.000 fps)
            Interval: Discrete 0.050s (20.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
        Size: Discrete 640x360
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.040s (25.000 fps)
            Interval: Discrete 0.050s (20.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
        Size: Discrete 640x480
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.040s (25.000 fps)
            Interval: Discrete 0.050s (20.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
        Size: Discrete 800x600
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.040s (25.000 fps)
            Interval: Discrete 0.050s (20.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
        Size: Discrete 848x480
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.040s (25.000 fps)
            Interval: Discrete 0.050s (20.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
        Size: Discrete 1024x768
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.040s (25.000 fps)
            Interval: Discrete 0.050s (20.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
        Size: Discrete 1280x800
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.040s (25.000 fps)
            Interval: Discrete 0.050s (20.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
        Size: Discrete 1280x720
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.040s (25.000 fps)
            Interval: Discrete 0.050s (20.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
    [1]: 'YUYV' (YUYV 4:2:2)
        Size: Discrete 1920x1080
            Interval: Discrete 0.200s (5.000 fps)
        Size: Discrete 160x120
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.040s (25.000 fps)
            Interval: Discrete 0.050s (20.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
        Size: Discrete 176x144
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.040s (25.000 fps)
            Interval: Discrete 0.050s (20.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
        Size: Discrete 320x240
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.040s (25.000 fps)
            Interval: Discrete 0.050s (20.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
        Size: Discrete 352x288
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.040s (25.000 fps)
            Interval: Discrete 0.050s (20.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
        Size: Discrete 640x360
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.040s (25.000 fps)
            Interval: Discrete 0.050s (20.000 fps)
            Interval: Discrete 0.200s (5.000 fps)
        Size: Discrete 640x480
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.040s (25.000 fps)
            Interval: Discrete 0.050s (20.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
        Size: Discrete 800x600
            Interval: Discrete 0.050s (20.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
        Size: Discrete 848x480
            Interval: Discrete 0.050s (20.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
        Size: Discrete 1024x768
            Interval: Discrete 0.100s (10.000 fps)
        Size: Discrete 1280x800
            Interval: Discrete 0.100s (10.000 fps)
        Size: Discrete 1280x720
            Interval: Discrete 0.100s (10.000 fps)

THE PROBLEM

I want to duplicate the camera stream using MJPG format as its has higher fps. I also dont know how to enable MJPG in gstreamer command.
The problem we are facing is that the camera playback is very slow (please check attached video). We thought it was CPU but CPU usage is very low (please check attached snapshot of frontend )

How to speed up camera feed ?

numbqq · December 23, 2021, 7:20am

For gstreamer, you can’t use hardware decoder so it is very slow.

Here is a demo about how to use the hardware decoder for UVC, you can check.

IGNEOUS · December 23, 2021, 7:40am

Gstreamer’s parameter will be same or different according to uvc?

rahul_s · December 23, 2021, 8:25am

How to duplicate the camera stream using uvc ?

numbqq · December 23, 2021, 8:49am

There is no hardware decoder gstreamer plugin. The amlvdec plugin only for display, you can’t fetch decoded frames from it.

numbqq · December 23, 2021, 8:49am

For hardware decoder, only support one instance.

rahul_s · December 24, 2021, 2:12am

But this won’t solve our purpose. Can you not suggest some optimized configuration with gstreamer itself. Atleast something better than the configuration we are using.
Secondly what should be the optimum temperature of the board ro avoid throttling. At what temp the throttling starts ?