GStreamer HW decoder

Which Khadas SBC do you use?

VIM3 Pro

Which system do you use? Android, Ubuntu, OOWOW or others?


Which version of system do you use? Khadas official images, self built images, or others?


Please describe your issue below:

Hello @numbqq,

We found that Amlogic decoder/encoder have been installed with Gstreamer in the host, but we’d like to use the hw plugins with gstreamer in OpenCV. If we reinstall gstreamer using apt to build with OpenCV, we can’t find the hw plugins, we think we should do something from souece code. Do you have information on how to use the hw plugins with gstreamer in OpneCV?

In addition, can we import some hw elements from host to use hw plugins in a docker container?

Thanks a lot,

Did you find a solution for it? I am trying to do the same thing. Thanks!

No, I haven’t found a solution.

@numbqq Could you please help us? Thanks

1 Like

Hello @JJ1997

Do you want to get the decoded frames with Gstreamer? But I’m afried to tell you that current plugin doesn’t support it, so even you can use it with OpenCV you will also can’t get the frames.

We are now working with the 5.15 kernel for VIM3, with this version of kernel it will support this feature.

Hello. I am trying to find a way to decode h264, h265 videostreams on the gpu. with ubuntu. I am not good with c and c++. I was hoping to find a way to do it using ffmpeg, gstreamer etc. Just need to decode and encode them for ip camera aplications. On a python script. Thanks for commenting about the new kernel.

Hello @numbqq, in fact, I have successfully used default h264 plugins (avdev_h264) of Gstreamer to decode and get the video stream in python, but I don’t know how to find and use the hardware decoders that come with vim3 Amlogic. I can’t find them after reinstalling Gstreamer to use with OpenCV. Do you have any information about how to install Gstreamer to find the hw decoders? Or we should also wait the new kernel to find the decoders?

Hello @Berk_Guvercin

We suggest you to wait for the new 5.15 kernel in Q3, it will have better support for hardware decoding & encoding.

Hello @JJ1997

Gstreamer hardware decoding plugin doesn’t work well with 4.9 kernel, you can wait for our new 5.15 kernel in Q3. Or you can try to use hardware decoding with other api: VIM3/3L Hardware Decoding [Khadas Docs]

1 Like

Thank you for replying. I am currently designing systems that work with ip cameras. Thus need maximum hardware utilization due to limited resources. Thank you for the reply. I will push them to the future development and develop other parts for now! Would love to have Khadas as a hardware provider of our new startup!

Hello @numbqq, we are testing gstreamer hardware decoder on Ubuntu 5.15 kernel on vim3. When I run the gstreamer decoding using v4l2h264dec plugin in python, we have the following warning:

However, we don’t have this issue when using the avdec_h264 plugin (software decoder).

I saw some discussions of a similar issue and it seems to be related to some specfic versions of gstreamer: v4l2h264dec0: Too old frames, bug in decoder -- please file a bug: NEW DETAILS (#1103) · Issues · GStreamer / gstreamer · GitLab.

Do you have any solutions or ideas ? Thank you in advance for your reply.

Hello @JJ1997

Which image you used? Could you provide the clear version number?

Could you share your source code for us to test?

BTW, could you reproduce this issue with the gstreamer command line pipe?

Hello @numbqq, I build the latest image of server from fenix.

== ENV CONFIG =======================

I reinstalled gstreamer to build opencv :

sudo apt remove libgstreamer-plugins-base1.0-0 libgstreamer-gl1.0-0 gir1.2-gst-plugins-base-1.0 libgstreamer1.0-0 gir1.2-gstreamer-1.0

sudo apt-get install libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev gstreamer1.0-libav gstreamer1.0-plugins-bad gstreamer1.0-tools gstreamer1.0-plugins-base gstreamer1.0-alsa gstreamer1.0-plugins-good libconfig++9v5 libconfig9

We want to run gstreamer to decode rtsp H.264 stream, this is our command:

gst-launch-1.0 rtspsrc location=rtsp://XXX ! rtph264depay ! h264parse ! v4l2h264dec ! queue ! videoconvert ! appsink

We encountered a new problem. The program runs but is killed after a short while (random time).

Setting pipeline to PAUSED …
Pipeline is live and does not need PREROLL …
Progress: (open) Opening Stream
Pipeline is PREROLLED …
Prerolled, waiting for progress to finish…
Progress: (connect) Connecting to rtsp://XXX
Progress: (open) Retrieving server options
Progress: (open) Retrieving media info
Progress: (request) SETUP stream 0
Progress: (open) Opened Stream
Setting pipeline to PLAYING …
New clock: GstSystemClock
Progress: (request) Sending PLAY request
Redistribute latency…
Progress: (request) Sending PLAY request
Redistribute latency…
Progress: (request) Sent PLAY request
Redistribute latency…
Redistribute latency…
(gst-launch-1.0:200956): GStreamer-CRITICAL **: 08:59:58.803: gst_value_collect_fraction: assertion ‘collect_values[1].v_int != 0’ failed
(gst-launch-1.0:200956): GStreamer-CRITICAL **: 08:59:58.803: passed ‘0’ as denominator for `GstFraction’
Redistribute latency…
Redistribute latency…
0:00:42.2 / 99:99:99.

If I run in opencv python, it will crash if using in thread to read 2 cameras. But the sw decoder always works well in thread.

I think maybe we didn’t build a good environment gstreamer. Could you give us more information about the hw decoder dependencies and its libraries ? To use gstreamer in opencv python, we had to reinstall gstreamer, are there any special requirements for the installation methode or version? Thanks a lot!

1 Like

Hello @numbqq, do you have any information about this issue? Could you give us more information about the hw decoder dependencies and its libraries to rebuild it for integration with opencv? Thanks a lot!

@numbqq , the hw decoder works with a single camera, but it crashes with 2 threads of 2 cameras. (the sw decoder works for both). I found some errors of memory in the log:

It looks like it might be related to CMA? Do you have any ideas about these errors?

Thanks a lot

I added a 1 second sleep between the 2 threads or lock for cv2.VideoCapture, it can work. With sw decoder, we don’t need this.

Can you try this on command line whit the gstreamer pipe to check whether has the same issue?