Qt/gstreamer app for H264 hardware decoding

VIM1
1

Which system do you use? Android, Ubuntu, OOWOW or others?

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.6 LTS
Release:	20.04
Codename:	focal

GStreamer : 1.16.3 installed using apt. I also tried using the latest version 1.23, compiled from sources.

Qt v5 installed using apt. If necessary, I can also use Qt v6.

Which version of system do you use? Khadas official images, self built images, or others?

Custom version generated from your fenix tool.

Please describe your issue below:

Hi,

I work on a VIM1 card with a Ubuntu 20.04 generated from your fenix application. I would like to create a custom video player using the Qt/QML library. This player must only display H264 videos (with related sound) into a Qt window and the QML part will be used to overlay some custom data (text, graphicsā€¦).

I try to do this using a GStreamer pipeline embedded into a Qt application. I can display a video into a Qt window using a 100% CPU pipeline like this :

gst-launch-1.0 \
    filesrc location=/home/user/videos/big_buck_bunny_1080p_h264.mov ! \
    qtdemux name=demux \
        demux.video_0 ! queue ! h264parse ! avdec_h264 ! glupload ! glimagesink \
        demux.audio_0 ! queue ! avdec_aac ! audioconvert ! autoaudiosink

but there are too much dropped frames. If I try to use your amlvdec plugin - to use hardware decoding, quality is good but display is done outside my Qt window.

According to this post, it seems that this amlvdec plugin can not be used directly into a GStreamer pipeline because it creates its own specific video context, which can not be chained to a generic sink (used to send frames into a QWidget). And in this post numbqq said that Gstreamer hardware decoding plugin doesnā€™t work well with 4.9 kernel.

Do you think it is possible to create a custom amlvdec plugin, inspired from yours, to decode H264 videos using hardware, but where decoded frames will be sent to a generic sink like a glimagesink or a ximagesinkā€¦ or your hardware decoding library is too specific/closed to be used in this generic context ?

If itā€™s possible, can you give me some advice or guidelines ? I saw there are lots of projects in the Khadas's github or numbqq's github, but I donā€™t know which projects/drivers I should use.

Using a newer kernel (5.xx, for example) on a VIM1, does your GStreamer plugin work better and this can solve my problem ?

Finally, if the GStreamer approach is not a good idea, do you have a better suggestion ?

Thank you for your advice.
If necessary, I can post a simple C++/Qt example including the previous GStreamer pipeline.

2

Hello @mdr.ai

For hardware decoding, you can check here: VIM1 Hardware Decoding [Khadas Docs]

3

Hi @numbqq,
thank you for your quick reply.

I have already read this doc. Unfortunately, I was not able to compile given projects due to missing related libraries. I will try again.

  1. Can you confirm that the amlvdec plugin can not be linked to a generic sink (glimagesink, xvimagesink) because it creates its own specific video context ?

  2. Iā€™ve also tried this gstreamer pipeline, on a Ubuntu 22.04.4 (kernel 6.2.0) generated by your fenix tool :

gst-launch-1.0 \
    filesrc location=/home/user/videos/big_buck_bunny_1080p_h264.mov ! \
    qtdemux name=demux \
        demux.video_0 ! queue ! h264parse ! v4l2h264dec ! xvimagesink

and results are better.
Can you also confirm that the v4l2h264dec plugin uses hardware decoding ?

Thank you.

4

No, it is not supported.

5

The ā€œbetween a rock and a hard placeā€ problem is: the media codec capabilities in the upstream kernel are not fully developed, and while everything in the downstream vendor kernel ā€œworksā€ with Amlogic provided tools; most of those tools are outdated and not what people are trying to use (and ā€œstandardsā€ is a dirty word).

Current versions of Gstreamer are focussed on upstream V4L2 capabilities and those are not suported in Amlogic vendor kernels. V4L2 support is allegedly better in the latest kernels, but those donā€™t support GXL boards because maintaining backwards compatibility is something Amlogicā€™s engineers also donā€™t understand (they are not alone; most SoC vendors have a project-based approach).

Iā€™d suggest you have a look at upstream kernels like Linux 6.8.0 as Amlogic GXL boards like VIM1 are well supported, and while the media codecs are overall not in brilliant shape (still in staging, some conformance issues) the H264 codec is the most-finished of them and generally works well (my own use-case is Kodi via ffmpeg not Gstreamer, but it works good). So if H264 is the only decoder important to your app, it could work, and if you needed to chat with Gstreamer devs to figure something out you are not using the vendor codebase which they have no interest in.

1 Like
6

Hi @chewitt,

thank you for your suggestion. Indeed, Iā€™ve tried with a V4L2 plugin for gstreamer onto a more recent kernel and results are better. Not perfect but betterā€¦
But I donā€™t know if I can upgrade the Ubuntuā€™s version (there are other devices plugged on the board).

7

Hi @numbqq,

as you suggested, I have compiled the esplayer example referenced in the documentation page (I have also installed the libamcodec and aml_libion projects first, to be able to compile the esplayer project). Unfortunately, esplayer doesnā€™t work correctly. Everything was fine during the compilation process but when I run this program, for example using this command line :

./esplayer /home/user/videos/bbb_sunflower_1080p_30fps_normal.mp4 1920 1080 30 2

the screen shows incoherent colors and freezes after a few seconds. I must do a Ctlr+C to stop application. Display is not restored correctly (wrong colors). Command line output :

*********CODEC PLAYER DEMO************

file /home/user/videos/bbb_sunflower_1080p_30fps_normal.mp4 to be played
codec_init amstream version : 2.0
video codec ok!
^CGet signum=2

Any ideas ? Does this esplayer program should work correctly or is it an outdated project ? Should I try the libplayer project ?
Thank you.

EDIT : I have also tried the ionvideo branch of the aml_hardware_decode_demo but the program ionplayer crashes with a Segmentation faultā€¦

8

Can you be specific/detailed about the issues?

9

Hi @chewitt,

using the gstreamer pipeline given previously on a Ubuntu 22.04.4 (kernel 6.2.0), results are good (but not as good as using hardware decoding). But currently, we have hundreds of card in production with a Ubuntu 20.04 (kernel 4.9) so Iā€™m not sure that it is possible to upgrade OS.

Thatā€™s why my first idea is to create a new gstreamer plugin, based on Khadasā€™s projects, to decode H264 frames using hardware but with no video output integrated, and running on a Ubuntu 20.04 (kernel 4.9).

10

@numbqq : from your projects aml_hardware_decode_demo, libplayer or libamcodec, do you think it is possible to extract only the hardware decoding part (= without output) of your programs to integrate it into a custom gstreamer pluginā€¦ or decoding part and output part are too much linked to split them ?

Thank youā€¦

11

Itā€™s not impossible but the amount of testing (time) required to execute the change with a level of confidence probably exceeds the time needed to manually reminage and redploy all the boards ā€¦ so not something to tackle :slight_smile:

12

Hi @numbqq,

I have compiled the amlvdec plugin for gstreamer from your projects gst-aml-plugins1 and libamcodec. But, as I have previously explained, when I try to decode a H264 video, output is always done, even if I use a fakesink as output of my gstreamer pipeline. Inside the code, there is a call to the codec_init() method, which initialize output display.

Do you think it is possible to disable the code used to display decoded frames and to only keep decoding part or these two parts (decoding and displaying) are too much dependent together to be split ? If it is possible, do you have some tips or some docs related to this code ?

Thank you for your helpā€¦

13

Hello @mdr.ai

This plugin only for display and you canā€™t get the decoded frames.

14

Hi,

ok, now I have understood that the gstreamer plugin is not a good way for my problem. To recall, I would like to decode H264 video frames using the VPU, to send decoded frames into a Qt Widget. So, Iā€™ve tried to compile/use projects given by numbqqā€™s link VIM1 Hardware Decoding [Khadas Docs]. Unfortunately without success.

I was able to compile the aml_hardware_decode_demo (with minor modifications into the Makefile) but if I run :

  • the master branch, black screen or randomized colors are shown
$ esplayer /home/user/videos/bbb_sunflower_1080p_30fps_normal.mp4 1920 1080 30 2

*********CODEC PLAYER DEMO************

file /home/user/videos/bbb_sunflower_1080p_30fps_normal.mp4 to be played
codec_init amstream version : 2.0
video codec ok!
$
  • the ionvideo branch, there is a segmentation fault
$ ./ionplayer /home/user/videos/bbb_sunflower_1080p_30fps_normal.mp4 1920 1080 30 2

*********CODEC PLAYER DEMO************

file /home/user/videos/bbb_sunflower_1080p_30fps_normal.mp4 to be played
codec_init amstream version : 2.0
video codec ok!
Segmentation fault
$

Does someone can compile and use these programs using an Ubuntu 20.04 ? I have tried using videos coming from the Big Buck Bunny project (with different resolutions) but with the same result all the time.

I have also tried to compile the libplayer project but I have errors during the processing of the amffmpeg subdirectory: lot of ā€˜undefined referenceā€™ā€¦ I donā€™t have investigate anymore. Should I do ?

@numbqq : do you have an english version of the user doc put inside the libplayer repository ? Does this documentation can help me anymore ?

Thank youā€¦

15

Could you tell us which image you used? We will try to check on our side.

16

For example, with the Big Buck Bunny video, the esplayer program displays frames with random colors and the ionplayer program returns a segmentation fault (full command lines and outputs for this video are given in my previous post).

Should I install some specific libraries before, like libamcodec or aml_libion ? I have tried with different combinations: with or without these libraries compiled from sources but without successā€¦

Thank you

17

Hello @mdr.ai

Could you tell us which image you used? @ivan.li will follow up to check this issue.

18

Thank you @numbqq and hello @ivan.liā€¦

I used an Ubuntu 20.04 generated by your fenix tool. Custom configuration has been made, principally because specific hardwares are plugged to the VIM1 for our project, but nothing related to the video (it is mainly for networking).

$ uname -a
Linux MyCard 4.9.241 #2 SMP PREEMPT Mon Feb 12 14:25:09 CET 2024 aarch64 aarch64 aarch64 GNU/Linux

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.6 LTS
Release:	20.04
Codename:	focal

I also tried to use these programs with an Ubuntu 22.04, also generated with your fenix tool and without custom configuration. In this case, I have the following issue:

*********CODEC PLAYER DEMO************

file /home/khadas/app/videos/bbb_sunflower_1080p_30fps_normal.mp4 to be played
OPEN es DEVICE
Init [/dev/amstream_vbuf] failed,ret = -1 error=2 retry_open!
retry_open [/dev/amstream_vbuf] failed,ret = -1 error=2 used_times=1000*10(ms)
Error=1 errno=2 : No such file or directory,func=codec_video_es_init,line=321
codec init failed, ret=-0xfefffff4

In fact, there is no /dev/am* filesā€¦ Should I install or configure something ?

$ uname -a
Linux Khadas 6.2.0 #1.5.2 SMP Mon Mar 11 15:38:25 CET 2024 aarch64 aarch64 aarch64 GNU/Linux

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.4 LTS
Release:	22.04
Codename:	jammy

If it is possible, priority should be given to the Ubuntu 20.04 because all our custom programs are already tested with it.

Thank you very much for your attention.

19

@mdr.ai

esplayer can only play raw video stream files (switch to framrbuffer mode: Ctrl+Alt+F1)

$ ./esplayer 1080p.h265 1920 1080 25 11
20

Thank you @ivan.li for your quick reply,

esplayer can only play raw video stream files

in fact, I have missed this information. So, I have converted my original video (MP4) using the following command:

$ ffmpeg -i my_input_video.mp4 -vcodec copy -an -f h264 my_output_video.h264

and now the esplayer works correctly !

I have still two questions:

  1. my initial problem was to use only your hardware decoding codeā€¦ do you know if I can extract this part (= hardware decoding) from your esplayer code or, like your gstreamer plugin, in this esplayer program, the hardware decoding code and the video display code are too much linked to be split ? If it is possible, I will continue to analyse your codeā€¦

  2. In the same github repository aml_hardware_decode_demo, there is a ionvideo branchā€¦ Is this program interesting for my problem ? and can you explain me what does your libion do ? Because I tried the ionplayer program compiled from this branch but there is a segmentation fault (even if I use the raw video stream as previous)ā€¦ but perhaps the previous point 1) will be enough for me

Thank you