Ffmpeg and hardware h264 decode

Which system do you use? Android, Ubuntu, OOWOW or others?

OOWOW

Which version of system do you use? Khadas official images, self built images, or others?

Self build Fenix debian 11

Please describe your issue below:

FFMPEG and H264 hardware decoding - Using the VIM4 with ffmpeg, is there a hardware enabled build or a method/example to add hardware decode in to ffmpeg

Regards
Richard

I see there are some plugins… I just get errors from it…

[44953.897804] [276]: H264 post event: 1024
[44953.898722] 0: MH264 the TEE fw loading failed, err: ffff0007
[44953.898729] [276]: changes: 400

Are the TEE images for firmware missing in the Fenix builds ?

There is no hardware decoding on FFmpeg.

There is hardware decoding… it works on the Fenix/Ubuntu images,

if you use the v4l m2m decoder, which is supported on the platform.

[h264_v4l2m2m @ 0x558f973a20] Using device /dev/video26
[h264_v4l2m2m @ 0x558f973a20] driver ‘aml-vcodec-dec’ on card ‘platform:amlogic’ in mplane mode
[h264_v4l2m2m @ 0x558f973a20] requesting formats: output=H264 capture=NM21

khadas@Khadas:~$  sudo ffmpeg  -c:v h264_v4l2m2m -i rtsp://127.0.0.1:8554/garden_low_cam  -f null /dev/null


ffmpeg version 4.3.5-0+deb11u1 Copyright (c) 2000-2022 the FFmpeg developers
  built with gcc 10 (Debian 10.2.1-6)
  configuration: --prefix=/usr --extra-version=0+deb11u1 --toolchain=hardened --libdir=/usr/lib/aarch64-linux-gnu --incdir=/usr/include/aarch64-linux-gnu --arch=arm64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
[rtsp @ 0x558f96f2a0] method SETUP failed: 461 Unsupported transport
Input #0, rtsp, from 'rtsp://127.0.0.1:8554/garden_low_cam':
  Metadata:
    title           : go2rtc/1.2.0
  Duration: N/A, start: 0.042000, bitrate: N/A
    Stream #0:0: Video: h264 (Main), yuv420p(tv, bt470bg/bt470bg/smpte170m, progressive), 640x352, 25 fps, 25 tbr, 90k tbn, 180k tbc
[h264_v4l2m2m @ 0x558f973a20] Using device /dev/video26
[h264_v4l2m2m @ 0x558f973a20] driver 'aml-vcodec-dec' on card 'platform:amlogic' in mplane mode
[h264_v4l2m2m @ 0x558f973a20] requesting formats: output=H264 capture=NM21
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (h264_v4l2m2m) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
^Came=    0 fps=0.0 q=0.0 size=N/A time=-577014:32:22.77 bitrate=N/A speed=N/A

DMESG

[12026.106170] 0: MH264 the TEE fw loading failed, err: ffff0007
[12026.106182] [4]: changes: 400
[12026.106206] [4]: H264 post event: 1024
[12026.107042] 0: MH264 the TEE fw loading failed, err: ffff0007
[12026.107051] [4]: changes: 400
[12026.107075] [4]: H264 post event: 1024
[12026.107953] 0: MH264 the TEE fw loading failed, err: ffff0007
[12026.107964] [4]: changes: 400
[12026.107988] [4]: H264 post event: 1024
[12026.108507] [4]: release decoder ffffff89d9154000
[12026.108826] 0: MH264 the TEE fw loading failed, err: ffff0007
[12026.108869] [4]: changes: 400
[12026.108983] [4]: fb_map_table_clean done
[12026.109268] [4]: H264 post event: 1024
[12026.109875] 0: MH264 the TEE fw loading failed, err: ffff0007
[12026.110290] [4]: changes: 400
[12026.110471] [4]: fb_token_clean done
[12026.110687] [4]: H264 post event: 1024
[12026.111164]
               ==== Show mmu buffer info ========
[12026.112221] 0: MH264 the TEE fw loading failed, err: ffff0007
[12026.112474]

but when you execute the command, it says missing firmware

The Fenix VIM4 Debian11 builds are completely missing all the OPTEE utililites

From Khadas Fenix Ubuntu

khadas@Khadas:/lib/firmware$ dpkg -S /usr/bin/tee-supplicant
optee-userspace: /usr/bin/tee-supplicant
khadas@Khadas:/lib/firmware$

Optee-userspace package does not exist on Fenix Debian11 - and I cant seem to find it (or what its really called)

Just for reference… this is the output from ffpmeg on Khadas Ubuntu - Working!

khadas@Khadas:~$ sudo ffmpeg  -c:v h264_v4l2m2m -i 09.22.mp4  -f null /dev/null
ffmpeg version 4.2.7-0ubuntu0.1 Copyright (c) 2000-2022 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
  configuration: --prefix=/usr --extra-version=0ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/aarch64-linux-gnu --incdir=/usr/include/aarch64-linux-gnu --arch=arm64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '09.22.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    title           : go2rtc/1.2.0
    encoder         : Lavf58.45.100
  Duration: 00:00:09.97, start: 0.000000, bitrate: 4195 kb/s
    Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p(tv, bt470bg/bt470bg/smpte170m), 1920x1056, 4192 kb/s, 25.08 fps, 90k tbr, 90k tbn, 180k tbc (default)
    Metadata:
      handler_name    : VideoHandler
[h264_v4l2m2m @ 0x5588624650] driver 'aml-vcodec-dec' on card 'platform:amlogic'
[h264_v4l2m2m @ 0x5588624650] Using device /dev/video26
[h264_v4l2m2m @ 0x5588624650] driver 'aml-vcodec-dec' on card 'platform:amlogic'
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (h264_v4l2m2m) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
Output #0, null, to '/dev/null':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    title           : go2rtc/1.2.0
    encoder         : Lavf58.29.100
    Stream #0:0(und): Video: wrapped_avframe, nv21, 1920x1056, q=2-31, 200 kb/s, 25.08 fps, 25.08 tbn, 25.08 tbc (default)
    Metadata:
      handler_name    : VideoHandler
      encoder         : Lavc58.54.100 wrapped_avframe

frame=  107 fps= 94 q=-0.0 Lsize=N/A time=00:00:10.24 bitrate=N/A speed=8.97x
video:56kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
khadas@Khadas:~$

The below messages are not present in the Debian11 build… but are showing in the Ubuntu build

[   21.633476] [TEE] E/TC:? 00 tee_rpmb_verify_key_sync_counter:967 Verify key returning 0xffff0008
[   21.633484] [TEE] E/TA:   MM-module-name:Video firmware TA,Version:1.0.5-g3604ffd(build:958)
[   21.633488] [TEE] E/TA:   fw_check_pack_version:292 the package has 15 fws totally.
[   21.633491] [TEE] E/TA:   fw_check_pack_version:309 The TA ver is v1.0
[   21.633494] [TEE] E/TA:   fw_check_pack_version:310 The fw ver is v0.4
[   21.633497] [TEE] E/TA:   fw_data_insert:389 the fw with 332 KB will be loaded.
[   21.633500] [TEE] M/TA: TEE_Video_Load_FW success

Oh, do you use the original ffmpeg from our Ubuntu image ?

Yes, debian may miss some packages, we can fix it.

The ubuntu Image the server one downloaded directly from OOWOW… - No changes.

The debian11 one is built from Fenix (debian 11 server) - (as there is no Debian11 on OOWOW)

I moved some the firmware package from Ubuntu to debian - but got an error saying ‘ree is not ready’ and the firmware doesnt load.

We will fix it.

Not tested with ffmpeg before, how is the performance ?

From the logs, its showing for decode (my video is 10 seconds long) - I havent mapped out CPU usage yet (that will be today)

frame=  107 fps= 94 q=-0.0 Lsize=N/A time=00:00:10.24 bitrate=N/A speed=8.97x

Below is the error I get from Debian11 when I try to load firmware…

khadas@Khadas:~$ dmesg |grep TEE
[   21.340780] [TEE] E/TC:? 00 ta_open:213 ree system is not ready, ta can not be loaded.
[   21.341285] [TEE] E/LD:   init_elf:259 sys_open_ta_bin(526fc4fc-7ee6-4a12-96e3-83da9565bce8)
[   21.342374] [TEE] E/TC:? 00 init_with_ldelf:229 ldelf failed with res: 0xffff0007
khadas@Khadas:~$

I have patches to fix the video firmware issue, you can update the Fenix code and rebuild debian again.

Thank you! … compiling now.

Which build did you fix? Debian11/Minimal ? or one of the others?

Debian 11 server is okay.

Awesome! … hardware decoders are running now!

Hello @RichardPar

What’s your usage of ffmpeg with hardware decoding? Could you provide an example of your usage?

The system is being used with HomeAssistant and Frigate NVR connected to CCTV cameras. At 1920x1056 resolution H264 video stream.

5% CPU with hardware decoder…43% without hardware decoder

— Without Acceleration —

khadas@Khadas:~$ ps -p 335299 -o %cpu,%mem,cmd
%CPU %MEM CMD
43.5  1.3 ffmpeg -i rtsp://localhost:8554/garden_high_cam -f null /dev/null
khadas@Khadas:~$

— WITH acceleration —

khadas@Khadas:~$ ps -p 334316 -o %cpu,%mem,cmd
%CPU %MEM CMD
 5.0  0.7 ffmpeg -c:v h264_v4l2m2m -i rtsp://localhost:8554/garden_high_cam -f null /dev/null
khadas@Khadas:~$
1 Like

Would there be any change to get this working with VIM3?

I have about identical usecase as @RichardPar has.

I could not get it working on the VIM3 :frowning: (I have one for experiments) - I was hoping Khadas or even amlogic could get the h264_v4l2m2m driver working on the vim3

1 Like

Hello @RichardPar @savikko

We plan to port a new 5.15 kernel to VIM3 later, at that time it should work.

2 Likes

That sounds good! Will look forward for that.