Ffmpeg and hardware h264 decode

VIM4
6

From Khadas Fenix Ubuntu

khadas@Khadas:/lib/firmware$ dpkg -S /usr/bin/tee-supplicant
optee-userspace: /usr/bin/tee-supplicant
khadas@Khadas:/lib/firmware$

Optee-userspace package does not exist on Fenix Debian11 - and I cant seem to find it (or what its really called)

Just for reference… this is the output from ffpmeg on Khadas Ubuntu - Working!

khadas@Khadas:~$ sudo ffmpeg  -c:v h264_v4l2m2m -i 09.22.mp4  -f null /dev/null
ffmpeg version 4.2.7-0ubuntu0.1 Copyright (c) 2000-2022 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
  configuration: --prefix=/usr --extra-version=0ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/aarch64-linux-gnu --incdir=/usr/include/aarch64-linux-gnu --arch=arm64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '09.22.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    title           : go2rtc/1.2.0
    encoder         : Lavf58.45.100
  Duration: 00:00:09.97, start: 0.000000, bitrate: 4195 kb/s
    Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p(tv, bt470bg/bt470bg/smpte170m), 1920x1056, 4192 kb/s, 25.08 fps, 90k tbr, 90k tbn, 180k tbc (default)
    Metadata:
      handler_name    : VideoHandler
[h264_v4l2m2m @ 0x5588624650] driver 'aml-vcodec-dec' on card 'platform:amlogic'
[h264_v4l2m2m @ 0x5588624650] Using device /dev/video26
[h264_v4l2m2m @ 0x5588624650] driver 'aml-vcodec-dec' on card 'platform:amlogic'
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (h264_v4l2m2m) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
Output #0, null, to '/dev/null':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    title           : go2rtc/1.2.0
    encoder         : Lavf58.29.100
    Stream #0:0(und): Video: wrapped_avframe, nv21, 1920x1056, q=2-31, 200 kb/s, 25.08 fps, 25.08 tbn, 25.08 tbc (default)
    Metadata:
      handler_name    : VideoHandler
      encoder         : Lavc58.54.100 wrapped_avframe

frame=  107 fps= 94 q=-0.0 Lsize=N/A time=00:00:10.24 bitrate=N/A speed=8.97x
video:56kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
khadas@Khadas:~$

The below messages are not present in the Debian11 build… but are showing in the Ubuntu build

[   21.633476] [TEE] E/TC:? 00 tee_rpmb_verify_key_sync_counter:967 Verify key returning 0xffff0008
[   21.633484] [TEE] E/TA:   MM-module-name:Video firmware TA,Version:1.0.5-g3604ffd(build:958)
[   21.633488] [TEE] E/TA:   fw_check_pack_version:292 the package has 15 fws totally.
[   21.633491] [TEE] E/TA:   fw_check_pack_version:309 The TA ver is v1.0
[   21.633494] [TEE] E/TA:   fw_check_pack_version:310 The fw ver is v0.4
[   21.633497] [TEE] E/TA:   fw_data_insert:389 the fw with 332 KB will be loaded.
[   21.633500] [TEE] M/TA: TEE_Video_Load_FW success
7

Oh, do you use the original ffmpeg from our Ubuntu image ?

Yes, debian may miss some packages, we can fix it.

8

The ubuntu Image the server one downloaded directly from OOWOW… - No changes.

The debian11 one is built from Fenix (debian 11 server) - (as there is no Debian11 on OOWOW)

I moved some the firmware package from Ubuntu to debian - but got an error saying ‘ree is not ready’ and the firmware doesnt load.

9

We will fix it.

Not tested with ffmpeg before, how is the performance ?

10

From the logs, its showing for decode (my video is 10 seconds long) - I havent mapped out CPU usage yet (that will be today)

frame=  107 fps= 94 q=-0.0 Lsize=N/A time=00:00:10.24 bitrate=N/A speed=8.97x

Below is the error I get from Debian11 when I try to load firmware…

khadas@Khadas:~$ dmesg |grep TEE
[   21.340780] [TEE] E/TC:? 00 ta_open:213 ree system is not ready, ta can not be loaded.
[   21.341285] [TEE] E/LD:   init_elf:259 sys_open_ta_bin(526fc4fc-7ee6-4a12-96e3-83da9565bce8)
[   21.342374] [TEE] E/TC:? 00 init_with_ldelf:229 ldelf failed with res: 0xffff0007
khadas@Khadas:~$
11

I have patches to fix the video firmware issue, you can update the Fenix code and rebuild debian again.

12

Thank you! … compiling now.

Which build did you fix? Debian11/Minimal ? or one of the others?

13

Debian 11 server is okay.

14

Awesome! … hardware decoders are running now!

15

Hello @RichardPar

What’s your usage of ffmpeg with hardware decoding? Could you provide an example of your usage?

16

The system is being used with HomeAssistant and Frigate NVR connected to CCTV cameras. At 1920x1056 resolution H264 video stream.

5% CPU with hardware decoder…43% without hardware decoder

— Without Acceleration —

khadas@Khadas:~$ ps -p 335299 -o %cpu,%mem,cmd
%CPU %MEM CMD
43.5  1.3 ffmpeg -i rtsp://localhost:8554/garden_high_cam -f null /dev/null
khadas@Khadas:~$

— WITH acceleration —

khadas@Khadas:~$ ps -p 334316 -o %cpu,%mem,cmd
%CPU %MEM CMD
 5.0  0.7 ffmpeg -c:v h264_v4l2m2m -i rtsp://localhost:8554/garden_high_cam -f null /dev/null
khadas@Khadas:~$
1 Like
17

Would there be any change to get this working with VIM3?

I have about identical usecase as @RichardPar has.

18

I could not get it working on the VIM3 :frowning: (I have one for experiments) - I was hoping Khadas or even amlogic could get the h264_v4l2m2m driver working on the vim3

1 Like
19

Hello @RichardPar @savikko

We plan to port a new 5.15 kernel to VIM3 later, at that time it should work.

2 Likes
20

That sounds good! Will look forward for that.

21

Hi, I got a similar one trying to setup hardware decode on armbian.

[   11.647444] [TEE] E/TC:? 00 tee_rpmb_verify_key_sync_counter:967 Verify key returning 0xffff0008
[   11.647463] [TEE] E/TA:   MM-module-name:Video firmware TA,Version:1.0.5-g3604ffd(build:958)
[   11.647467] [TEE] E/TA:   load_video_fw:650 Verify failed

Any idea how I can solve this ?

22

Hello @sousmangoosta

It seems the video firmware load failed. Which Armbian you used?

@ivan.li please follow up.

23

Thanks @numbqq @ivan.li
I used that one Armbian 24.2.1 bookworm
I needed to add some packages by myself optee-userspace_0.5-202311_arm64.deband optee-video-firmware_0.4-202311_arm64.deb from https://dl.khadas.com/repos/vim4-5.15/pool/main/o/.
The firmware /lib/firmware/video/video_ucode.bin is provided by the armbian-firmware package.

Package: armbian-firmware
Version: 24.2.1
Priority: optional
Section: kernel
Maintainer: Armbian Linux <info@armbian.com>
Installed-Size: 235 MB
Provides: linux-firmware, firmware-brcm80211, firmware-ralink, firmware-samsung, firmware-realtek, armbian-firmware-full
Conflicts: linux-firmware, firmware-brcm80211, firmware-ralink, firmware-samsung, firmware-realtek, armbian-firmware-full
Armbian-Original-Hash: 1-SA6b6f-B90f5-R448a
Download-Size: 75,2 MB
APT-Manual-Installed: yes
APT-Sources: http://imola.armbian.com/apt bookworm/main arm64 Packages
Description: Armbian - Linux firmware

dpkg -L armbian-firmware |grep video_ucode.bin
/lib/firmware/video/video_ucode.bin

sha1sum /lib/firmware/video/video_ucode.bin
19d8bbabf3dd03170f55943c46074920504fa7c4  /lib/firmware/video/video_ucode.bin

systemctl status video-firmware-preload-khadas
× video-firmware-preload-khadas.service - Amlogic OPTEE Video Firmware Preload Service
     Loaded: loaded (/lib/systemd/system/video-firmware-preload-khadas.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Sat 2024-05-11 12:45:15 CEST; 1s ago
   Duration: 2.022s
    Process: 130209 ExecStart=/usr/bin/tee_preload_fw /lib/firmware/video/video_ucode.bin (code=exited, status=255/EXCEPTION)
   Main PID: 130209 (code=exited, status=255/EXCEPTION)
        CPU: 2.005s

mai 11 12:45:13 khadas-vim4 systemd[1]: Started video-firmware-preload-khadas.service - Amlogic OPTEE Video Firmware Preload Service.
mai 11 12:45:15 khadas-vim4 tee_preload_fw[130209]: fw_path = /lib/firmware/video/video_ucode.bin
mai 11 12:45:15 khadas-vim4 tee_preload_fw[130209]: MM-module-name:Video firmware CA,Version:1.0.5-g3604ffd(build:958)
mai 11 12:45:15 khadas-vim4 tee_preload_fw[130209]: TEEC_InvokeCommand failed with code 0xffff000f origin 0x4
mai 11 12:45:15 khadas-vim4 tee_preload_fw[130209]: tee preload video fw fail
mai 11 12:45:15 khadas-vim4 systemd[1]: video-firmware-preload-khadas.service: Main process exited, code=exited, status=255/EXCEPTION
mai 11 12:45:15 khadas-vim4 systemd[1]: video-firmware-preload-khadas.service: Failed with result 'exit-code'.
mai 11 12:45:15 khadas-vim4 systemd[1]: video-firmware-preload-khadas.service: Consumed 2.005s CPU time.
24

It’s ok with this one https://github.com/numbqq/fenix/blob/master/archives/hwpacks/video-firmware/Amlogic/VIM4/video/video_ucode.bin

/usr/bin/tee_preload_fw video_ucode.bin 
fw_path = video_ucode.bin
MM-module-name:Video firmware CA,Version:1.0.5-g3604ffd(build:958)
tee preload video fw ok

[74669.200349] [TEE] E/TA:   MM-module-name:Video firmware TA,Version:1.0.5-g3604ffd(build:958)
[74673.296468] [TEE] E/TA:   fw_check_pack_version:292 the package has 15 fws totally.
[74673.296498] [TEE] E/TA:   fw_check_pack_version:309 The TA ver is v1.0
[74673.296509] [TEE] E/TA:   fw_check_pack_version:310 The fw ver is v0.4
[74673.296518] [TEE] E/TA:   fw_data_insert:389 the fw with 332 KB will be loaded.
[74673.296527] [TEE] M/TA: TEE_Video_Load_FW success
25

Thanks for reporting it. We are working on it and this will be fixed in the upcoming 24.5 release

Regards
The Armbian Team

1 Like