OOWOW
Self build Fenix debian 11
FFMPEG and H264 hardware decoding - Using the VIM4 with ffmpeg, is there a hardware enabled build or a method/example to add hardware decode in to ffmpeg
Regards
Richard
I see there are some plugins… I just get errors from it…
[44953.897804] [276]: H264 post event: 1024
[44953.898722] 0: MH264 the TEE fw loading failed, err: ffff0007
[44953.898729] [276]: changes: 400
Are the TEE images for firmware missing in the Fenix builds ?
There is no hardware decoding on FFmpeg.
There is hardware decoding… it works on the Fenix/Ubuntu images,
if you use the v4l m2m decoder, which is supported on the platform.
[h264_v4l2m2m @ 0x558f973a20] Using device /dev/video26
[h264_v4l2m2m @ 0x558f973a20] driver ‘aml-vcodec-dec’ on card ‘platform:amlogic’ in mplane mode
[h264_v4l2m2m @ 0x558f973a20] requesting formats: output=H264 capture=NM21
khadas@Khadas:~$ sudo ffmpeg -c:v h264_v4l2m2m -i rtsp://127.0.0.1:8554/garden_low_cam -f null /dev/null
ffmpeg version 4.3.5-0+deb11u1 Copyright (c) 2000-2022 the FFmpeg developers
built with gcc 10 (Debian 10.2.1-6)
configuration: --prefix=/usr --extra-version=0+deb11u1 --toolchain=hardened --libdir=/usr/lib/aarch64-linux-gnu --incdir=/usr/include/aarch64-linux-gnu --arch=arm64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
libavutil 56. 51.100 / 56. 51.100
libavcodec 58. 91.100 / 58. 91.100
libavformat 58. 45.100 / 58. 45.100
libavdevice 58. 10.100 / 58. 10.100
libavfilter 7. 85.100 / 7. 85.100
libavresample 4. 0. 0 / 4. 0. 0
libswscale 5. 7.100 / 5. 7.100
libswresample 3. 7.100 / 3. 7.100
libpostproc 55. 7.100 / 55. 7.100
[rtsp @ 0x558f96f2a0] method SETUP failed: 461 Unsupported transport
Input #0, rtsp, from 'rtsp://127.0.0.1:8554/garden_low_cam':
Metadata:
title : go2rtc/1.2.0
Duration: N/A, start: 0.042000, bitrate: N/A
Stream #0:0: Video: h264 (Main), yuv420p(tv, bt470bg/bt470bg/smpte170m, progressive), 640x352, 25 fps, 25 tbr, 90k tbn, 180k tbc
[h264_v4l2m2m @ 0x558f973a20] Using device /dev/video26
[h264_v4l2m2m @ 0x558f973a20] driver 'aml-vcodec-dec' on card 'platform:amlogic' in mplane mode
[h264_v4l2m2m @ 0x558f973a20] requesting formats: output=H264 capture=NM21
Stream mapping:
Stream #0:0 -> #0:0 (h264 (h264_v4l2m2m) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
^Came= 0 fps=0.0 q=0.0 size=N/A time=-577014:32:22.77 bitrate=N/A speed=N/A
DMESG
[12026.106170] 0: MH264 the TEE fw loading failed, err: ffff0007
[12026.106182] [4]: changes: 400
[12026.106206] [4]: H264 post event: 1024
[12026.107042] 0: MH264 the TEE fw loading failed, err: ffff0007
[12026.107051] [4]: changes: 400
[12026.107075] [4]: H264 post event: 1024
[12026.107953] 0: MH264 the TEE fw loading failed, err: ffff0007
[12026.107964] [4]: changes: 400
[12026.107988] [4]: H264 post event: 1024
[12026.108507] [4]: release decoder ffffff89d9154000
[12026.108826] 0: MH264 the TEE fw loading failed, err: ffff0007
[12026.108869] [4]: changes: 400
[12026.108983] [4]: fb_map_table_clean done
[12026.109268] [4]: H264 post event: 1024
[12026.109875] 0: MH264 the TEE fw loading failed, err: ffff0007
[12026.110290] [4]: changes: 400
[12026.110471] [4]: fb_token_clean done
[12026.110687] [4]: H264 post event: 1024
[12026.111164]
==== Show mmu buffer info ========
[12026.112221] 0: MH264 the TEE fw loading failed, err: ffff0007
[12026.112474]
but when you execute the command, it says missing firmware
The Fenix VIM4 Debian11 builds are completely missing all the OPTEE utililites
From Khadas Fenix Ubuntu
khadas@Khadas:/lib/firmware$ dpkg -S /usr/bin/tee-supplicant
optee-userspace: /usr/bin/tee-supplicant
khadas@Khadas:/lib/firmware$
Optee-userspace package does not exist on Fenix Debian11 - and I cant seem to find it (or what its really called)
Just for reference… this is the output from ffpmeg on Khadas Ubuntu - Working!
khadas@Khadas:~$ sudo ffmpeg -c:v h264_v4l2m2m -i 09.22.mp4 -f null /dev/null
ffmpeg version 4.2.7-0ubuntu0.1 Copyright (c) 2000-2022 the FFmpeg developers
built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
configuration: --prefix=/usr --extra-version=0ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/aarch64-linux-gnu --incdir=/usr/include/aarch64-linux-gnu --arch=arm64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
libavutil 56. 31.100 / 56. 31.100
libavcodec 58. 54.100 / 58. 54.100
libavformat 58. 29.100 / 58. 29.100
libavdevice 58. 8.100 / 58. 8.100
libavfilter 7. 57.100 / 7. 57.100
libavresample 4. 0. 0 / 4. 0. 0
libswscale 5. 5.100 / 5. 5.100
libswresample 3. 5.100 / 3. 5.100
libpostproc 55. 5.100 / 55. 5.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '09.22.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
title : go2rtc/1.2.0
encoder : Lavf58.45.100
Duration: 00:00:09.97, start: 0.000000, bitrate: 4195 kb/s
Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p(tv, bt470bg/bt470bg/smpte170m), 1920x1056, 4192 kb/s, 25.08 fps, 90k tbr, 90k tbn, 180k tbc (default)
Metadata:
handler_name : VideoHandler
[h264_v4l2m2m @ 0x5588624650] driver 'aml-vcodec-dec' on card 'platform:amlogic'
[h264_v4l2m2m @ 0x5588624650] Using device /dev/video26
[h264_v4l2m2m @ 0x5588624650] driver 'aml-vcodec-dec' on card 'platform:amlogic'
Stream mapping:
Stream #0:0 -> #0:0 (h264 (h264_v4l2m2m) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
Output #0, null, to '/dev/null':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
title : go2rtc/1.2.0
encoder : Lavf58.29.100
Stream #0:0(und): Video: wrapped_avframe, nv21, 1920x1056, q=2-31, 200 kb/s, 25.08 fps, 25.08 tbn, 25.08 tbc (default)
Metadata:
handler_name : VideoHandler
encoder : Lavc58.54.100 wrapped_avframe
frame= 107 fps= 94 q=-0.0 Lsize=N/A time=00:00:10.24 bitrate=N/A speed=8.97x
video:56kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
khadas@Khadas:~$
The below messages are not present in the Debian11 build… but are showing in the Ubuntu build
[ 21.633476] [TEE] E/TC:? 00 tee_rpmb_verify_key_sync_counter:967 Verify key returning 0xffff0008
[ 21.633484] [TEE] E/TA: MM-module-name:Video firmware TA,Version:1.0.5-g3604ffd(build:958)
[ 21.633488] [TEE] E/TA: fw_check_pack_version:292 the package has 15 fws totally.
[ 21.633491] [TEE] E/TA: fw_check_pack_version:309 The TA ver is v1.0
[ 21.633494] [TEE] E/TA: fw_check_pack_version:310 The fw ver is v0.4
[ 21.633497] [TEE] E/TA: fw_data_insert:389 the fw with 332 KB will be loaded.
[ 21.633500] [TEE] M/TA: TEE_Video_Load_FW success
Oh, do you use the original ffmpeg from our Ubuntu image ?
Yes, debian may miss some packages, we can fix it.
The ubuntu Image the server one downloaded directly from OOWOW… - No changes.
The debian11 one is built from Fenix (debian 11 server) - (as there is no Debian11 on OOWOW)
I moved some the firmware package from Ubuntu to debian - but got an error saying ‘ree is not ready’ and the firmware doesnt load.
We will fix it.
Not tested with ffmpeg before, how is the performance ?
From the logs, its showing for decode (my video is 10 seconds long) - I havent mapped out CPU usage yet (that will be today)
frame= 107 fps= 94 q=-0.0 Lsize=N/A time=00:00:10.24 bitrate=N/A speed=8.97x
Below is the error I get from Debian11 when I try to load firmware…
khadas@Khadas:~$ dmesg |grep TEE
[ 21.340780] [TEE] E/TC:? 00 ta_open:213 ree system is not ready, ta can not be loaded.
[ 21.341285] [TEE] E/LD: init_elf:259 sys_open_ta_bin(526fc4fc-7ee6-4a12-96e3-83da9565bce8)
[ 21.342374] [TEE] E/TC:? 00 init_with_ldelf:229 ldelf failed with res: 0xffff0007
khadas@Khadas:~$
I have patches to fix the video firmware issue, you can update the Fenix code and rebuild debian again.
Thank you! … compiling now.
Which build did you fix? Debian11/Minimal ? or one of the others?
Debian 11 server is okay.
Awesome! … hardware decoders are running now!
Hello @RichardPar
What’s your usage of ffmpeg with hardware decoding? Could you provide an example of your usage?
The system is being used with HomeAssistant and Frigate NVR connected to CCTV cameras. At 1920x1056 resolution H264 video stream.
5% CPU with hardware decoder…43% without hardware decoder
— Without Acceleration —
khadas@Khadas:~$ ps -p 335299 -o %cpu,%mem,cmd
%CPU %MEM CMD
43.5 1.3 ffmpeg -i rtsp://localhost:8554/garden_high_cam -f null /dev/null
khadas@Khadas:~$
— WITH acceleration —
khadas@Khadas:~$ ps -p 334316 -o %cpu,%mem,cmd
%CPU %MEM CMD
5.0 0.7 ffmpeg -c:v h264_v4l2m2m -i rtsp://localhost:8554/garden_high_cam -f null /dev/null
khadas@Khadas:~$
Would there be any change to get this working with VIM3?
I have about identical usecase as @RichardPar has.
I could not get it working on the VIM3 
(I have one for experiments) - I was hoping Khadas or even amlogic could get the h264_v4l2m2m driver working on the vim3
Hello @RichardPar @savikko
We plan to port a new 5.15 kernel to VIM3 later, at that time it should work.
That sounds good! Will look forward for that.