FFMpeg Hardware Transcoding on VIM3?

VIM3
1

Hi all, I’ve looked around for some pointers to enable HW transcoding on the VIM3 using AML, but only found dead or old threads. Does anyone @ khadas have recommendations for compiling ffmpeg with HW encoding support?

Relevant links:

2

Nothing? Even a “no this doesn’t exist and won’t ever” would be nice

3

bought 2 vim3 pros recently , even the official roms have poor gpu performance (Desktop and Web) , i was getting 100% frame drops on youtube (4K), i think it will take aleast 6 months to get a decent rom with GPU HW support.

2 Likes
4

HW acceleration is a pain, I posted 2 posts about it and got 0 response.

But at least for videos on Youtube, you can try using Firefox. It gives me far better frame rate than chromium.

5

For hw acc we can only use android, or libreelec or coreelec.

Linux de dont have good hw acc yet. It won’t be stable until panfrost is stable for all the codecs.

6

Linux mainline kernel has (work in progress) support for hardware decoding, but encoding is not on any roadmap that I’m aware of. I also can’t comment on the BSP kernel(s) as I don’t use them. NB: Encoding/Transcoding also requires support in userspace (e.g. ffmpeg) and as a broad rule the best transcoding support is found on Intel chips.

7

Hi chewitt,

I see you have added vdec for meson in your amlogic/5.4/integ branch.

I would like to know which commit should I use to built a kernel with panfrost and audio for vim3?

We have tried panfrost for rk3399 and it seems so work quite well. Not sure if it would work the same way for aml soc.

Please advice. Thanks

8

The vdec commits are in limbo at the moment. Upstream APIs have evolved since the original work that Maxime did and this necessitates rework, hence the two commits submitted to the linux-amlogic mailing list in October to resolve some API compliance issues and add H264 support. It’s not clear to me if other codecs (not yet submitted to the list but available if you know where to look) also need rework or can be used as-is. The upstream firmware set also needs revision and has some known issues. On the ffmpeg side the existing stateful support is also outdated and needs rework to handle the revised vdec code. The current challenge is that Maxime is MIA for personal reasons, so i’m not getting answers to questions and the userspace rework should really start with the updated kernel vdec code and firmware. I’m also submerged in a work project at the moment so I haven’t had time to do much experimenting to self-answer questions, so I’m kind of waiting for 5.4 to ship and hopefully a clearer path to emerge. I may also skip 5.4 and go directly to 5.5-rc kernels in order to reduce the patch count I’m working with.

Panfrost for T860 doesn’t require anything more than CONFIG_DRM_PANFROST=m with any 5.4-rc kernel but for userspace it’s best to track mesa master as development continues at pace. The T820 in the VIM2 is less well tested compared to the T860 but works fine for Kodi - I can’t speak for wider desktop use. There is currently no mesa userspace support for bifrost so it’s not an option for VIM3. NB: On the kernel side the panfrost driver will match a bifrost device-tree node and load, which will interfere with the mali_kbase driver. So in LE we package a kernel defconfig with both lima/panfrost enabled to support GX hardware and then sed/disable the modules in the defconfig at build-time if the target device is G12.

9

Thanks for the details reply.

For Mesa were using mesa-git which is the 19.3-devel branch, so were able to get somewhat stable panfrost for other devices.

In 5.4-rc we were not able to get audio neither in mainline and nor in your branch, maybe we used wrong branch.

It would be helpful if you can advice which branch to use at the moment which can give us audio and panfrost for vim3. Also which defconf to be used along with it.
Will try to build one tonight and test it.

Thanks

10

@chewitt
I see that kernel 5.4.0 release have panfrost support for AML devices, I see Mali450 and Mali-G52 support is present.
We tried it but still panfrost driver is not detected. We’re not using fbturbo so it is on modesetting.

Same kernel we use on rk3399 PineBookPro and it works on panfrost and we’re getting good feedback on it, I am aware that rockchip have better support in that side but my question is regarding 5.4.0 which have support for AML with panfrost. So what is still needing to make panfrost work on Vim3 for testing purpose only.

I don’t have much knowledge on why panfrost or lima is not working on Vim3 and Vim1. Not to forget we’re using the Mesa-git which is the latest RC version of Mesa.

Thanks.

11

Mainline support Lima and Panfrost. Lima supports Utgard GPUs only … Mali 400/450/470. Panfrost supports Midgard GPUs … Amlogic uses T820 and RK uses T760/T860. There is no support for Bifrost GPUs at this time (G31/G52 etc.) although the panfrost kernel driver has some device ID placeholders so it might claim to see the chip, but that’s all it can do. I’m not aware of the Panfrost developers doing any work on Bifrost yet - but their stated goal is to get Midgard support into a solid state before they think about expanding coverage so that’s expected.

Applications like Kodi are linked against libgbm which is provided differently for kbase+blob and kernel+mesa; hence why LE needs two separate Amlogic images for GX (using the FOSS drivers in the kernel) and G12 devices (using kbase/blobs). The alternative route is to support VIM1/VIM3 and RK devices with blobs, and VIM2 with panfrost. The holy grail to make single-image nirvabna possible is the T820 blob. It exists (i’ve had versions for some time) but it’s not from Amlogic and it’s not clearly redistributable. I’m still working on the source to get them released, but it’s taking time and I wouldn’t be surprised if panfrost evolves support for Bifrost first.

TL/DR; it’s not possible to use panfrost/mesa with VIM3/VIM3L, only VIM2, and with lima supporting the VIM1.

3 Likes
12

About H.264 hardware encoding, I have been hacking c2enc/c2_vpcodec to work on VIM3. I could get some of the frame back. The problem is that the frame is corrupted more than half of the time ( for video size of 320x320). If I feed the same video in, I would always get the same size of the output from kernel, but if I look in the output when it’s corrupted, there appears to be chucks of corrupt frames. ( usually, the corrupt sections would be 0x00 in 16 or 32 bytes alignment to 16 bytes) When the NAL return is not corrupted, the frame could be decoded correctly.

Strangely enough, if I use taskset and run on core 0 or 1, the frame corrupts more. I think there is something wrong with either the bus setting or the firmware. It looks like the cache from CPU is flushing while encoder is writing back to the memory. I changed the memory allocation, but it does not seem to help. Changing the clocks of the encoder makes the corrupts frame behavior change, but it’s still corrupted.

I’m on 4.9 from Khadas. If you have any suggestion of what I should try, please let me know.

Thanks!!

13

Hello,
I finally got it to work realiably. What need to be done is dma_map_single and dma_unmap_single. I also changed the allocator to dma_alloc_coherent/dma_free_coherent.

I may post the encoder later. Right now, it’s only working with intra frame.

Bests,

2 Likes
14

@pitchaya can you share your code\repo?
I’m also interested in HW encoding.

15

Let me clean up my code a little bit, and I’ll upload somewhere in the next few days. I assume that you can compile your own kernel. I’m using 4.9. The mainline is probably working, but I have not tested.

1 Like
16

mainline has no hardware encoding at this time, only decoding.

17

Hi, I’m a bit confused: is Vim3 (4.9 image from Khadas) able to decode in hardware H.264 / H.265 videos? If yes, does it support any profiles? And what’s the way to achieve it?

18

Hey,
Here is the library. https://github.com/pitchaya/c2_vpcodec
You can use this updated library with a slightly modified original code from https://github.com/OtherCrashOverride/c2enc
If you have problems making their code to run, I can send you my version.

The encoder come with the VIM3 kernel is a bit broken. You may have to copy


to your kernel and recompile the kernel module. (replacing this file ./drivers/amlogic/media_modules/frame_sink/encoder/h264/encoder.c ) Note that my version will allocate about 280MB of data. I can probably reduce the memory usage to 80MB, but it is not my priority.

If you have trouble compiling, I can provide you the binary version if you like.

The default compression is not great. ( you can set it in the constcbr argument of vl_video_encoder_init ) I don’t really know how the value is being used later on, but for 0x1XXYY. XX seems to be the quantization for the I frame, and YY is the motion quantization. the first 1 is something I added… not really used later on in the encoder.

5 Likes
19

Does this mean we can use FFMPEG and take advantage of hardware decoding…?

20

Yes with caveats: Only MPEG2, H264 and VP9 are upstream. Other codecs (MPEG4 etc.) are 1080p only and VIM3 has enough CPU grunt to software decode them. The exception is HEVC which is work in progress (next in the line). FFMpeg also needs some rework to adapt to recent V4L2 stateful API changes but it’s usable. Use the most recent FFMpeg releases.

1 Like