Consult solution for ubuntu app that need opencl、opengl and npu ability

I have a ubuntu app that need opencl、opengl and npu ability. We want to run it in vim3.

it works in following 3 steps:

  1. recv image from uvc/mipi camera, do some transform and send result to vision ipc. this steps mainly run in opencl and cpu.
  2. another process recv image from vision ipc,then do model predict with npu, result will be send to msg queue.
  3. user gui process recv image from vision ipc and result from npu model predicts, use opengl to render and display result in x window.

In last two days, I have do some study in ubuntu 4.9 kernel and mainline. Found that:

  1. 4.9 kernel, opencl and npu work well, but opengl can only work in fbdev mode, not x window. It’s a pity.
  2. mainline kernel, mesa related stuff work well when do opengl job, but not support opencl hardware accelerate. Also npu in mainline kernel not work(including Segmentation fault for image_classify_224x224, Segmentation fault for tengine lite when doing prerun_graph_multithread).

I do want to use vim3 and had put lot of energy in this platform. But today I do really confuse.

So would you please share some light with me?
Thanks very much.

@ideafold The latest version of NPU no longer maintains amlogic nn api, tengine needs to wait for the official tengine version to be updated to us. What can be used on the latest NPU is KSNN and the original API

the mentioned amlogic nn api is the libnnsdk.so in aml-npu_6.4.6.2.5.3.2_arm64.deb?

Tengline lite rely on GitHub - VeriSilicon/TIM-VX: Verisilicon Tensor Interface Module, not the nnsdk.
https://tengine-docs.readthedocs.io/en/latest/source_compile/compile_timvx.html

aarch64_A311D_6.4.8/lib# ll galcore.ko lib*
-rw-r–r-- 1 khadas khadas 10204368 Oct 8 03:32 galcore.ko
-rwxr-xr-x 1 khadas khadas 381896 Oct 8 03:34 libArchModelSw.so*
-rwxr-xr-x 1 khadas khadas 2484984 Oct 8 03:33 libCLC.so*
-rwxr-xr-x 1 khadas khadas 2134480 Oct 8 03:32 libGAL.so*
-rwxr-xr-x 1 khadas khadas 397344 Oct 8 03:33 libNNArchPerf.so*
-rwxr-xr-x 1 khadas khadas 13816768 Oct 8 03:34 libNNGPUBinary.so*
-rwxr-xr-x 1 khadas khadas 53093504 Oct 8 03:35 libNNVXCBinary.so*
-rwxr-xr-x 1 khadas khadas 61160 Oct 8 03:36 libOpenVXC.so*
lrwxrwxrwx 1 khadas khadas 18 Oct 8 03:36 libOpenVX.so → libOpenVX.so.1.3.0*
lrwxrwxrwx 1 khadas khadas 18 Oct 8 03:36 libOpenVX.so.1 → libOpenVX.so.1.3.0*
-rwxr-xr-x 1 khadas khadas 2941928 Oct 8 03:36 libOpenVX.so.1.3.0*
-rwxr-xr-x 1 khadas khadas 36760 Oct 8 03:36 libOpenVXU.so*
-rwxr-xr-x 1 khadas khadas 6752888 Oct 8 03:33 libOvx12VXCBinary.so*
-rwxr-xr-x 1 khadas khadas 16846736 Oct 8 03:33 libVSC.so*

@ideafold

We no longer maintain amlogic nn api, so the latest version no longer has this library

The version on our board is the official customized version of tengine, so we need to wait for the official update of tengine. We are considering using the community version, but not currently, so the latest driver cannot use tengine temporarily. If you want to transplant the community version yourself, then You can try. I suggest you use raw api or ksnn

It looks like ksnn use acuity-toolkit to do model convert.
https://docs.khadas.com/linux/vim3/KSNNConvert.html

I had try acuity-toolkit with pytorch, hit a painful process.

This day, does acuity-toolkit support pytorch model well ??

@ideafold The current version of pytorch is still 1.2, and the official tool has not updated the version of pytorch for a long time.