Gstreamer issues in kernel 5.15

[smatbd]

Which system do you use? Android, Ubuntu, OOWOW or others?

OOWOW to install Ubuntu 22.04 kernel 5.15

Which version of system do you use? Khadas official images, self built images, or others?

khadas official

Please describe your issue below:

I have the VIM4 with NPU and fresh install of ubuntu 22.04 with kernel 5.15 from OOWOW. I am trying various gstreamer pipelines and finding the performance is pretty poor and/or often fails. Doing something like “gst-launch-1.0 videotestsrc ! fpsdisplaysink” is very laggy and gstreamer reports that my device is slow. .

I also have one of the khadas cameras and the gstreamer command in the doc works, with alot of console logs, but modifying it to use fpsdisplaysink does not.
https://docs.khadas.com/products/sbc/vim4/add-ons/new-vim4-mipi-camera#tab__kernel1

I also tried a USB camera which I was not able to get working with gstreamer.

I do notice that various hdmi displays don’t work very well with the vim4 but I have no issues on other sbcs with them. What should I look into for resolving hdmi display issues and the very poor performance of gstreamer?

Khadas:~$ uname -a
Linux Khadas 5.15.119 #1.6.4 SMP PREEMPT Sat Feb 3 10:31:33 CST 2024 aarch64 aarch64 aarch64 GNU/Linux

Post a console log of your issue below:

khadas@Khadas:~$ gst-launch-1.0 v4l2src device=/dev/media0 io-mode=mmap num-buffers=300 ! video/x-raw,format=NV21,width=1920,height=1080,framerate=30/1 ! queue ! fpsdisplaysink
Setting pipeline to PAUSED …
[API:aml_v4l2src_connect:271]Enter, devname : /dev/media0
func_name: aml_src_get_cam_method
initialize func addr: 0x7f95ea167c
finalize func addr: 0x7f95ea1948
start func addr: 0x7f95ea199c
stop func addr: 0x7f95ea1a4c
[2024-02-10 17:26:34] DEBUG [amlv4l2src camsrc.c:235:carm_src_is_usb]carm_src_is_usb:info(id 1, type 0x20000, name isp-csiphy), ret 0
[2024-02-10 17:26:34] DEBUG [amlv4l2src camsrc.c:235:carm_src_is_usb]carm_src_is_usb:info(id 4, type 0x20000, name isp-adapter), ret 0
[2024-02-10 17:26:34] DEBUG [amlv4l2src camsrc.c:235:carm_src_is_usb]carm_src_is_usb:info(id 7, type 0x20000, name isp-test-pattern-gen), ret 0
[2024-02-10 17:26:34] DEBUG [amlv4l2src camsrc.c:235:carm_src_is_usb]carm_src_is_usb:info(id 9, type 0x20000, name isp-core), ret 0
[2024-02-10 17:26:34] DEBUG [amlv4l2src camsrc.c:235:carm_src_is_usb]carm_src_is_usb:info(id 20, type 0x20001, name ov08a10-0), ret 0
[2024-02-10 17:26:34] DEBUG [amlv4l2src camsrc.c:235:carm_src_is_usb]carm_src_is_usb:info(id 24, type 0x10001, name isp-ddr-input), ret 0
[2024-02-10 17:26:34] DEBUG [amlv4l2src camsrc.c:235:carm_src_is_usb]carm_src_is_usb:info(id 28, type 0x10001, name isp-param), ret 0
[2024-02-10 17:26:34] DEBUG [amlv4l2src camsrc.c:235:carm_src_is_usb]carm_src_is_usb:info(id 32, type 0x10001, name isp-stats), ret 0
[2024-02-10 17:26:34] DEBUG [amlv4l2src camsrc.c:235:carm_src_is_usb]carm_src_is_usb:info(id 36, type 0x10001, name isp-output0), ret 0
[2024-02-10 17:26:34] DEBUG [amlv4l2src camsrc.c:235:carm_src_is_usb]carm_src_is_usb:info(id 40, type 0x10001, name isp-output1), ret 0
[2024-02-10 17:26:34] DEBUG [amlv4l2src camsrc.c:235:carm_src_is_usb]carm_src_is_usb:info(id 44, type 0x10001, name isp-output2), ret 0
[2024-02-10 17:26:34] DEBUG [amlv4l2src camsrc.c:235:carm_src_is_usb]carm_src_is_usb:info(id 48, type 0x10001, name isp-output3), ret 0
[2024-02-10 17:26:34] DEBUG [amlv4l2src camsrc.c:235:carm_src_is_usb]carm_src_is_usb:info(id 52, type 0x10001, name isp-raw), ret 0
[2024-02-10 17:26:34] DEBUG [amlv4l2src camsrc.c:235:carm_src_is_usb]carm_src_is_usb:info(id -2147483596, type 0x0, name ), ret -1
[2024-02-10 17:26:34] DEBUG [amlv4l2src camsrc.c:239:carm_src_is_usb]carm_src_is_usb:error Invalid argument
[2024-02-10 17:26:34] DEBUG [amlv4l2src camsrc.c:79:cam_src_select_socket]select socket:/tmp/camctrl0.socket
[2024-02-10 17:26:34] DEBUG [amlv4l2src camsrc.c:103:cam_src_obtain_devname]fork ok, pid:1321
[2024-02-10 17:26:34] DEBUG [amlv4l2src camsrc.c:103:cam_src_obtain_devname]fork ok, pid:0
[2024-02-10 17:26:34] DEBUG [amlv4l2src camsrc.c:107:cam_src_obtain_devname]execl /usr/bin/camctrl
[2024-02-10 17:26:34] DEBUG [amlv4l2src camctrl.cc:925:main][camctrl.cc:main:925]

[2024-02-10 17:26:34] DEBUG [amlv4l2src camctrl.cc:889:parse_opt]media device name: /dev/media0
[2024-02-10 17:26:34] DEBUG [amlv4l2src camctrl.cc:898:parse_opt]Server socket: /tmp/camctrl0.socket
Opening media device /dev/media0
Enumerating entities
Found 13 entities
Enumerating pads and links
mediaStreamInit[35]: mediaStreamInit ++.

mediaStreamInit[39]: media devnode: /dev/media0
mediaStreamInit[56]: ent 0, name isp-csiphy
mediaStreamInit[56]: ent 1, name isp-adapter
mediaStreamInit[56]: ent 2, name isp-test-pattern-gen
mediaStreamInit[56]: ent 3, name isp-core
mediaStreamInit[56]: ent 4, name ov08a10-0
mediaStreamInit[56]: ent 5, name isp-ddr-input
mediaStreamInit[56]: ent 6, name isp-param
mediaStreamInit[56]: ent 7, name isp-stats
mediaStreamInit[56]: ent 8, name isp-output0
mediaStreamInit[56]: ent 9, name isp-output1
mediaStreamInit[56]: ent 10, name isp-output2
mediaStreamInit[56]: ent 11, name isp-output3
mediaStreamInit[56]: ent 12, name isp-raw
mediaStreamInit[96]: get lens_ent fail
mediaLog[30]: v4l2_video_open: open subdev device node /dev/video63 ok, fd 8

mediaStreamInit[151]: mediaStreamInit open video0 fd 8
mediaLog[30]: v4l2_video_open: open subdev device node /dev/video64 ok, fd 9

mediaStreamInit[155]: mediaStreamInit open video1 fd 9
mediaLog[30]: v4l2_video_open: open subdev device node /dev/video65 ok, fd 10

mediaStreamInit[159]: mediaStreamInit open video2 fd 10
mediaLog[30]: v4l2_video_open: open subdev device node /dev/video66 ok, fd 13

mediaStreamInit[163]: mediaStreamInit open video3 fd 13
mediaStreamInit[172]: media stream init success
fetchPipeMaxResolution[27]: find matched sensor configs 3840x2160
media_set_wdrMode[420]: media_set_wdrMode ++ wdr_mode : 0

media_set_wdrMode[444]: media_set_wdrMode success –

media_set_wdrMode[420]: media_set_wdrMode ++ wdr_mode : 4

mediaLog[30]: v4l2_subdev_open: fd 15

mediaLog[30]: v4l2_subdev_open: fd 16

media_set_wdrMode[444]: media_set_wdrMode success –

mediaLog[30]: v4l2_subdev_open: fd 14

mediaLog[30]: v4l2_subdev_open: fd 17

mediaLog[30]: v4l2_subdev_open: fd 15

mediaLog[30]: v4l2_subdev_open: fd 15

mediaLog[30]: v4l2_subdev_open: fd 16

[2024-02-10 17:26:34] DEBUG [amlv4l2src camctrl.cc:374:link_and_activate_subdev]link and activate subdev successfully
[2024-02-10 17:26:34] DEBUG [amlv4l2src camctrl.cc:407:media_stream_config]config media stream successfully
mediaLog[30]: v4l2_video_open: open subdev device node /dev/video62 ok, fd 18

mediaLog[30]: VIDIOC_QUERYCAP: success

[2024-02-10 17:26:34] DEBUG [amlv4l2src camctrl.cc:172:check_capability]entity[isp-stats] → video[/dev/video62], cap.driver:aml-camera, capabilities:0x85200001, device_caps:0x5200001
mediaLog[30]: v4l2_video_open: open subdev device node /dev/video61 ok, fd 19

mediaLog[30]: VIDIOC_QUERYCAP: success

[2024-02-10 17:26:34] DEBUG [amlv4l2src camctrl.cc:172:check_capability]entity[isp-param] → video[/dev/video61], cap.driver:aml-camera, capabilities:0x85200001, device_caps:0x5200001
mediaLog[30]: set format ok, ret 0.

mediaLog[30]: set format ok, ret 0.

mediaLog[30]: request buf ok

mediaLog[30]: request buf ok

mediaLog[30]: query buffer success

[2024-02-10 17:26:34] DEBUG [amlv4l2src camctrl.cc:546:isp_alg_param_init]isp stats query buffer, length: 262144, offset: 0
mediaLog[30]: query buffer success

[2024-02-10 17:26:34] DEBUG [amlv4l2src camctrl.cc:546:isp_alg_param_init]isp stats query buffer, length: 262144, offset: 262144
mediaLog[30]: query buffer success

[2024-02-10 17:26:34] DEBUG [amlv4l2src camctrl.cc:546:isp_alg_param_init]isp stats query buffer, length: 262144, offset: 524288
mediaLog[30]: query buffer success

[2024-02-10 17:26:34] DEBUG [amlv4l2src camctrl.cc:546:isp_alg_param_init]isp stats query buffer, length: 262144, offset: 786432
mediaLog[30]: query buffer success

[2024-02-10 17:26:34] DEBUG [amlv4l2src camctrl.cc:568:isp_alg_param_init]isp param query buffer, length: 262144, offset: 0
alg2User func addr: 0x7f9bad8ed8
alg2Kernel func addr: 0x7f9bad8f08
algEnable func addr: 0x7f9bad8d70
algDisable func addr: 0x7f9bad8e90
algFwInterface func addr: 0x7f9bad9008
matchLensConfig[43]: LKK: fail to match lensConfig

cmos_get_ae_default_ov08a10[64]: cmos_get_ae_default

cmos_get_ae_default_ov08a10[64]: cmos_get_ae_default

aisp_enable[984]: tuning device not exist!

aisp_enable[987]: 3a commit b56e430e80b995bb88cecff66a3a6fc17abda2c7

cmos_fps_set_ov08a10[206]: cmos_fps_set: 7680.000000

mediaLog[30]: v4l2_subdev_open: fd 14

cmos_inttime_calc_table_ov08a10[180]: expo: 1741

mediaLog[30]: queue buffer success

mediaLog[30]: queue buffer success

mediaLog[30]: queue buffer success

mediaLog[30]: queue buffer success

mediaLog[30]: queue buffer success

mediaLog[30]: streamon success

mediaLog[30]: streamon success

[2024-02-10 17:26:35] DEBUG [amlv4l2src camctrl.cc:650:isp_alg_param_init]Finish initializing amlgorithm parameter …
[2024-02-10 17:26:35] DEBUG [amlv4l2src camctrl.cc:971:main]UNIX domain socket bound
[2024-02-10 17:26:35] DEBUG [amlv4l2src camctrl.cc:977:main]Accepting connections …
[2024-02-10 17:26:35] DEBUG [amlv4l2src camsrc.c:122:cam_src_obtain_devname]udp_sock_create
[2024-02-10 17:26:35] DEBUG [amlv4l2src common/common.c:70:udp_sock_create][211940005][/tmp/camctrl0.socket] start connect
[2024-02-10 17:26:35] DEBUG [amlv4l2src camsrc.c:124:cam_src_obtain_devname]udp_sock_recv
[2024-02-10 17:26:35] DEBUG [amlv4l2src camctrl.cc:985:main]connected_sockfd: 21
[2024-02-10 17:26:35] DEBUG [amlv4l2src camctrl.cc:989:main]video_dev_name: /dev/video63
[2024-02-10 17:26:35] DEBUG [amlv4l2src camsrc.c:282:cam_src_initialize]obtain devname: /dev/video63
devname : /dev/video63
driver : aml-camera
device : Amlogic Camera Card
bus_info : platform:aml-cam
version : 331639
error tvin-port use -1
[API:aml_v4l2src_streamon:373]Enter
[2024-02-10 17:26:35] DEBUG [amlv4l2src camsrc.c:298:cam_src_start]start …
[API:aml_v4l2src_streamon:376]Exit
[2024-02-10 17:26:35] DEBUG [amlv4l2src camctrl.cc:860:process_socket_thread]receive streamon notification
Pipeline is live and does not need PREROLL …
Pipeline is PREROLLED …
Setting pipeline to PLAYING …
New clock: GstSystemClock
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
Additional debug info:
…/libs/gst/base/gstbasesrc.c(3127): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming stopped, reason not-negotiated (-4)
Execution ended after 0:00:00.003382757
Setting pipeline to NULL …
[API:aml_v4l2src_streamoff:381]Enter
[2024-02-10 17:26:35] DEBUG [amlv4l2src camsrc.c:307:cam_src_stop]stop …
[2024-02-10 17:26:35] DEBUG [amlv4l2src camctrl.cc:866:process_socket_thread]receive streamoff notification

And this is what one of my displays looks when using gstreamer like but they all do this. You can see the console on the edges and the text is out of focus on the console. On other sbc this would just work, so I am not sure how to resolve this.

based on another post for vim3

I found I can check this
khadas@Khadas:~$ grep “” /sys/devices/virtual/amhdmitx/amhdmitx0/edid
Rx Manufacturer Name: OLE
Rx Product Code: 0071
Rx Serial Number: 00000000
Rx Product Name: OL51F
Manufacture Week: 0
Manufacture Year: 2023
Physical size(mm): 320 x 240
EDID Version: 1.3
native Mode 0, VIC (native 3):
ColorDeepSupport 0
3 4 5 16
Audio {format, channel, freq, cce}
{1, 1, 7, 1}
Speaker Allocation: 0
Vendor: 0x0 ( DVI device)
MaxTMDSClock1 165 MHz
vLatency: Invalid/Unknown
aLatency: Invalid/Unknown
i_vLatency: Invalid/Unknown
i_aLatency: Invalid/Unknown

and this
khadas@Khadas:~$ grep “” /sys/devices/virtual/amhdmitx/amhdmitx0/disp_cap
480p60hz*
720p60hz
1080i60hz
1080p60hz

[smatbd]

Some updates, I don’t know how performant kmssink is for gstreamer, but that is the plugin being selected currently by fpsdisplaysink. I found that some useful plugins for displaying video to screen via hardware are blacklisted. Here is the debug log. How can I get these missing libraries?

khadas@Khadas:~$ GST_DEBUG=4 gst-inspect-1.0 /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstamlvideosink.so
0:00:00.000411829 1434 0x55a0229200 INFO GST_INIT gst.c:592:init_pre: Initializing GStreamer Core Library version 1.20.3
0:00:00.000677035 1434 0x55a0229200 INFO GST_INIT gst.c:593:init_pre: Using library installed in /usr/lib/aarch64-linux-gnu
0:00:00.000795033 1434 0x55a0229200 INFO GST_INIT gst.c:611:init_pre: Linux Khadas 5.15.119 #1.6.4 SMP PREEMPT Sat Feb 3 10:31:33 CST 2024 aarch64
0:00:00.002111478 1434 0x55a0229200 INFO GST_INIT gstmessage.c:129:_priv_gst_message_initialize: init messages
0:00:00.005988478 1434 0x55a0229200 INFO GST_INIT gstcontext.c:86:_priv_gst_context_initialize: init contexts
0:00:00.007649710 1434 0x55a0229200 INFO GST_PLUGIN_LOADING gstplugin.c:324:_priv_gst_plugin_initialize: registering 0 static plugins
0:00:00.008246246 1434 0x55a0229200 INFO GST_PLUGIN_LOADING gstplugin.c:232:gst_plugin_register_static: registered static plugin “staticelements”
0:00:00.008354786 1434 0x55a0229200 INFO GST_PLUGIN_LOADING gstplugin.c:234:gst_plugin_register_static: added static plugin “staticelements”, result: 1
0:00:00.009073570 1434 0x55a0229200 INFO GST_REGISTRY gstregistry.c:1827:ensure_current_registry: reading registry cache: /home/khadas/.cache/gstreamer-1.0/registry.aarch64.bin
0:00:00.078245251 1434 0x55a0229200 INFO GST_REGISTRY gstregistrybinary.c:683:priv_gst_registry_binary_read_cache: loaded /home/khadas/.cache/gstreamer-1.0/registry.aarch64.bin in 0.068983 seconds
0:00:00.078805454 1434 0x55a0229200 INFO GST_REGISTRY gstregistry.c:1693:scan_and_update_registry: Validating plugins from registry cache: /home/khadas/.cache/gstreamer-1.0/registry.aarch64.bin
0:00:00.086636745 1434 0x55a0229200 INFO GST_REGISTRY gstregistry.c:1785:scan_and_update_registry: Registry cache has not changed
0:00:00.086707744 1434 0x55a0229200 INFO GST_REGISTRY gstregistry.c:1862:ensure_current_registry: registry reading and updating done, result = 1
0:00:00.086738952 1434 0x55a0229200 INFO GST_INIT gst.c:833:init_post: GLib runtime version: 2.72.4
0:00:00.086783410 1434 0x55a0229200 INFO GST_INIT gst.c:835:init_post: GLib headers version: 2.72.4
0:00:00.086826702 1434 0x55a0229200 INFO GST_INIT gst.c:837:init_post: initialized GStreamer successfully
0:00:00.096029729 1434 0x55a0229200 WARN GST_PLUGIN_LOADING gstplugin.c:875:_priv_gst_plugin_load_file_for_registry: module_open failed: libmediahal_videorender.so: cannot open shared object file: No such file or directory

**(gst-inspect-1.0:1434): GStreamer-WARNING : 18:33:05.370: Failed to load plugin ‘/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstamlvideosink.so’: libmediahal_videorender.so: cannot open shared object file: No such file or directory
Could not load plugin file: Opening module failed: libmediahal_videorender.so: cannot open shared object file: No such file or directory
khadas@Khadas:~$ GST_DEBUG=4 gst-inspect-1.0 /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstopengl.so
0:00:00.000408454 1438 0x55a354c200 INFO GST_INIT gst.c:592:init_pre: Initializing GStreamer Core Library version 1.20.3
0:00:00.000693368 1438 0x55a354c200 INFO GST_INIT gst.c:593:init_pre: Using library installed in /usr/lib/aarch64-linux-gnu
0:00:00.000813825 1438 0x55a354c200 INFO GST_INIT gst.c:611:init_pre: Linux Khadas 5.15.119 #1.6.4 SMP PREEMPT Sat Feb 3 10:31:33 CST 2024 aarch64
0:00:00.002190353 1438 0x55a354c200 INFO GST_INIT gstmessage.c:129:_priv_gst_message_initialize: init messages
0:00:00.005988814 1438 0x55a354c200 INFO GST_INIT gstcontext.c:86:_priv_gst_context_initialize: init contexts
0:00:00.007587339 1438 0x55a354c200 INFO GST_PLUGIN_LOADING gstplugin.c:324:_priv_gst_plugin_initialize: registering 0 static plugins
0:00:00.008197041 1438 0x55a354c200 INFO GST_PLUGIN_LOADING gstplugin.c:232:gst_plugin_register_static: registered static plugin “staticelements”
0:00:00.008306873 1438 0x55a354c200 INFO GST_PLUGIN_LOADING gstplugin.c:234:gst_plugin_register_static: added static plugin “staticelements”, result: 1
0:00:00.009038866 1438 0x55a354c200 INFO GST_REGISTRY gstregistry.c:1827:ensure_current_registry: reading registry cache: /home/khadas/.cache/gstreamer-1.0/registry.aarch64.bin
0:00:00.081659619 1438 0x55a354c200 INFO GST_REGISTRY gstregistrybinary.c:683:priv_gst_registry_binary_read_cache: loaded /home/khadas/.cache/gstreamer-1.0/registry.aarch64.bin in 0.072435 seconds
0:00:00.082256946 1438 0x55a354c200 INFO GST_REGISTRY gstregistry.c:1693:scan_and_update_registry: Validating plugins from registry cache: /home/khadas/.cache/gstreamer-1.0/registry.aarch64.bin
0:00:00.090900941 1438 0x55a354c200 INFO GST_REGISTRY gstregistry.c:1785:scan_and_update_registry: Registry cache has not changed
0:00:00.090971440 1438 0x55a354c200 INFO GST_REGISTRY gstregistry.c:1862:ensure_current_registry: registry reading and updating done, result = 1
0:00:00.091003856 1438 0x55a354c200 INFO GST_INIT gst.c:833:init_post: GLib runtime version: 2.72.4
0:00:00.091040064 1438 0x55a354c200 INFO GST_INIT gst.c:835:init_post: GLib headers version: 2.72.4
0:00:00.091076647 1438 0x55a354c200 INFO GST_INIT gst.c:837:init_post: initialized GStreamer successfully
0:00:00.106357198 1438 0x55a354c200 WARN GST_PLUGIN_LOADING gstplugin.c:875:_priv_gst_plugin_load_file_for_registry: module_open failed: libMali.so: cannot open shared object file: No such file or directory

**(gst-inspect-1.0:1438): GStreamer-WARNING : 18:33:57.758: Failed to load plugin ‘/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstopengl.so’: libMali.so: cannot open shared object file: No such file or directory
Could not load plugin file: Opening module failed: libMali.so: cannot open shared object file: No such file or directory

[smatbd]

I found a script here, but doesn’t seem to work on vim4

How to change the display resolution of the Ubuntu on VIM3

Yes, you can use this script to setup the resolution: $ cat hdmi.sh #!/bin/sh hpd_state=`cat /sys/class/amhdmitx/amhdmitx0/hpd_state` #bpp=24 bpp=32 mode=${1:-720p60hz} #mode=2160p60hz #mode=720p60hz if [ $hpd_state -eq 0 ]; then # Exit if HDMI cable is not connected exit 0 fi common_display_setup() { M="0 0 $(($X - 1)) $(($Y - 1))" Y_VIRT=$(($Y * 2)) fbset -fb /dev/fb0 -g $X $Y $X $Y_VIRT $bpp echo null > /sys/class/display/mode echo $mode > /sys/class/display/mode echo $M > /sys/c…

[smatbd]

Do we have any ideas? Is libmali missing because its headless OS version?

[Electr1]

@smatbd install this package for libmali.so

$ wget https://dl.khadas.com/repos/vim3-5.15/pool/main/l/linux-gpu-mali-wayland/linux-gpu-mali-wayland_1.2-r44p0-202311_arm64.deb
$ sudo dpkg -i linux-gpu-mali-wayland_1.2-r44p0-202311_arm64.deb