Issues with recording custom video + audio when using `v4l2-ctl` and `gst-launch-1.0`

Which system do you use? Android, Ubuntu, OOWOW or others?

Ubuntu 22.04, (5.15.119 kernel)

Which version of system do you use? Please provide the version of the system here:

VIM4 with NPU
+
Khadas OS08A10 camera

Please describe your issue below:

Issues with recording video + audio using v4l2-ctl and gst-launch-1.0.

I am trying to record a video with gst-launch-1.0 as described in the documentation:

gst-launch-1.0 -e v4l2src device=/dev/media0 io-mode=mmap num-buffers=300 ! video/x-raw,format=NV21,width=1920,height=1080,framerate=30/1 ! amlvenc bitrate=8000 ! h265parse ! qtmux ! filesink location=test-h265.mp4

(New VIM4 MIPI camera usage [Khadas Docs])

However, I noticed that format changes like the framerate are ignored when running the command. Even the example in the documentation (shown before) does not record the video at 30 framerate; it records the video at 25.

I changed the frame rate to a lower value (10/1) to see significant changes but the video was still recorded at 25.

My final goal is to record a video + audio using the OS08A10 + the integrated microphone in the VIM4. I want to change the exposure of the camera using v4l2-ctl with v4l2-ctl -d /dev/v4l-subdev4 --set-ctrl exposure=32768 but even this command seems to not take effect when using gst-launch-1.0 afterward. Are parameters hardcoded?

The only parameter that seems to change is the video size.

What am I missing from the documentation? Do I need to change some configuration values at startup to fix the parameters?

Can you provide an example of a gst-launch-1.0 pipeline that records video and audio for the specific video + mic devices in the vim4?

As an additional note, when I record a video using gst-launch-1.0, I see a lot of debug logging messages coming from amlv4l2src module, is this normal? For example,
[2024-06-12 18:19:13] DEBUG [amlv4l2src camctrl.cc:758:isp_alg_process_one]process one frame successfully

Post a console log of your issue below:

Hello @omarc

@ivan.li will help you then.

Hello @numbqq ,

Great, I look forward to @ivan.li help.

Omar

Hi @ivan.li ,

In case it is needed, this is the result of v4l2-ctl -d /dev/v4l-subdev4 -l

User Controls
                       exposure 0x00980911 (int)    : min=0 max=65535 step=1 default=0 value=451
                           gain 0x00980913 (int)    : min=0 max=240 step=1 default=0 value=0
                     sensor_fps 0x00981900 (int)    : min=1 max=30 step=1 default=30 value=30 flags=volatile, execute-on-write
                       wdr_mode 0x00981904 (int)    : min=0 max=2 step=1 default=0 value=0
                   sensor_lanes 0x00981907 (int)    : min=1 max=4 step=1 default=4 value=4 flags=read-only, volatile

Image Processing Controls

                 link_frequency 0x009f0901 (intmenu): min=0 max=1 default=0 value=0 (1440000000 0x55d4a800) flags=read-only
                     pixel_rate 0x009f0902 (int64)  : min=1 max=2147483647 step=1 default=1152000000 value=1152000000 flags=read-only

I also noticed that each time I record a video via gst-launch-1.0, the exposure parameter changes. I thought it was related to the wdr_mode parameter, so I changed it to see if it lets me keep the parameters I set before, but it does not. It seems there is an auto mode that sets those values and ignores the ones set by the v4l2-ctl --set-control

Hi @ivan.li ,

I have managed to record a video with audio using the OS08A10 and the onboard microphone using this command:

gst-launch-1.0 -e v4l2src device=/dev/media0 io-mode=mmap ! video/x-raw,format=NV21,width=1920,height=1080,framerate=30/1 ! queue ! amlvenc bitrate=4000 ! h264parse ! mux. alsasrc device="hw:AMLAUGESOUND,4" ! queue ! audio/x-raw,format=S16LE,channels=2,rate=48000 ! volume volume=2.0 ! audioconvert ! audioresample ! voaacenc ! aacparse ! qtmux name=mux ! filesink location=/home/khadas/src/web/app/static/preview.mp4 sync=true

The microphone sounds OK, but there is a static noise on the back all the time. I thought it was because of the volume module I added, but even at value 1.0, the static is still quite present. (here is a link to a preview of the recording SWITCHfilesender)

Nevertheless, I still have the same issues I mentioned before. The controls I set via v4l2-ctl are still ignored by gst-launch-1.0.

I looked at the v4l2_test_raw repository mentioned in the khadas docs about the new mipi camera documentation:

(New VIM4 MIPI camera usage [Khadas Docs])

The source code of your tool seems to consider the camera source as a dual camera and uses a WDR mode = 4,

if defined (DUAL_CAMERA)
        .fmt_code   = MEDIA_BUS_FMT_SRGGB12_1X12,
        .wdr_mode   = ISP_SDR_DCAM_MODE,

Maybe this code can help to see why the control settings seem to be ignored.

@numbqq, I was using the Edge2 and the OS08A10 camera to record in night mode (without the IR filter), and you helped me use the camera and check if the IR filter was enabled.

However, now that I am doing tests with the VIM4 and the same OS08A10 camera, it seems that the camera has IR Filter enabled. Is that correct? Or maybe this behavior has to do with the WDR mode, too.

You mentioned
```In dts file, the IR-CUT disabled in the GPIO by default. So if one day you need the function you can open the node manully. We only set the reset-gpio.``

Where can I find this dts file in the VIM4 to disable the IR CUT of the OS08A10?

Thanks in advance,

@omarc

1.Get source code and compile reference documentation:
http://docs.khadas.com/products/sbc/vim4/development/start

2.You can annotate ircat gpio in DTS or modify the driver program.

vim /common_drivers/arch/arm64/boot/dts/amlogic/kvim4n.dts
     amlsens_0: sensor0@36 {
         compatible = "amlogic, sensor";
         status = "okay";
         index = <0>;
         reg = <0x36>;
         reg-addr-type = <2>;
         reg-data-type = <1>;
 
         clocks = <&clkc CLKID_MCLK_0>,
                 <&clkc CLKID_MCLK_0_SEL>,
                 <&clkc CLKID_MCLK_0_PRE>,
                 <&clkc CLKID_MCLK_PLL>,
                 <&xtal>;
         clock-names = "mclk","mclk_sel","mclk_pre","mclk_p","mclk_x";
         reset-gpios = <&gpio GPIOM_5 GPIO_ACTIVE_HIGH>;
 //      ircut-gpios = <&gpio GPIOM_11 GPIO_ACTIVE_HIGH>;                                       
         port {
             amlsens_0_ep: endpoint {
                 data-lanes = <0 1 2 3>;
                 link-frequencies = /bits/ 64 <1440000000>;
                 remote-endpoint = <&csiphy0_ep_2>;
             };
         };
     }

Or

vim common_drivers/drivers/amlisp/amlsens/aml_sens.c
112     // IRCUT                                                                                    
113     ircut = devm_gpiod_get_optional(sensor->dev, "ircut", GPIOD_OUT_LOW);
114     if (IS_ERR(ircut)) {
115         rtn = PTR_ERR(ircut);
116         dev_err(sensor->dev, "Cannot get ircut gpio: %d\n", rtn);
117         goto err_return;
118     }
119     gpiod_set_value(ircut, 1);   //1:open IRCUT , 0:close IRCUT
120     pr_info("ircut init\n");

Hello @ivan.li ,

Thanks for the information; I will download the source code for the common drivers only (is it right?), change and compile it right away.

Do you have some updates regarding why the parameters set via v4l2-ctl are being ignored?

@omarc

http://docs.khadas.com/products/sbc/vim4/development/linux/build-ubuntu
No, you need to use fenix, please refer to this document. repositories (U-Boot, Linux,common drivers) will be cloned automatically from the Khadas GitHub. then modify and compile on this basis.

Sorry, I am also analyzing this issue. It seems that these parameters can only be set by modifying the driver source code.

common_drivers/drivers/amlisp

@ivan.li, thanks for the prompt reply.

I used fenix in docker to do the compilation of the kernel, boot, and common_drivers (I used the steps in the documentation).

Then, I modified the automatically downloaded common_drivers repository with the changes you suggested and compiled it again. This time, it took less time (I disabled git_update so that the tool would not override my changes).

I saw that the compilation tool generated deb files in an OTA folder and in a vim4 folder. Can I copy the deb files to my vim4 and update the packages via apt update path_file?

linux-board-package-jammy-vim4_1.6.8_arm64
linux-dtb-amlogic-5.15_1.6.8_arm64
linux-headers-amlogic-5.15_1.6.8_arm64
linux-image-amlogic-5.15_1.6.8_arm64
linux-u-boot-vim4-vendor_1.6.8-2019.01_arm64

Or do I need to create the USB drive and reinstall the vim4 with the generated image? I don’t want to reinstall the system again, as I have other tools installed already.

Thanks for looking into the v4l2-ctl issue. If you find out that the solution (for the moment) is to recompile the driver to fix the desired controls, I can do it.

@omarc

Yes ,it is the right way.

eg.
on PC

scp linux-dtb-amlogic-5.15_1.6.8_arm64 khadas@192.xxx.xx.xx:~

on VIM4

sudo dpkg -i linux-dtb-amlogic-5.15_1.6.8_arm64

@ivan.li , I have tried the modified dtb deb, and it works! Well, the IRCUT is disabled now. Thanks.

You mentioned that I can set the other parameters in the amlisp driver, do you mean in common_drivers/drivers/amlisp/amlcam or /amllens?

@omarc

You can try this.

amlcam/amlsens/ov08a10/aml_ov08a10.c

Hi @ivan.li I will look into that c file and try to change the camera’s parameters. Then, recompile the common_drivers module and install the OTA deb file.

I will let you know how it goes.

Hi @omarc

:clinking_glasses: If you have any questions, please leave a message in time.