VIM3(L) as powerful DSP

OldNavi · November 9, 2021, 4:28pm

Hi Khadas team,
(@numbqq @hyphop @Frank and @Terry )

I’ve started to explore a topic to turn VIM3/VIM3L into powerful signal processor as seems that SBC and Amlogic chip is much better suited for that rather then Raspberry PI4 due onboard NPU and built-in audio-codec.

And it would be much appreciated if you can give some answers and guidance on that long way.

I’m inspired by another projects which really does exists for RasPI (Elk AudioOS https://elk.audio) and GuitarML/NeuralPI https://github.com/GuitarML/NeuralPi

From my perspective if I could make NPU to run the same - it would be really great.

But the target is a bit difficult to reach as there are some steps needs to be taken
1/ Adopt Xenomai and Elk audio project (realtime linux microkernel ) to Amlogic chip
2/ Adapt Amlogic audio drivers to RTDM (realtime driver model) in Xenomai taking other as example - https://github.com/elk-audio/rpi-rtdm-audio-driver
3/ Attach some ADC/DAC hardware to I2S bus.
and etc…

So to just explore possibilities without going into much HW changes - I’m planning to tryout SPDIF IN/OUT ports on VIM3/L SBC and luckily all that available on 40 pin header

SPDIF out is already there, but for SPDIF I wanted to change PWM_F pin into SPDIF IN - could you please let me know what needs to be changed (u-boot, dts, smth else) to change this pin into SPDIF mode ?

Also there is a DAC already on board

Does it only work in balanced mode or can be switched to unbalanced ? Do I need just to get an output signal from TP(3-6) pinounts on a board ?

More questions will follow…

Many thanks in advance

numbqq · November 10, 2021, 2:19am

Here is a patch for linux to enable SPDIF IN on PIN35:

diff --git a/arch/arm64/boot/dts/amlogic/kvim3_linux.dts b/arch/arm64/boot/dts/amlogic/kvim3_linux.dts
index 38d9ec47338d..1dc99e31a50d 100644
--- a/arch/arm64/boot/dts/amlogic/kvim3_linux.dts
+++ b/arch/arm64/boot/dts/amlogic/kvim3_linux.dts
@@ -656,7 +656,7 @@
                                system-clock-frequency = <6144000>;
                        };
                        codec {
-                               sound-dai = <&spdif_codec>;
+                               sound-dai = <&dummy_codec>;
                        };
                };
        };
@@ -1023,7 +1023,7 @@
                interrupt-names = "irq_spdifin";
                pinctrl-names = "spdif_pins",
                                        "spdif_pins_mute";
-               pinctrl-0 = <&spdifout>;
+               pinctrl-0 = <&spdifout &spdifin>;
                pinctrl-1 = <&spdifout_a_mute>;
                status = "okay";
        };
@@ -1232,6 +1232,13 @@
                };
        };
 
+       spdifin: spdifin {
+               mux {/* GPIOH_5 */
+                       groups = "spdif_in_h";
+                       function = "spdif_in";
+               };
+       };
+
        pdmin: pdmin {
                mux { /* gpioa_5, gpioa_6, gpioa_7, gpioa_8, gpioa_9*/
                        groups = "pdm_din0_a",
diff --git a/arch/arm64/boot/dts/amlogic/overlays/kvim3/i2s.dts b/arch/arm64/boot/dts/amlogic/overlays/kvim3/i2s.dts
index 5cf3a162e13e..72b4b0506309 100644
--- a/arch/arm64/boot/dts/amlogic/overlays/kvim3/i2s.dts
+++ b/arch/arm64/boot/dts/amlogic/overlays/kvim3/i2s.dts
@@ -112,7 +112,7 @@
                                        system-clock-frequency = <6144000>;
                                };
                                codec {
-                                       sound-dai = <&spdif_codec>;
+                                       sound-dai = <&dummy_codec>;
                                };
                        };
                };

Check this documentation to upgrade the kernel.

https://docs.khadas.com/linux/vim3/UpgradeLinuxKernel.html

You also need to disable PWM_F, remove pwm_f node in overlays in file /boot/env.txt , and reboot the board.

SPDIF OUT playback:

$ aplay -Dhw:0,4 test.wav

SPDIF IN capture:

$ arecord -Dhw:0,4 -r 48000 -f S16_LE  test.wav

OldNavi · November 10, 2021, 7:29am

Many thanks for that!

dick7 · November 12, 2021, 4:52am

good ，as i think this way may works much better than a market SCOPE.

countzero · November 25, 2021, 5:51pm

Dear Khadas Team,

Is it possible to use the SPDIF out GPIO pins with linux mainline kernel also (with VIM3L)?

With 4.9 linux kernel built with Fenix it works fine, but with mainline kernel only sound via hdmi is possible (no sound device with output to spdif is configured). Can this be achived by modifying and compiling the device tree source for the VIM3L board or is there some kind of hardware support still missing in mainline linux?

Many thanks

numbqq · November 26, 2021, 1:21am

@hyphop Can you check SDPIF out with mainline kernel?

hyphop · November 26, 2021, 3:30am

VIM3L same VIM3 VIM2 VIM1

spdif and i2s output for mainline works fine but only with patches

fenix/packages/linux-mainline/patches/5.16-rc1 at master · khadas/fenix · GitHub

can build with GitHub - khadas/fenix: One-stop script set to build Ubuntu/Debian images

or just try build image https://dl.khadas.com/Firmware/Krescue/images/Generic_Ubuntu-minimal-focal_Linux-5.16-rc2_arm64_SD-USB_V1.0.7-211122-develop.img.xz

or install same image it via krescue online scripts

PS: same can ckeck it via Krescue - audio tests (krescue same used mainline kernel source)

countzero · November 26, 2021, 8:01am

Thank you both for your quick support. I really appreciate that.

countzero · November 30, 2021, 2:53pm

Hi @hyphop, thank you for your support. I tried your suggestions, but i have sound over gpio-spdif-pins only with Krescue Audio Tests. Here is what else i have tried:

burn the image you linked to sdcard: only sound via hdmi
build generic image with fenix, burn to sdcard: only sound via hdmi
build vim3l image with fenix (mainline kernel, mainline u-boot), burn to sdcard: only sound via hdmi
download linux-mainline 5.16-rc1, patched with patches you linked, configured boot partition of sdcard to use new kernel, new initrd and new dtb.img, which booted successfully, but again: only sound via hdmi

I kindly ask if you have any hint, why sound via spdif-pins is only working in krescue audio tests?

Many thanks

hyphop · December 1, 2021, 3:54am

you have 3 options

uncomment /* pinctl-0 … */ next blocks - and rebuild dtb in fenix
manualy rebuild by hands dtb just put same pinctl-0 blocks inside right places
creale and applay dt overlays from GitHub - khadas/khadas-linux-kernel-dt-overlays: khadas-linux-kernel-dt-overlays

# grep uncomment linux-mainline-5.16-rc2/arch/arm64/boot/dts/amlogic/meson-khadas-vim3.dtsi -B2 -A10

&tdmif_b {
	// uncomment next block  enable I2S output to GPIO HEADERS
	//
	assigned-clocks = <&clkc_audio AUD_CLKID_TDM_SCLK_PAD1>,
			  <&clkc_audio AUD_CLKID_TDM_LRCLK_PAD1>;
	assigned-clock-parents = <&clkc_audio AUD_CLKID_MST_B_SCLK>,
				 <&clkc_audio AUD_CLKID_MST_B_LRCLK>;
	assigned-clock-rates = <0>, <0>;
	/*
	pinctrl-0 = <&tdm_b_fs_pins>, <&tdm_b_dout0_pins>,
		    <&tdm_b_sclk_pins>, <&mclk0_a_pins>;
	*/
--

&spdifout {
	// uncomment next block  enable SPDIF output to GPIO HEADERS
	/*
	pinctrl-0 = <&spdif_ao_out_pins>;
	*/
	pinctrl-names = "default";
	status = "okay";
};

&cec_AO {
	pinctrl-0 = <&cec_ao_a_h_pins>;
	pinctrl-names = "default";

countzero · December 1, 2021, 3:29pm

Thank you again. I will try this soon.

[edit]: Today i tried your suggestion 3. (use spdifout overlay spdifout.dtbo) and succeeded. Many thanks. I am happy now.