OldNavi Guitar ML AMP/CAB simulator project - Khadas’ 1st Community Competition

OldNavi · December 13, 2021, 9:20am

Well, yet another DIY project from me and now it is time to turn Khadas SOC board into to power AudioDSP :-). An idea of this project is try to build Guitar Amplifier / Cabinet emulator with various effects using neural networks and deep-learning. Good to know that it can be hardware accelerated as Amlogic SOC has a built-in NPU module.

I was inspired by the following article

and couple of project build around it https://github.com/GuitarML

However, in that projects RaspberryPI 4 was used and we have Khadas SOC.

This DIY project is pure experimental and this project journey seems to be pretty hard but I still I’ll get something working and valuable at the end. I’ll have to learn a lot as I’m pretty noob in Neural networks programming and deep-learning.

So I’ve ordered some details (and Khadas free sample for VIM3) and came out with the following to do plan:

1/. Arrange all basic schematics - preamp, ADC/DAC section (at the moment I decided to stick with PCM chips as starting point), VU meter - should be pretty easy to make that.

2/. Develop microcontroller code to handle all Knobs/Switches/Encoders - usually should something build on Atmel microchips - ATMega328/128

3/. Port Xenomai/EVL dual linux kernel - to get realtime kernel capabilities. I’ll cover some things about realtime kernels in a next post.

4/. Rewrite Amlogic SOC sound subsystem drivers to be capable handle RT work - that’s gonna be hard piece of work

5/ Adopt realtime libraries from Elk.Audio projects (RASPA and TWINE) https://github.com/elk-audio

6/. Port headless DAW software from Elk too (Sushi and Sensei)

7/ Develop Neural Network model for AMP/CAB simulation and try to make it run on Amlogic NPU (a bit of worries here - don’t know if we could get a realtime handling here)

8/ Develop some extra VST plugins (Impulse Response Cab simulator) with JUCE framework

9/ Develop a body for all of that in CAD/CAM software and 3D print…

10/ If time permit - start to add extra functionality like Midi-over-Ethernet to control all parameters and etc…

Don’t like long to do list… 10 points usually tends to be enough.

To be continued…

Frank · December 14, 2021, 2:39am

@OldNavi Looking forward to your work

OldNavi · December 22, 2021, 9:56am

Warning: Long read!!!

Chapter I. «The Kernel and latencies»

Preambula: It is an experimental project with an outcome to be pretty unknown - it may succeed or it may fail. Who knows :-). And in any case it will take a bit more time to research and implement so I’ve started a bit earlier rather than February timeline.

Well, to get a really working solution with a «almost» realtime audio DSP on Khadas board - stock kernel isn’t good. As everybody knows signal processing require timing guaranties, It means that hardware and software events like IRQs from audio codecs should reach user process as quick as possible and in absolutely in reliable manner. Standard kernel isn’t good for that (it is not a problem of just Linux itself - it is general issue on most modern OSes)

So what to do here ? We need a kernel with timing guaranties and lowest possible latency for interrupt-to-process event. That means that we need 2 aspects implemented in a kernel -
1/. We need a realtime scheduling for all pieces of work (user threads, kernel threads and so on) - PREEMPTIVE_RT kernel option could be helpful here but it not just enough.
2/ We need also another way to handle IRQs -with ability to deliver it as quick as possible - nothing can be found in standard kernel.

Luckily there was a lot of job done in the past in area of realtime kernels. It is a Xenomai project xenomai.org which creates a almost true realtime kernel, however it is bit outdated and doesn’t support newer kernel. Its successor EVL project (also known as Xenomai 4) http://evlproject.org is to the rescue.

So, at first I was going to create a Xenomai RT kernel based on Amlogc vendor kernel 4.9. as it have everything for Amlogic SOC including TS050 panel working. However spending 3 weeks trying to adopt Xenomai kernel to Amlogic kernel version - I gave up, too much changes required because Amlogic added so many changes into the Linux kernel and it will take really huge amount of work to do.

So I decide to go to mainline kernel. I understood that I’ll loose some functionality like TS050 (that hurts badly ) but that in any case this is a future :-). Dealing with mainline kernel is much easier to merge Xenomai kernel codebase (as they already have mainline kernel source with incorporated changes)

So I took Fenix build script and slightly modified it, adding yet another configuration. Also I created another package in Fenix directory pointing to Xenomai RT kernel and incorporating latest Khadas patches. And after a week or so I finally got a dual Realtime kernel based on Xenomai and Amlogic/Khadas patches.

Dual kernel configuration means - that there are 2 kernel running together on the same SOC, realtime (so called Out-Of-Band kernel) and non-realtime (In-Band-Kernel) legacy kernel and have a common so called IRQ-pipeline (Dovetail Project). To familiarize yourself - please have a look to http://evlproject.org wiki pages. In my setup I’ve assigned 2 CPUs to inband kernel and 2 CPUs to out-of-band kernel. Hope when Khadas give me a VIM3 sample - I’ll use 4 small cores for inband kernel and 4 big cores for realtime kernel

What is interesting - that switching to RT kernel exposed some legacy drivers misbehavior (as in RT kernel no IRQs can be masked out). So booted with default drivers showed some unpleasant picture of CPU usage on Inband assigned CPU. Most of the CPU consumption was generated by Broadcom firmware driver (sic) - so as result I’ve blacklisted that module - (no plans for Wifi at the moment ).
Htop results with all drivers loaded

Broadcom drivers unloaded

EVL (aka Xenomai 4) has a buit-in tools to measure how good system is to handle interrupt to process latencies. So let measure them using builtin latmus utility, but before do calibration tasks

Idle mode: No apps running - CPU usage idle

Running latency test

# test started on: Wed Dec 22 10:55:45 2021
# Linux version 5.15.0 (root@OldNavi) (aarch64-none-linux-gnu-gcc (GNU Toolchain for the A-profile Architecture 9.2-2019.12 (arm-9.10)) 9.2.1 20191025, GNU ld (GNU Toolchain for the A-profile Architecture 9.2-2019.12 (arm-9.10)) 2.33.1.20191209) #1.0.7 SMP IRQPIPE Thu Dec 16 10:55:48 MSK 2021
# root=UUID=29631686-a319-4357-b990-9234333a47a2 rootfstype=ext4 rootflags=data=writeback rw ubootpart=715e90c6-01 console=ttyAML0,115200n8  no_console_suspend consoleblank=0 loglevel=0 logo=osd0,loaded,0x3d800000,panel vout=panel,enable hdmimode=1080p60hz  fbcon=rotate:90 fsck.repair=yes net.ifnames=0  wol_enable=0  jtag=disable mac=c8:63:14:71:2b:98  fan=auto khadas_board=VIM3L hwver=VIM3.V12 coherent_pool=2M pci=pcie_bus_perf reboot_mode=cold_boot imagetype=SD-USB uboottype=vendor splash quiet plymouth.ignore-serial-consoles vt.handoff=7 isolcpus=2,3 evl.oob_cpus=2
# libevl version: evl.0 -- #41dc88a (2021-08-21 18:13:57 +0200)
# sampling period: 1000 microseconds
# clock gravity: 500i 1000k 2500u
# clocksource: arch_sys_counter
# vDSO access: architected
# context: user
# thread priority: 98
# thread affinity: CPU3
# C-state restricted
# duration (hhmmss): 00:00:49
# peak (hhmmss): 00:00:02
# min latency: 0.052
# avg latency: 0.344
# max latency: 8.575
# sample count: 49468
0 49364
1 52
2 0
3 26
4 21
5 1
6 2
7 1
8 1

No apps are running here, and results are astonishing - in some cases it is just 80 nanoseconds latency from interrupt to the user process.
Histogram data

High load - run stress test with heitic forcing 10 microsecond context switching…
Turning on some stress load using heitic utility with 200 miscroseconds context switch - 100% CPU utilization

# test started on: Wed Dec 22 11:26:36 2021
# Linux version 5.15.0 (root@OldNavi) (aarch64-none-linux-gnu-gcc (GNU Toolchain for the A-profile Architecture 9.2-2019.12 (arm-9.10)) 9.2.1 20191025, GNU ld (GNU Toolchain for the A-profile Architecture 9.2-2019.12 (arm-9.10)) 2.33.1.20191209) #1.0.7 SMP IRQPIPE Thu Dec 16 10:55:48 MSK 2021
# root=UUID=29631686-a319-4357-b990-9234333a47a2 rootfstype=ext4 rootflags=data=writeback rw ubootpart=715e90c6-01 console=ttyAML0,115200n8  no_console_suspend consoleblank=0 loglevel=0 logo=osd0,loaded,0x3d800000,panel vout=panel,enable hdmimode=1080p60hz  fbcon=rotate:90 fsck.repair=yes net.ifnames=0  wol_enable=0  jtag=disable mac=c8:63:14:71:2b:98  fan=auto khadas_board=VIM3L hwver=VIM3.V12 coherent_pool=2M pci=pcie_bus_perf reboot_mode=cold_boot imagetype=SD-USB uboottype=vendor splash quiet plymouth.ignore-serial-consoles vt.handoff=7 isolcpus=2,3 evl.oob_cpus=2
# libevl version: evl.0 -- #41dc88a (2021-08-21 18:13:57 +0200)
# sampling period: 1000 microseconds
# clock gravity: 500i 1000k 2500u
# clocksource: arch_sys_counter
# vDSO access: architected
# context: user
# thread priority: 98
# thread affinity: CPU3
# C-state restricted
# duration (hhmmss): 00:00:37
# peak (hhmmss): 00:00:36
# min latency: 6.554
# avg latency: 9.504
# max latency: 13.850
# sample count: 36570
6 8
7 836
8 9973
9 16235
10 7374
11 1936
12 196
13 12

Histogram of IRQs timing

High load (all CPU at 100%)- run stress test with heitic forcing 10 microsecond context switching…
Same as previous but now context switching at maximum with 10 microseconds between ctx switcher plus adding CPU cache wiping with dd utility

Context switching per CPU - 1 second sampling period

Test details

# test started on: Wed Dec 22 11:28:14 2021
# Linux version 5.15.0 (root@OldNavi) (aarch64-none-linux-gnu-gcc (GNU Toolchain for the A-profile Architecture 9.2-2019.12 (arm-9.10)) 9.2.1 20191025, GNU ld (GNU Toolchain for the A-profile Architecture 9.2-2019.12 (arm-9.10)) 2.33.1.20191209) #1.0.7 SMP IRQPIPE Thu Dec 16 10:55:48 MSK 2021
# root=UUID=29631686-a319-4357-b990-9234333a47a2 rootfstype=ext4 rootflags=data=writeback rw ubootpart=715e90c6-01 console=ttyAML0,115200n8  no_console_suspend consoleblank=0 loglevel=0 logo=osd0,loaded,0x3d800000,panel vout=panel,enable hdmimode=1080p60hz  fbcon=rotate:90 fsck.repair=yes net.ifnames=0  wol_enable=0  jtag=disable mac=c8:63:14:71:2b:98  fan=auto khadas_board=VIM3L hwver=VIM3.V12 coherent_pool=2M pci=pcie_bus_perf reboot_mode=cold_boot imagetype=SD-USB uboottype=vendor splash quiet plymouth.ignore-serial-consoles vt.handoff=7 isolcpus=2,3 evl.oob_cpus=2
# libevl version: evl.0 -- #41dc88a (2021-08-21 18:13:57 +0200)
# sampling period: 1000 microseconds
# clock gravity: 500i 1000k 2500u
# clocksource: arch_sys_counter
# vDSO access: architected
# context: user
# thread priority: 98
# thread affinity: CPU3
# C-state restricted
# duration (hhmmss): 00:00:39
# peak (hhmmss): 00:00:38
# min latency: 7.531
# avg latency: 14.013
# max latency: 42.643
# sample count: 39489
7 3
8 578
9 4323
10 5495
11 3379
12 2870
13 3689
14 4409
15 4146
16 3386
17 2534
18 1748
19 1116
20 648
21 367
22 214
23 102
24 63
25 44
26 27
27 36
28 42
29 37
30 67
31 52
32 32
33 27
34 25
35 8
36 8
37 7
38 3
39 2
40 0
41 1
42 1

Histogram

High load extreme. Previous test plus ping flood (ping at maximum possible rate ) via ethernet
Everything as in previous test plus added a ping flood test (pinging Khadas board at maximum possible rate)

Test details

# test started on: Wed Dec 22 11:30:03 2021
# Linux version 5.15.0 (root@OldNavi) (aarch64-none-linux-gnu-gcc (GNU Toolchain for the A-profile Architecture 9.2-2019.12 (arm-9.10)) 9.2.1 20191025, GNU ld (GNU Toolchain for the A-profile Architecture 9.2-2019.12 (arm-9.10)) 2.33.1.20191209) #1.0.7 SMP IRQPIPE Thu Dec 16 10:55:48 MSK 2021
# root=UUID=29631686-a319-4357-b990-9234333a47a2 rootfstype=ext4 rootflags=data=writeback rw ubootpart=715e90c6-01 console=ttyAML0,115200n8  no_console_suspend consoleblank=0 loglevel=0 logo=osd0,loaded,0x3d800000,panel vout=panel,enable hdmimode=1080p60hz  fbcon=rotate:90 fsck.repair=yes net.ifnames=0  wol_enable=0  jtag=disable mac=c8:63:14:71:2b:98  fan=auto khadas_board=VIM3L hwver=VIM3.V12 coherent_pool=2M pci=pcie_bus_perf reboot_mode=cold_boot imagetype=SD-USB uboottype=vendor splash quiet plymouth.ignore-serial-consoles vt.handoff=7 isolcpus=2,3 evl.oob_cpus=2
# libevl version: evl.0 -- #41dc88a (2021-08-21 18:13:57 +0200)
# sampling period: 1000 microseconds
# clock gravity: 500i 1000k 2500u
# clocksource: arch_sys_counter
# vDSO access: architected
# context: user
# thread priority: 98
# thread affinity: CPU3
# C-state restricted
# duration (hhmmss): 00:00:40
# peak (hhmmss): 00:00:01
# min latency: 7.170
# avg latency: 13.763
# max latency: 41.726
# sample count: 39882
7 119
8 2053
9 5322
10 3995
11 2414
12 2815
13 4155
14 4933
15 4426
16 3342
17 2358
18 1473
19 911
20 536
21 303
22 166
23 90
24 48
25 30
26 21
27 37
28 41
29 50
30 41
31 64
32 36
33 36
34 34
35 18
36 8
37 4
38 1
39 1
40 0
41 1

Is these numbers good ? They are excellent !!! There were no overruns (IRQ realtime requirements met) at all and we newer had a latencies greater then 50 microseconds!!!

Just for comparison - here is a latency measurement on very well tuned Windows 10 with AMD Ryzen 12 core CPU, 32Gb RAM and extreme tweaking of everything (MSI-X Priority, Drivers affinity to CPU setup, Drivers threads priority tweaking to RT and so on)

Windows IDLE

Windows 70% LOAD

Windows 100% LOAD

Amlogic VIM3L and Khadas board with mainline Linux seems to be an absolute winner here.

As the next task I’ll have to rewrite Amlogic SOC sound drivers to realtime manner - throwing away all ALSA layer. Stay tuned !

Frank · December 23, 2021, 12:59am

@OldNavi Amazing work, I look forward to your work ！！！

OldNavi · January 2, 2022, 9:03am

Now it is time for new year fairy tales.

Chapter II - Realtime audio driver

Well, that was hell long and bumpy road to go, cause…

Default ALSA SOC driver is good enough for ordinary sound task like playing you favorite music or record something… But truly incapable to do realtime work, as there is no synchronization between capturing audio and playing it back…

So after long study of SOC datasheet a decision was made how things should be organized.

At first, I need to make input and output synchronous - that means that clocks, frequencies and bit depths should be equal for both input and output TDM. No resample engine allowed and by this time, no support for EQDRC or Sharebuffer (for parallel SPDIF output interface) will be implemented.

At second point - SOC itself allows circular 2 buffers scheme (not implemented In Amlogic ALSA drivers) - so I’ll implement this. What is more important - an SOC raises an interrupt when switching buffers - good point to sync kernel task with user space - saying - Hey, I have a new buffer for DSP to handle!!!

Need to say that without oscilloscope - there are no chances to debug things like audio driver…. I spent a lot of hours dealing with a stupid mistake … For some reasons I was getting data from ADC via TDM DATA IN slot … but sometimes it was purely wrong… At the end I’ve found that pinctrl was not applying right amperage to the clock pins (my mistake - as in mainline DTS pin amperage is now described as drive-strength-microamp = directive instead one used in 4.9 kernerls)

However it was still not enough to get a realtime driver. Through several test - it was unexpected floats of times when interrupts happens - even up to 300 microseconds… Many days were spent trying to figure out a root cause and finally SOC internal fifo’s were found culprit… So I had to make extreme changes to fifos config - and yes - finally I got my realtime driver working as expected…

root@Khadas:~# modinfo evl_audio_rt
filename:       /lib/modules/5.15.0/kernel/drivers/evl/sound/evl-amlogic/evl-audio-rt.ko
license:        GPL
description:    Realtime Amlogic SOC AUGE direct sound driver
author:         Old Navi (old.navi@bk.ru)
alias:          of:N*T*Camlogic,axg-sound-realtimeC*
alias:          of:N*T*Camlogic,axg-sound-realtime
alias:          of:N*T*Camlogic,_sm1-audio-ddr-managerC*
alias:          of:N*T*Camlogic,_sm1-audio-ddr-manager
alias:          of:N*T*Camlogic,_tl1-audio-ddr-managerC*
alias:          of:N*T*Camlogic,_tl1-audio-ddr-manager
alias:          of:N*T*Camlogic,_g12a-audio-ddr-managerC*
alias:          of:N*T*Camlogic,_g12a-audio-ddr-manager
alias:          of:N*T*Camlogic,_axg-audio-ddr-managerC*
alias:          of:N*T*Camlogic,_axg-audio-ddr-manager
alias:          of:N*T*Camlogic,_tm2-audio-clocksC*
alias:          of:N*T*Camlogic,_tm2-audio-clocks
alias:          of:N*T*Camlogic,_sm1-audio-clocksC*
alias:          of:N*T*Camlogic,_sm1-audio-clocks
alias:          of:N*T*Camlogic,_tl1-audio-clocksC*
alias:          of:N*T*Camlogic,_tl1-audio-clocks
alias:          of:N*T*Camlogic,_g12a-audio-clocksC*
alias:          of:N*T*Camlogic,_g12a-audio-clocks
alias:          of:N*T*Camlogic,_axg-audio-clocksC*
alias:          of:N*T*Camlogic,_axg-audio-clocks
alias:          of:N*T*Camlogic,_tm2-snd-tdmlbC*
alias:          of:N*T*Camlogic,_tm2-snd-tdmlb
alias:          of:N*T*Camlogic,_tm2-snd-tdmcC*
alias:          of:N*T*Camlogic,_tm2-snd-tdmc
alias:          of:N*T*Camlogic,_tm2-snd-tdmbC*
alias:          of:N*T*Camlogic,_tm2-snd-tdmb
alias:          of:N*T*Camlogic,_tm2-snd-tdmaC*
alias:          of:N*T*Camlogic,_tm2-snd-tdma
alias:          of:N*T*Camlogic,_sm1-snd-tdmlbC*
alias:          of:N*T*Camlogic,_sm1-snd-tdmlb
alias:          of:N*T*Camlogic,_sm1-snd-tdmcC*
alias:          of:N*T*Camlogic,_sm1-snd-tdmc
alias:          of:N*T*Camlogic,_sm1-snd-tdmbC*
alias:          of:N*T*Camlogic,_sm1-snd-tdmb
alias:          of:N*T*Camlogic,_sm1-snd-tdmaC*
alias:          of:N*T*Camlogic,_sm1-snd-tdma
alias:          of:N*T*Camlogic,_tl1-snd-tdmlbC*
alias:          of:N*T*Camlogic,_tl1-snd-tdmlb
alias:          of:N*T*Camlogic,_tl1-snd-tdmcC*
alias:          of:N*T*Camlogic,_tl1-snd-tdmc
alias:          of:N*T*Camlogic,_tl1-snd-tdmbC*
alias:          of:N*T*Camlogic,_tl1-snd-tdmb
alias:          of:N*T*Camlogic,_tl1-snd-tdmaC*
alias:          of:N*T*Camlogic,_tl1-snd-tdma
alias:          of:N*T*Camlogic,_g12a-snd-tdmlbC*
alias:          of:N*T*Camlogic,_g12a-snd-tdmlb
alias:          of:N*T*Camlogic,_g12a-snd-tdmcC*
alias:          of:N*T*Camlogic,_g12a-snd-tdmc
alias:          of:N*T*Camlogic,_g12a-snd-tdmbC*
alias:          of:N*T*Camlogic,_g12a-snd-tdmb
alias:          of:N*T*Camlogic,_g12a-snd-tdmaC*
alias:          of:N*T*Camlogic,_g12a-snd-tdma
alias:          of:N*T*Camlogic,_axg-snd-tdmlbC*
alias:          of:N*T*Camlogic,_axg-snd-tdmlb
alias:          of:N*T*Camlogic,_axg-snd-tdmcC*
alias:          of:N*T*Camlogic,_axg-snd-tdmc
alias:          of:N*T*Camlogic,_axg-snd-tdmbC*
alias:          of:N*T*Camlogic,_axg-snd-tdmb
alias:          of:N*T*Camlogic,_axg-snd-tdmaC*
alias:          of:N*T*Camlogic,_axg-snd-tdma
depends:        
intree:         Y
name:           evl_audio_rt
vermagic:       5.15.0 SMP mod_unload aarch64
parm:           audio_buffer_size:uint
parm:           audio_hat:charp
parm:           audio_enable_low_latency:uint
root@Khadas:~#

(As side effect - I’m not longer limited to standard I2S frequencies - now I can have any - like 33333Hz - and everything works)

So I’ve added some extra code for measurement into the driver (I’m measuring a nanoseconds timestamps when interrupt arrives) and wrote a simple userpace test program - which receives an event when driver switches to the new buffer and print some information

It is also required some kernel and driver tuning… for kernel config it was require to isolate some CPUs from Realtime Kernel and virse-versa using isolcpus and oobcpus arguments

[    0.000000] Kernel command line: root=UUID=641c1c63-3430-4636-9ab3-0002f130cd43 rootfstype=ext4 rootflags=data=writeback rw ubootpart=357d90c6-01 console=ttyAML0,115200n8  no_console_suspend consoleblank=0 loglevel=0 osd12  fbcon=rotate:90 fsck.repair=yes net.ifnames=0    jtag=disable mac=c8:63:14:71:2b:98  fan=auto khadas_board=VIM3L hwver=VIM3.V12 coherent_pool=2M pci=pcie_bus_perf  imagetype=EMMC_MBR uboottype=mainline splash quiet plymouth.ignore-serial-consoles vt.handoff=7 isolcpus=1,2,3 evl.oobcpus=1,2,3

Also realtime driver code required to use interrupt affinity to OOB cpus - just to avoid interfere of interrupts (by default Linux kernel pins most of interrupt to CPU0)

root@Khadas:/opt/evl/bin# cat /proc/interrupts 
           CPU0       CPU1       CPU2       CPU3       
  9:          0          0          0          0     GICv2  25 Level         vgic
 10:          0  128451273    6961558    6622463     SIRQC  10 Edge          in-band work
 12:          0          0          0          0     GICv2  30 Level         kvm guest ptimer
 13:          0          0          0          0     GICv2  27 Level         kvm guest vtimer
 14:     693238  273781873     828541     647049     GICv2  26 Level    oob  arch_timer
 16:     217688          0          0          0     GICv2  40 Level         eth0
 20:          4          0          0          0     GICv2 225 Edge          ttyAML0
 21:         97          0          0          0     GICv2 227 Edge          ff805000.i2c
 22:          0          0          0          0     GICv2 228 Edge          ff808000.ir
 23:          2          0          0          0     GICv2 232 Edge          ff809000.adc
 24:          0          0          0          0     GICv2  76 Edge          vdec
 25:          0          0          0          0     GICv2  64 Edge          esparserirq
 27:          0          0          0          0     GICv2  71 Edge          ffd1c000.i2c
 28:       4199          0          0          0     GICv2  58 Edge          ttyAML6
 29:     223581          0          0          0     GICv2 221 Edge          ffe03000.sd
 30:          0          0          0          0     GICv2 222 Edge          ffe05000.sd
 31:       6603          0          0          0     GICv2 223 Edge          ffe07000.mmc
 33:          0          0          0          0     GICv2 194 Level         panfrost-job
 34:          0          0          0          0     GICv2 193 Level         panfrost-mmu
 35:          3          0          0          0     GICv2 192 Level         panfrost-gpu
 36:          0          0          0          0     GICv2 218 Level         galcore:0
 37:     693182     831246     828514     647021     SIRQC  37 Edge          proxy tick
 38:          0          0          0          0     GICv2  63 Level         ff400000.usb, ff400000.usb
 39:         33          0          0          0     GICv2  62 Level         xhci-hcd:usb1
 41:          0          0          0          0   PCI-MSI   0 Edge          PCIe PME, aerdrv
 42:         11          0          0          0   PCI-MSI 524288 Edge          nvme0q0
 43:          1          0          0          0  meson-gpio-irqchip  26 Level         0.0:00
 45:      10240          0          0          0   PCI-MSI 524289 Edge          nvme0q1
 46:          0          0          0          0   PCI-MSI 524290 Edge          nvme0q2
 47:          0          0          0          0   PCI-MSI 524291 Edge          nvme0q3
 48:          0          0          0          0   PCI-MSI 524292 Edge          nvme0q4
 49:          0     202121          0          0     GICv2 180 Edge     oob  sound@0xFF660000
 52:          0     202157          0          0     GICv2 184 Edge     oob  sound@0xFF660000
IPI0:    209671        703   43306187   41385913       Rescheduling interrupts
IPI1:       118        210        856        640       Function call interrupts
IPI2:         0          0          0          0       CPU stop interrupts
IPI3:         0          0          0          0       CPU stop (for crash dump) interrupts
IPI4:         0          0          0          0       Timer broadcast interrupts
IPI5:         0          0          0          0       IRQ work interrupts
IPI6:         0          0          0          0       CPU wake-up interrupts

So my driver is smart enough to set affinity to OOB cpus, notice OOB flags set for interrupt lines 49 and 52

So now it is time for measurements… I will make a test with smallest sample buffer possible - just 16 samples… bigger buffers shows the same picture

Testcase 1 - IDLE load
Idle scenario - no load at system - sampling frequency 96Khz, 24 bit, buffer size 16 samples

root@Khadas:~# ./test 96000 16
.....
nsec: 4637152559669  - Waited for IRQ for 166µs, IRQ time = 166µs, retcode = 0
nsec: 4637152726336  - Waited for IRQ for 166µs, IRQ time = 166µs, retcode = 0
nsec: 4637152893002  - Waited for IRQ for 166µs, IRQ time = 166µs, retcode = 0
nsec: 4637153059669  - Waited for IRQ for 166µs, IRQ time = 166µs, retcode = 0
nsec: 4637153226336  - Waited for IRQ for 166µs, IRQ time = 166µs, retcode = 0
nsec: 4637153393002  - Waited for IRQ for 166µs, IRQ time = 166µs, retcode = 0
nsec: 4637153559669  - Waited for IRQ for 166µs, IRQ time = 166µs, retcode = 0
nsec: 4637153726336  - Waited for IRQ for 166µs, IRQ time = 166µs, retcode = 0
nsec: 4637153893002  - Waited for IRQ for 166µs, IRQ time = 166µs, retcode = 0
nsec: 4637154059669  - Waited for IRQ for 166µs, IRQ time = 166µs, retcode = 0
nsec: 4637154226336  - Waited for IRQ for 166µs, IRQ time = 166µs, retcode = 0
nsec: 4637154393002  - Waited for IRQ for 166µs, IRQ time = 166µs, retcode = 0
nsec: 4637154559711  - Waited for IRQ for 166µs, IRQ time = 166µs, retcode = 0
nsec: 4637154726336  - Waited for IRQ for 166µs, IRQ time = 166µs, retcode = 0
nsec: 4637154893002  - Waited for IRQ for 166µs, IRQ time = 166µs, retcode = 0
nsec: 4637155059669  - Waited for IRQ for 166µs, IRQ time = 166µs, retcode = 0
nsec: 4637155226336  - Waited for IRQ for 166µs, IRQ time = 166µs, retcode = 0
nsec: 4637155393002  - Waited for IRQ for 166µs, IRQ time = 166µs, retcode = 0
nsec: 4637155559669  - Waited for IRQ for 166µs, IRQ time = 166µs, retcode = 0
User process minumum wait time = 163µs,  maximum wait time = 170µs, average = 166µs
IRQ minumum wait time = 166µs,  maximum wait time = 166µs, average = 166µs

Wow!!! Drift of times is not bigger than 4 microseconds… Audio in latency is just 0.166 milliseconds … no interrupt or buffer overrun loss…

Tescase 2: Light load on the system with several realtime threads running

root@Khadas:/opt/evl/bin# ./hectic 
== Testing FPU check routines...
== FPU check routines: OK.
== Threads: sleeper_ufps1-0 rtk1-1 rtk1-2 rtup1-3 rtup1-4 rtup_ufpp1-5 rtup_ufpp1-6 rtus1-7 rtus1-8 rtus_ufps1-9 rtus_ufps1-10 rtuo1-11 rtuo1-12 rtuo_ufpp1-13 rtuo_ufpp1-14 rtuo_ufps1-15 rtuo_ufps1-16 rtuo_ufpp_ufps1-17 rtuo_ufpp_ufps1-18 sleeper_ufps2-0 rtk2-1 rtk2-2 rtup2-3 rtup2-4 rtup_ufpp2-5 rtup_ufpp2-6 rtus2-7 rtus2-8 rtus_ufps2-9 rtus_ufps2-10 rtuo2-11 rtuo2-12 rtuo_ufpp2-13 rtuo_ufpp2-14 rtuo_ufps2-15 rtuo_ufps2-16 rtuo_ufpp_ufps2-17 rtuo_ufpp_ufps2-18 sleeper_ufps3-0 rtk3-1 rtk3-2 rtup3-3 rtup3-4 rtup_ufpp3-5 rtup_ufpp3-6 rtus3-7 rtus3-8 rtus_ufps3-9 rtus_ufps3-10 rtuo3-11 rtuo3-12 rtuo_ufpp3-13 rtuo_ufpp3-14 rtuo_ufps3-15 rtuo_ufps3-16 rtuo_ufpp_ufps3-17 rtuo_ufpp_ufps3-18
RTT|  00:00:01
RTH|---------cpu|ctx switches|-------total
RTD|           3|       16359|       16359
RTD|           2|       16414|       16414
RTD|           1|       16361|       16361
RTD|           3|       16414|       32773
RTD|           1|       16416|       32777
RTD|           2|       16416|       32830
RTD|           3|       16363|       49136
RTD|           2|       16416|       49246
RTD|           1|       16416|       49193
RTD|           3|       16414|       65550
RTD|           1|       16414|       65607
RTD|           2|       16416|       65662

And results are

root@Khadas:~# ./test 96000 16
.....

nsec: 4883259795328  - Waited for IRQ for 166µs, IRQ time = 166µs, retcode = 0
nsec: 4883259962786  - Waited for IRQ for 169µs, IRQ time = 167µs, retcode = 0
nsec: 4883260129453  - Waited for IRQ for 165µs, IRQ time = 166µs, retcode = 0
nsec: 4883260296203  - Waited for IRQ for 166µs, IRQ time = 166µs, retcode = 0
nsec: 4883260462036  - Waited for IRQ for 165µs, IRQ time = 165µs, retcode = 0
nsec: 4883260628661  - Waited for IRQ for 166µs, IRQ time = 166µs, retcode = 0
nsec: 4883260795370  - Waited for IRQ for 166µs, IRQ time = 166µs, retcode = 0
nsec: 4883260962036  - Waited for IRQ for 166µs, IRQ time = 166µs, retcode = 0
nsec: 4883261128661  - Waited for IRQ for 167µs, IRQ time = 166µs, retcode = 0
nsec: 4883261295328  - Waited for IRQ for 166µs, IRQ time = 166µs, retcode = 0
nsec: 4883261462536  - Waited for IRQ for 167µs, IRQ time = 167µs, retcode = 0
nsec: 4883261628703  - Waited for IRQ for 165µs, IRQ time = 166µs, retcode = 0
nsec: 4883261795328  - Waited for IRQ for 166µs, IRQ time = 166µs, retcode = 0
nsec: 4883261961995  - Waited for IRQ for 166µs, IRQ time = 166µs, retcode = 0
nsec: 4883262128661  - Waited for IRQ for 167µs, IRQ time = 166µs, retcode = 0
User process minumum wait time = 161µs,  maximum wait time = 171µs, average = 166µs
IRQ minumum wait time = 163µs,  maximum wait time = 169µs, average = 166µs

Drift is ±5 microseconds - notice that interrupt time started to float, but that is still damn good values

Testcase 3: High load with process contact switch at maximum rate (every 3 microsecond)

root@Khadas:/opt/evl/bin# ./hectic -s 3
== Testing FPU check routines...
== FPU check routines: OK.
== Threads: switcher_ufps1-0 rtk1-1 rtk1-2 rtup1-3 rtup1-4 rtup_ufpp1-5 rtup_ufpp1-6 rtus1-7 rtus1-8 rtus_ufps1-9 rtus_ufps1-10 rtuo1-11 rtuo1-12 rtuo_ufpp1-13 rtuo_ufpp1-14 rtuo_ufps1-15 rtuo_ufps1-16 rtuo_ufpp_ufps1-17 rtuo_ufpp_ufps1-18 fpu_stress_ufps1-19 switcher_ufps2-0 rtk2-1 rtk2-2 rtup2-3 rtup2-4 rtup_ufpp2-5 rtup_ufpp2-6 rtus2-7 rtus2-8 rtus_ufps2-9 rtus_ufps2-10 rtuo2-11 rtuo2-12 rtuo_ufpp2-13 rtuo_ufpp2-14 rtuo_ufps2-15 rtuo_ufps2-16 rtuo_ufpp_ufps2-17 rtuo_ufpp_ufps2-18 fpu_stress_ufps2-19 switcher_ufps3-0 rtk3-1 rtk3-2 rtup3-3 rtup3-4 rtup_ufpp3-5 rtup_ufpp3-6 rtus3-7 rtus3-8 rtus_ufps3-9 rtus_ufps3-10 rtuo3-11 rtuo3-12 rtuo_ufpp3-13 rtuo_ufpp3-14 rtuo_ufps3-15 rtuo_ufps3-16 rtuo_ufpp_ufps3-17 rtuo_ufpp_ufps3-18 fpu_stress_ufps3-19
RTT|  00:00:01
RTH|---------cpu|ctx switches|-------total
RTD|           2|       57454|       57454
RTD|           1|       45484|       45484
RTD|           3|       54946|       54946
RTD|           2|       57114|      114568
RTD|           1|       45771|       91255
RTD|           3|       57000|      111946
RTD|           2|       57114|      171682
RTD|           1|       45771|      137026
RTD|           3|       57057|      169003
RTD|           2|       57114|      228796
RTD|           1|       45657|      182683
RTD|           3|       56886|      225889
RTD|           2|       57342|      286138
RTD|           1|       45602|      228285
RTD|           3|       57000|      282889
RTD|           2|       57228|      343366

And test results are:

./test 96000 16
.....
nsec: 5159810749877  - Waited for IRQ for 166µs, IRQ time = 167µs, retcode = 0
nsec: 5159810915752  - Waited for IRQ for 167µs, IRQ time = 165µs, retcode = 0
nsec: 5159811082377  - Waited for IRQ for 164µs, IRQ time = 166µs, retcode = 0
nsec: 5159811249085  - Waited for IRQ for 168µs, IRQ time = 166µs, retcode = 0
nsec: 5159811417252  - Waited for IRQ for 168µs, IRQ time = 168µs, retcode = 0
nsec: 5159811582627  - Waited for IRQ for 165µs, IRQ time = 165µs, retcode = 0
nsec: 5159811749377  - Waited for IRQ for 165µs, IRQ time = 166µs, retcode = 0
nsec: 5159811915960  - Waited for IRQ for 167µs, IRQ time = 166µs, retcode = 0
nsec: 5159812082793  - Waited for IRQ for 166µs, IRQ time = 166µs, retcode = 0
nsec: 5159812250293  - Waited for IRQ for 167µs, IRQ time = 167µs, retcode = 0
nsec: 5159812416043  - Waited for IRQ for 163µs, IRQ time = 165µs, retcode = 0
nsec: 5159812582502  - Waited for IRQ for 167µs, IRQ time = 166µs, retcode = 0
nsec: 5159812748877  - Waited for IRQ for 166µs, IRQ time = 166µs, retcode = 0
nsec: 5159812916877  - Waited for IRQ for 169µs, IRQ time = 168µs, retcode = 0
nsec: 5159813082502  - Waited for IRQ for 163µs, IRQ time = 165µs, retcode = 0
nsec: 5159813250918  - Waited for IRQ for 169µs, IRQ time = 168µs, retcode = 0
nsec: 5159813415877  - Waited for IRQ for 164µs, IRQ time = 164µs, retcode = 0
User process minumum wait time = 156µs,  maximum wait time = 175µs, average = 166µs
IRQ minumum wait time = 158µs,  maximum wait time = 173µs, average = 166µs

Delta microseconds went to ± 10 microseconds and it is still excellent !!!

Testcase 4: Highest load plus CPU cache wiping out
Same as previous but I’m adding CPU cache line wiping using dd if=/dev/zero of=/dev/null command. It will run only on inband kernel CPUS

And results are:

root@Khadas:~# ./test 96000 16
......
sec: 5383588888358  - Waited for IRQ for 163µs, IRQ time = 166µs, retcode = 0
nsec: 5383589055150  - Waited for IRQ for 171µs, IRQ time = 166µs, retcode = 0
nsec: 5383589221025  - Waited for IRQ for 164µs, IRQ time = 165µs, retcode = 0
nsec: 5383589390567  - Waited for IRQ for 169µs, IRQ time = 169µs, retcode = 0
nsec: 5383589556025  - Waited for IRQ for 161µs, IRQ time = 165µs, retcode = 0
nsec: 5383589722233  - Waited for IRQ for 165µs, IRQ time = 166µs, retcode = 0
nsec: 5383589888067  - Waited for IRQ for 166µs, IRQ time = 165µs, retcode = 0
nsec: 5383590054775  - Waited for IRQ for 168µs, IRQ time = 166µs, retcode = 0
nsec: 5383590221317  - Waited for IRQ for 164µs, IRQ time = 166µs, retcode = 0
nsec: 5383590388150  - Waited for IRQ for 167µs, IRQ time = 166µs, retcode = 0
nsec: 5383590554858  - Waited for IRQ for 172µs, IRQ time = 166µs, retcode = 0
nsec: 5383590720942  - Waited for IRQ for 160µs, IRQ time = 166µs, retcode = 0
nsec: 5383590888817  - Waited for IRQ for 167µs, IRQ time = 167µs, retcode = 0
nsec: 5383591054567  - Waited for IRQ for 167µs, IRQ time = 165µs, retcode = 0
nsec: 5383591221858  - Waited for IRQ for 165µs, IRQ time = 167µs, retcode = 0
nsec: 5383591388567  - Waited for IRQ for 165µs, IRQ time = 166µs, retcode = 0
User process minumum wait time = 146µs,  maximum wait time = 186µs, average = 166µs
IRQ minumum wait time = 149µs,  maximum wait time = 183µs, average = 166µs

Delta of changes became bigger but doesn’t exceeds ±20 microseconds… Hell yeah!!! Excelent

Summing up - yes it works… even in rough conditions with 16 sample size buffer, no loss of interrupts, and timing are great. In reality nobody is using such small sample buffer… just for understanding - 10 milliseconds audio delay isn’t hearable at all…

So running that test with full load with realistic buffer size of 256 samples and 48Khz sampling frequency (giving 5,3ms input latency and 10,6ms round trip latency)

root@Khadas:~# ./test 48000 256
.......
nsec: 5680108422583  - Waited for IRQ for 5338µs, IRQ time = 5331µs, retcode = 0
nsec: 5680113756708  - Waited for IRQ for 5332µs, IRQ time = 5334µs, retcode = 0
nsec: 5680119088750  - Waited for IRQ for 5331µs, IRQ time = 5332µs, retcode = 0
nsec: 5680124423333  - Waited for IRQ for 5335µs, IRQ time = 5334µs, retcode = 0
nsec: 5680129756750  - Waited for IRQ for 5335µs, IRQ time = 5333µs, retcode = 0
nsec: 5680135090000  - Waited for IRQ for 5327µs, IRQ time = 5333µs, retcode = 0
nsec: 5680140423791  - Waited for IRQ for 5336µs, IRQ time = 5333µs, retcode = 0
nsec: 5680145754875  - Waited for IRQ for 5325µs, IRQ time = 5331µs, retcode = 0
nsec: 5680151091000  - Waited for IRQ for 5338µs, IRQ time = 5336µs, retcode = 0
nsec: 5680156424041  - Waited for IRQ for 5338µs, IRQ time = 5333µs, retcode = 0
nsec: 5680161757166  - Waited for IRQ for 5329µs, IRQ time = 5333µs, retcode = 0
nsec: 5680167087958  - Waited for IRQ for 5334µs, IRQ time = 5330µs, retcode = 0
nsec: 5680172422000  - Waited for IRQ for 5331µs, IRQ time = 5334µs, retcode = 0
nsec: 5680177760750  - Waited for IRQ for 5338µs, IRQ time = 5338µs, retcode = 0
nsec: 5680183089791  - Waited for IRQ for 5331µs, IRQ time = 5329µs, retcode = 0
User process minumum wait time = 5313µs,  maximum wait time = 5353µs, average = 5333µs
IRQ minumum wait time = 5319µs,  maximum wait time = 5345µs, average = 5333µs

±20 microseconds latency drifting play zero role when we are dealing with milliseconds

That is unreachable with stock Linux kernel… and light years unreachable on Windows

Next step is to polish driver code… and start porting DAW software

hyphop · January 8, 2022, 1:20am

wow wow wow its will amazing

OldNavi · January 22, 2022, 9:04am

Chapter III — «Sensei, make me Sushi please»

Well, I spent a lot of time polishing realtime audio driver. I’ve added a possibility to output signal not just to DAC attached, but using sharebuffer feature of Amlogic SOC sound system - now my driver also capable to output audio through SPDIF and HDMI interfaces… I must admit that this option also available in stock drivers - so it is just a small enhancement for my driver.

Having working realtime audio driver it came a time to work on really interesting things…
I need to remind about ELK audio project https://elk.audio from which I’ll take the most of the software. They are using Xenomai 3 kernel on RPI 3 and 4. And their software rely on that. Now I have to bump their software to upper level - to use EVL project (upcoming Xenomai 4) kernel - which I’ve ported to Khadas HW before…

First thing I had to do - is port their userspace library RASPA (GitHub - elk-audio/raspa: Userspace library to access RTDM audio driver in Elk Audio OS) to work with my realtime audio driver. Not just straightforward - I’m adding additional features (as their driver can’t work with different frequencies and buffer rates), plus an ablity to turn on/off digital outputs (SPDIF and HDMI).
That took a while - to adopt this because it was relaying on Xenomai 3 and I’m using libevl now (Xenomai 4) and they are pretty different in terms of handling realtime threads. However port was successful.

Next in the long todo list - was to port Twine (GitHub - elk-audio/twine: Thread and Worker Interface for Elk Audio OS)- a simple framework to wrap around realtime thread management. That was pretty straightforward until I tried to run…. Had to spend sometime working out differences in RT kernel handling between Xenomai 3 and newer one. However it works now.

Another piece of software required to be added - was RTPMidi project (rtpMIDI | Tobias Erichsen) which adds MIDI over IP functionality into the project. This feature natively supported by Apple and built into the OSes. Worked like charm - just compiled it. However I had to enable Bonjour (zerconf ) via additional AVAHI daemon.

So next port was Sensei (GitHub - elk-audio/sensei: Sensors & control abstraction daemon for Elk Audio OS.) - nice program which works with hardware backends to control things like buttons, leds, encoders and so on. However I’ll have to write my backend for Atmega128 controller which will control all hardware inputs like buttons, leds, encoders and NeoPixel stripe (which is planned to be a VU meter) - that gonna be in the next chapters.

Last but not the least - is a Sushi (GitHub - elk-audio/sushi: Elk Audio OS Plugin host and DAW) - headless digital audio workstation (DAW) piece of software. It has no GUI at all and is controlled by simple json configuration file in which you can defined different tracks, plugins, software instrument ( VST2 and VST3 plugin formats supported). It also supports Open Sound Control protocol - which allows to control in over network using different programs supporting OSC protocol (like TouchOSC and etc). It has also GPRC protocol supported - so it has virtually unlimited functionality of controlling it. You can write simple python or QT GUI shell and control all aspects of DAW.

So better to see once, rather than read 10 times about it. So here a short video (sorry for not good quality). I’ve loaded a Sushi config with Simple Electronic Piano DX10 plugin with Delay and Reverb plugin from mda plugin suite - http://mda.smartelectronix.com - (I’ve compiled for Sushi) and turned a demo song built into the midi keyboard.

Next tasks from a big list - is to model a case to fit all components, write firmware for Atmega128 controller (to control buttons and etc).

I really hope I’ll get VIM3 pro free (no luck so far) which should be more suitable and powerful for DAW.

Frank · January 23, 2022, 1:05am

@OldNavi Our colleagues should have sent you a coupon, have you received it? You can use that coupon to buy VIM3. We will be on vacation in two days

OldNavi · January 23, 2022, 7:27am

It is problem with PayPal. When I try to pay through it - it rejects a payment with “In accordance with the requirements of international law, this operation was rejected.” message. So I can’t make an order from your shop directly and use a coupon.

numbqq · January 23, 2022, 9:20am

Hello @OldNavi

Have you tried nowadays? We have a system upgrade some days ago so the payment maybe abnormal, but it is OK now. Can you try again?

OldNavi · January 23, 2022, 9:37am

Hi @numbqq

yes. I was aware about upgrade - before it was an issue with you site and PayPal integration. But in anycase it doesn’t work for me now too… I’ve tried just now - it still says that message “In accordance with the requirements of international law, this operation was rejected.” - perhaps it is an issue not on your side.

Frank · January 23, 2022, 10:01am

@OldNavi You can contact the email address of the colleague who contacted you about this sample before, explain the situation to her, and she will solve it for you through other methods

Vladimir.v.v · January 23, 2022, 8:35pm

Привет ,я так понимаю ты из России.Скажу так,на своём личном опыте ,у меня не было проблем с Khadas через PayPal.
Возможно тебе так же будет удобно воспользоваться купоном через AliExpress.

OldNavi · January 26, 2022, 1:39pm

Small interlude between chapters.

Whilst waiting for VIM3 Pro and for replacement all of sudden broken Atmega128 (which I was planning to use as HW controller) - I’ve decided to go with case design for my DAW/AMP/CAB/Whatever.
As I really love old style boxes - I bought some old fashioned switches from 80’s, rectangular leds, encoders and etc… And decided somehow combine them together…

After tons of attempts, sketches and other design work - I ended up with this one… That’s process is really hard - a lot of things needs to be considered, like case wouldn’t fit into the 3D printing area, so it has to be split into the smaller details… Some printing forms should be better avoid and so on.

So I expect my case will be the following

Arraging VIM3 + New M2X board

To be continued…

hyphop · February 3, 2022, 10:16am

nice and look like same as motu products

OldNavi · February 20, 2022, 5:01pm

Chapter IV - «Microcontrollers are everywhere»

My VIM3Pro is on the way to myself, many thanks to Khadas team, and specially to @sunny - did a really great job to send me a free sample, as PayPal (sic!) isn’t working for me, even it was in use since 2013…

I’ve done with body design and started a print process…

That’s gonna be long process, but I’ll devote separate chapter to it.

When we have to deal with Hardware stuff - I can’t imagine any piece of it without use of microcontrollers, almost any SBC have it on board. They are everywhere In my project - I’m using them too.

As my SBC is using mainline kernel, I’ve not output from TS050 screen (but it is added into project config) yet, due to the not-yet ready drivers from MIPI-DSI and so on, but I really hope that Neil @narmstrong will again do the brilliant job and will make them soon.

In any case I had a VU meters in initial design of a DAW/CAB project - but was thinking that it will do only dumb thing of controlling input signal. But due to the silent TS050 display - I’ve decided to make it more functional.

At the beginning I was planning to use just one Amtel Atmega128 chip clocked with 12Mhz, but after counting pin budget and amount of sensor I need to handle realtime - I made a decision to have an extra controller to deal with VU meters (based on NeoPixel leds ) and I’m using STM32F0x. Chip for that purpose.

At overall with 2 MCUs I need to control the following list:

1/ 12 Buttons - have to use them through ADC keyboard scan.
2/ 12 Single color leds
3/ 6 2-way Encoders with built-in button
4/ 6 2-color Leds
5/ 64 Neopixel leds stripe

All of them require 1 + 12 + 12 + 12 = 25 pins at least, so smaller Atmel chips doesn’t fit into the pins budget… + I need communication pins and so on… Firmware for Atmega is in develop mode now - but it won’t take much time… more time will go for soldering and assembling all together.

As for VU meter (64 neopixel leds total divided into 2 strips of 32 leds) - STM32 is better choice because of performance, even that small chip (4K RAM and 16K Flash) is running at 48Mhz, 4 times faster than AVR chips, and I’ve requirement to reach at least 30FPS of led refresh rate.
But (there is always but) - it is really memory limited - so I had no luck to use modern HAL frameworks from STM32Cube… as it by default eats half of the RAM for it’s own needs…
I had to write with pure CMSIS using register configuring, ISR designing and so on.

So, despite of very limited number of pins on STM32 chip, I had a luck to fit all I need:

1/ 4 ADC channels to control stereo input and output
2/ 1 pin to control NeoPixel leds chained
3/ 2 pins for I2C bus - yep, I prefer this bus, over UART, cause in I can run it in Fast Mode on 400Khz.

Very nice thing that STM32 chips have built-in DMA controller which is good for exact timing required to send data to neopixel leds as it very time exact bound protocol - no interrupts allowed whilst sending data to leds chain, 50 mS delay between transfers - do a reset…. It is possible to do that with Atmega - but then it won’t be capable to do anything else useful ( as it need good timings for encoders scan).

So just instead of using 2 by 32 stripes as initially thought - I wrote a firmware capable to do the following

1/. Use layout technique - ie - I can have 2 x 32 led stripes, or I can have I 32 led plus 2 16 leds stripes (either on top physical stripe or bottom). Or I can have 4 x 16leds stripes
2/ I can control a direction where VU meter show grow either left or right
3/ I can define starting color point, peak meter point
4/ I can control color and intensity of any cell for
5/ I can have a raw access (not attached to any ADC ) of any layout stripe - that is needed for DAW - for example I can use any stripe to show a pan/balance point, or use it as tuner, or show CPU usage — whatever I want
6/ I’ve control of over leds framwbuffer in direct mode - setting any led with their corresponding RGB value.
7/ every data is send through I2C line at 400khz fast mode
8/ leds have 30FPS refresh rate

As the next step - it will need to write a daemon or adopt sensei software to have a control over all of that…. And make it fully configurable for user needs.
As example, pushing one button will change sources and layout of VU meter, some LEDs can react to BPM pulse, encoders will control any plugins parameter and so on…

But that is for the next chapters….

PS: Whilst was writing this post - first detail finished it’s printing

PPS: Need to write some update to the first project… how could i forget about it

Frank · February 21, 2022, 12:37am

@OldNavi Great to see you sharing your creative process here. I can’t wait to see the final piece

OldNavi · February 21, 2022, 6:59am

Same here… I also wanna see a final piece. Some design mix of modern LCD and buttons from 80-s.

Almost complete from panel look

OldNavi · March 11, 2022, 3:32pm

Chapter V - " Life is like a box of chocolate"

4MO4vl0Jed0

Or won’t get at all. My VIM3 for the project won’t reach me, that’s sad news cause UPS is dead in my country. Thanks @sunny for helping me with that but seems there are things we can’t control at all.

Anyway, I’ll try to finish this project with a hardware I already have, despite it might not be powerful enough.

I’ve finished a design, and decided to give a name to the project - it is now
F.A.W.N² which stands for Flexible Audio Workstation with Neural Network. (so N x N = N²)
And I can use a fawn as a my own logo

So now it is time for printing and assembling… this is kind of iteration process and half of the material goes to the trash because of some design or printing errors. Need to say that is a bit expensive cause I’m using engineering plastics which is based on ABS plastic armoured with 4% of glass and that required a set of modifications to my 3D printer. And of course that material has much higher price that ordinary ABS plastic. Needles to say - that printing at high quality takes a lot of time…

Anyway I’m progressing here…

Main body with partial electronics mounted.

Some side views of half assembly

Small update:

Body is almost ready.

To be continued…

OldNavi · April 17, 2022, 11:04am

Chapter VI - «Soldering, Assembly and that’s it»

Long time no see…. Well I have spend numerous hours assembling this all… and as I’m very unskilled in electronics - I did a lot of mistakes and had to redo things at minimum twice, including details 3D reprinting…

Most of issues came from noise feedback into analog chains due to the various reasons… eIther it was a ground loop from all of sudden happening when input jack is in or STM32 processor gave a lot of hiss when it turned their ADC (had to add buffering amplifier into the chain)… Or EMI is making SPDIF to go out of synchronisation…. Some of the problems were resolved or temporary avoided by disabling some functionality - at the modem VU-Meter is only working in «digital mode» showing values coming from sound processor.

Here are some pictures of assembly.

And finally all together

After assembly checked if TS050 works with android image…

And example when LEDs are blinking - I’m getting around -50db noise feedback into input amplifier because of some ground loop happens when I short down one of input channel to the ground via mono jack plugged in.

Anyway this gonna be resolved, at least I hope, by means of new and redesigned PSU (power supply unit) which I’m going to make «smart». It will be able to monitor state of the VIM board and initiate normal shutdown procedure on power button press, just like usual computers do with ACPI power button. Also it will receive galvanically isolated DC-DC convertors and that, I hope will eliminate any ground loops in analog domain.

How I wish that Khadas will engineer some new and simple «Tone board with ADC/DAC - without any XMOS codec (it doesn’t needed actually) as a «hat» on top of SBC like Raspberry PI has… That could be very interesting…

And decided to write a small demo with my humble guitar mumbling - of how it is started to sound…

Next chapter will be about love and hate in C++

To be continued…