Painlessly usable Linux distro?

tkaiser · November 13, 2018, 5:25pm

Hmm… Average load is constantly around 1.0 which prevents my sbc-bench from starting (checks average load and when above 0.1 then the system is too busy to be considered for benchmarking)

kworker seems to be the culprit, unfortunately your kernel lacks the necessary settings to diagnose with iotop:

root@Khadas:~# iotop
Could not run iotop as some of the requirements are not met:
- Linux >= 2.6.20 with
  - I/O accounting support (CONFIG_TASKSTATS, CONFIG_TASK_DELAY_ACCT, CONFIG_TASK_IO_ACCOUNTING)

Same with perf – no appropriate linux-tools version available.

The task sits on cpu1 and consumes around 10% of the CPU (overall reported by top is 7-7.5% CPU utilization). Do you have an idea what could be causing this?

Gouwa · November 13, 2018, 5:16pm

@tkaiser it’s mid-might here in China now, @numbqq will follow up and update here tomorrow

numbqq · November 14, 2018, 6:22am

Hi tkaiser,

When I enable the configurations iotop works:

I found some errors in /var/log/syslog

Nov 14 06:19:38 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:39 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:39 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:40 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:40 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:41 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:41 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:42 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:42 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:43 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:43 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:44 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:44 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:45 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:45 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:46 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:46 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:47 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:47 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:48 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:48 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:49 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:49 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:51 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:51 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:52 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:52 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:53 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:53 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:54 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:54 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*

Maybe this is the cause? I will look into this…

tkaiser · November 14, 2018, 8:27am

Seem not related. I stopped everything with lirc in its name but still kworker is consuming CPU cycles:

root@Khadas:~# systemctl list-unit-files | grep lirc
lircd-setup.service                        disabled       
lircd-uinput.service                       disabled       
lircd.service                              disabled       
lircmd.service                             disabled       
lircd.socket                               disabled

numbqq · November 15, 2018, 2:51am

Hello tkaiser,

I build a new version V181115, it seems that the CPU usage is normal, can you check it?

top - 02:51:37 up 2 min,  1 user,  load average: 0.19, 0.15, 0.06
Tasks: 198 total,   1 running, 196 sleeping,   0 stopped,   1 zombie
%Cpu(s):  1.7 us,  1.7 sy,  0.0 ni, 94.0 id,  2.5 wa,  0.0 hi,  0.1 si,  0.0 st
KiB Mem :  1945096 total,  1552188 free,   154052 used,   238856 buff/cache
KiB Swap:   972544 total,   972544 free,        0 used.  1743788 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND     
 1258 root      20   0    7700   3500   2848 R  19.0  0.2   0:00.11 top         
    1 root      20   0  161020   7988   5728 S   0.0  0.4   0:03.56 systemd     
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.01 kthreadd    
    3 root      20   0       0      0      0 S   0.0  0.0   0:00.06 ksoftirqd/0 
    4 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0 
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:+ 
    6 root      20   0       0      0      0 S   0.0  0.0   0:00.60 kworker/u1+ 
    7 root      20   0       0      0      0 S   0.0  0.0   0:00.21 rcu_sched   
    8 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh      
    9 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 migration/0 
   10 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 watchdog/0  
   11 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 watchdog/1  
   12 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 migration/1 
   13 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/1 
   14 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/1:0 
   15 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/1:+ 
   16 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 watchdog/2

Thanks.

tkaiser · November 15, 2018, 12:11pm

Yes, the latest version has fixed the issue. Now starting to benchmark after bringing rock64_fix_performance.sh into place and adjusting DT (allowing for 2.0/1.5GHz and higher throttling tresholds):

root@Khadas:/boot/dtb# diff *.dts
864c864
< 					temperature = <0x11170>;
---
> 					temperature = <0x14c08>;
871c871
< 					temperature = <0x14c08>;
---
> 					temperature = <0x17318>;
3662a3663,3672
> 
> 		opp-1512000000 {
>                         opp-hz = <0x0 0x5a1f4a00>;
>                         opp-microvolt = <0x124f80>;
>                         opp-microvolt-L0 = <0x124f80>;
>                         opp-microvolt-L1 = <0x11edd8>;
>                         opp-microvolt-L2 = <0x118c30>;
>                         opp-microvolt-L3 = <0x112a88>;
>                         clock-latency-ns = <0x9c40>;
>                 };
3764a3775,3785
> 
> 		opp-1992000000 {
> 			opp-hz = <0x0 0x76bb8200>;
>                         opp-microvolt = <0x13d620>;
>                         opp-microvolt-L0 = <0x13d620>;
>                         opp-microvolt-L1 = <0x137478>;
>                         opp-microvolt-L2 = <0x1312d0>;
>                         opp-microvolt-L3 = <0x12b128>;
>                         clock-latency-ns = <0x9c40>;
>                 };
>

tkaiser · November 15, 2018, 12:58pm

BTW: IMO you should switch to CONFIG_HZ_250 (see reasons).

Gouwa · November 15, 2018, 1:56pm

If I don’t get wrong, RK3399 SoC can only run 1.8GHz and 1.5GHz, only the Chromebook version RK3399C can up to 2.0GHz of bigger cluster.

Surely, you can still find out a few RK3399 SoC can run 2.0GHz, as both RK3399 and RK3399C are same thing, the only different is that when Rockchip filter the SoC, good ones label as RK3399C, and not so good one with RK3399.

Good day!

tkaiser · November 15, 2018, 2:06pm

I understood something different: RK filters out those SoCs as RK3399C/OP1 that run reliably at lower DVFS operating points so they are able to clock them up to 2GHz in their Chromebook without wasting too much energy and generating too much heat. An RK3399C at 2.0GHz will stay as cool or even cooler as an RK3399 at 1.8GHz.

I simply used unlock-temperature.patch and overclock-rk3399-to-1.5-2.0.patch and the latter of course uses higher DVFS operating points therefore leading to higher consumption and temperatures under load.

Please note overclock-rk3399-to-1.8-2.2.patch.disabled some users activate but I don’t consider running RK3399 with A53 cores at 1.8GHz and A72 at 2.2GHz a sane default. But with appropriate cooling it also works: https://twitter.com/hjc4869/status/1006849763067621378

tkaiser · November 15, 2018, 2:55pm

@numbqq and I would also suggest applying this patch (see reasons)

numbqq · November 15, 2018, 4:28pm

Hello tkaiser,

You can check the result here:

Retest with -c option. Will update here tomorrow.

numbqq · November 16, 2018, 3:23am

OK. Seems that you have already upload the result to your repo. Thanks.

tkaiser · November 16, 2018, 6:47am

If you rebuilt the kernel (especially with CONFIG_HZ_250) and numbers differ I would be happy to include another result to the listing. So please share

numbqq · November 16, 2018, 2:14pm

You can check the result here:

sbc-bench v0.6.2

Installing needed tools. This may take some time... Done.
Checking cpufreq OPP... Done.
Executing tinymembench. This will take a long time... Done.
Executing OpenSSL benchmark. This will take 3 minutes... Done.
Executing 7-zip benchmark. This will take a long time...+ Done.
Executing cpuminer. This will take 5 minutes... Done.
Checking cpufreq OPP... Done.


Memory performance (big.LITTLE cores measured individually):
memcpy: 1312.5 MB/s 
memset: 4809.7 MB/s 
memcpy: 2825.3 MB/s 
memset: 4907.6 MB/s (0.5%)

Cpuminer total scores (5 minutes execution): 10.61,10.60,10.55,10.53,10.49,10.44,10.41,10.38,10.37,10.36,10.35,10.33,10.32,10.31,10.30,10.29,10.28,10.26 kH/s

7-zip total scores (3 consecutive runs): 6444,6468,6465

OpenSSL results (big.LITTLE cores measured individually):
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-128-cbc     135585.72k   400632.06k   765320.11k  1021268.65k  1131615.57k  1138731.69k
aes-128-cbc     403032.81k   906390.44k  1292418.65k  1423920.13k  1495247.53k  1495979.35k
aes-192-cbc      96385.22k   286495.02k   558528.51k   753120.60k   839611.73k   848134.14k
aes-192-cbc     376892.10k   829469.82k  1097246.98k  1259299.16k  1310932.99k  1320004.27k
aes-256-cbc     125190.33k   327254.49k   540443.73k   658430.63k   703100.25k   705779.03k
aes-256-cbc     370017.61k   753589.61k   998980.01k  1088732.16k  1125856.60k  1126438.23k

Full results uploaded to http://ix.io/1s6C. Please check the log for anomalies (e.g. swapping
or throttling happenend) and otherwise share this URL.

tkaiser · November 17, 2018, 2:03am

Thank you. But that’s still with CONFIG_HZ_1000, right?

numbqq · November 17, 2018, 2:07am

I have set CONFIG_HZ=250.

tkaiser · November 17, 2018, 10:36pm

That’s really strange since when I tested with CONFIG_HZ=1000 I got similar results wrt memory performance: https://github.com/ThomasKaiser/sbc-bench/blob/master/Results.md

According to ayufan the way lower memory performance with Rockchip’s BSP kernel compared to mainline is related to CONFIG_HZ and switching from 1000 to 250 fixed it for him. Hmm… I need to test with my RockPro64 tomorrow…

numbqq · November 22, 2018, 2:29am

So the result in https://github.com/ThomasKaiser/sbc-bench/blob/master/Results.md for RockPro64 4.4 kernel is CONFIG_HZ=250 ? I found the memory performance is lower then mainline kernel.

tkaiser · November 22, 2018, 12:29pm

Nope, that was also with CONFIG_HZ=1000 – I need to retest soon.

Now that @balbes150 pushed his Armbian changes (using 250Hz and the other tweaks already) I’ll test with his images.

tkaiser · November 22, 2018, 4:27pm

New result with latest ayufan image where he switched to CONFIG_HZ=250 but still at 1.8/1.4GHz:

Memory performance (big.LITTLE cores measured individually):
memcpy: 1800.4 MB/s 
memset: 8313.6 MB/s 
memcpy: 3513.9 MB/s 
memset: 8410.0 MB/s (0.5%)

Vs. Edge/Captain also with CONFIG_HZ=250 but at 2.0/1.5GHz:

Memory performance (big.LITTLE cores measured individually):
memcpy: 1391.1 MB/s (0.2%)
memset: 4821.1 MB/s
memcpy: 2856.5 MB/s (0.3%)
memset: 4882.2 MB/s (0.6%)

Significant change was this and afterwards memory performance was reported at same level as mainline kernel.

Maybe CONFIG_ARM_ROCKCHIP_DMC_DEVFREQ is the culprit? https://github.com/ayufan-rock64/linux-kernel/blob/ed3ce4d15ec1c8721b55fbc6860b44907ea9eba5/arch/arm64/configs/rockchip_linux_defconfig#L1146-L1159 vs. https://github.com/armbian/build/blob/master/config/kernel/linux-rk3399-default.config#L5174-L5179

Interestingly on ayufan’s image /sys/bus/platform/drivers/rockchip-dmc/dmc/ is missing completely…