tkaiser
November 13, 2018, 4:01pm
#5
Hmm… Average load is constantly around 1.0 which prevents my sbc-bench from starting (checks average load and when above 0.1 then the system is too busy to be considered for benchmarking)
kworker seems to be the culprit, unfortunately your kernel lacks the necessary settings to diagnose with iotop:
root@Khadas:~# iotop
Could not run iotop as some of the requirements are not met:
- Linux >= 2.6.20 with
- I/O accounting support (CONFIG_TASKSTATS, CONFIG_TASK_DELAY_ACCT, CONFIG_TASK_IO_ACCOUNTING)
Same with perf – no appropriate linux-tools version available.
The task sits on cpu1 and consumes around 10% of the CPU (overall reported by top is 7-7.5% CPU utilization). Do you have an idea what could be causing this?
Gouwa
November 13, 2018, 5:16pm
#6
@tkaiser it’s mid-might here in China now, @numbqq will follow up and update here tomorrow
1 Like
numbqq
November 14, 2018, 6:13am
#7
Hi tkaiser,
When I enable the configurations iotop works:
I found some errors in /var/log/syslog
Nov 14 06:19:38 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:39 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:39 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:40 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:40 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:41 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:41 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:42 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:42 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:43 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:43 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:44 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:44 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:45 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:45 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:46 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:46 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:47 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:47 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:48 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:48 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:49 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:49 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:51 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:51 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:52 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:52 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:53 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:53 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:54 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:54 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Maybe this is the cause? I will look into this…
tkaiser
November 14, 2018, 8:27am
#8
numbqq:
Maybe this is the cause?
Seem not related. I stopped everything with lirc in its name but still kworker is consuming CPU cycles:
root@Khadas:~# systemctl list-unit-files | grep lirc
lircd-setup.service disabled
lircd-uinput.service disabled
lircd.service disabled
lircmd.service disabled
lircd.socket disabled
numbqq
November 15, 2018, 2:51am
#9
Hello tkaiser,
I build a new version V181115 , it seems that the CPU usage is normal, can you check it?
top - 02:51:37 up 2 min, 1 user, load average: 0.19, 0.15, 0.06
Tasks: 198 total, 1 running, 196 sleeping, 0 stopped, 1 zombie
%Cpu(s): 1.7 us, 1.7 sy, 0.0 ni, 94.0 id, 2.5 wa, 0.0 hi, 0.1 si, 0.0 st
KiB Mem : 1945096 total, 1552188 free, 154052 used, 238856 buff/cache
KiB Swap: 972544 total, 972544 free, 0 used. 1743788 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1258 root 20 0 7700 3500 2848 R 19.0 0.2 0:00.11 top
1 root 20 0 161020 7988 5728 S 0.0 0.4 0:03.56 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:00.06 ksoftirqd/0
4 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:+
6 root 20 0 0 0 0 S 0.0 0.0 0:00.60 kworker/u1+
7 root 20 0 0 0 0 S 0.0 0.0 0:00.21 rcu_sched
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
9 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
10 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
11 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
12 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/1
13 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1
14 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/1:0
15 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:+
16 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/2
Thanks.
1 Like
tkaiser
November 15, 2018, 12:11pm
#10
Yes, the latest version has fixed the issue. Now starting to benchmark after bringing rock64_fix_performance.sh into place and adjusting DT (allowing for 2.0/1.5GHz and higher throttling tresholds):
root@Khadas:/boot/dtb# diff *.dts
864c864
< temperature = <0x11170>;
---
> temperature = <0x14c08>;
871c871
< temperature = <0x14c08>;
---
> temperature = <0x17318>;
3662a3663,3672
>
> opp-1512000000 {
> opp-hz = <0x0 0x5a1f4a00>;
> opp-microvolt = <0x124f80>;
> opp-microvolt-L0 = <0x124f80>;
> opp-microvolt-L1 = <0x11edd8>;
> opp-microvolt-L2 = <0x118c30>;
> opp-microvolt-L3 = <0x112a88>;
> clock-latency-ns = <0x9c40>;
> };
3764a3775,3785
>
> opp-1992000000 {
> opp-hz = <0x0 0x76bb8200>;
> opp-microvolt = <0x13d620>;
> opp-microvolt-L0 = <0x13d620>;
> opp-microvolt-L1 = <0x137478>;
> opp-microvolt-L2 = <0x1312d0>;
> opp-microvolt-L3 = <0x12b128>;
> clock-latency-ns = <0x9c40>;
> };
>
tkaiser
November 15, 2018, 12:58pm
#11
BTW: IMO you should switch to CONFIG_HZ_250 (see reasons ).
Gouwa
November 15, 2018, 1:56pm
#12
If I don’t get wrong, RK3399 SoC can only run 1.8GHz and 1.5GHz, only the Chromebook version RK3399C can up to 2.0GHz of bigger cluster.
Surely, you can still find out a few RK3399 SoC can run 2.0GHz, as both RK3399 and RK3399C are same thing, the only different is that when Rockchip filter the SoC, good ones label as RK3399C, and not so good one with RK3399.
Good day!
1 Like
tkaiser
November 15, 2018, 2:05pm
#13
I understood something different: RK filters out those SoCs as RK3399C/OP1 that run reliably at lower DVFS operating points so they are able to clock them up to 2GHz in their Chromebook without wasting too much energy and generating too much heat. An RK3399C at 2.0GHz will stay as cool or even cooler as an RK3399 at 1.8GHz.
I simply used unlock-temperature.patch and overclock-rk3399-to-1.5-2.0.patch and the latter of course uses higher DVFS operating points therefore leading to higher consumption and temperatures under load.
Please note overclock-rk3399-to-1.8-2.2.patch.disabled some users activate but I don’t consider running RK3399 with A53 cores at 1.8GHz and A72 at 2.2GHz a sane default. But with appropriate cooling it also works: https://twitter.com/hjc4869/status/1006849763067621378
2 Likes
tkaiser
November 15, 2018, 2:55pm
#14
@numbqq and I would also suggest applying this patch (see reasons )
1 Like
numbqq
November 15, 2018, 3:07pm
#15
Hello tkaiser,
You can check the result here:
Retest with -c option. Will update here tomorrow.
numbqq
November 16, 2018, 3:23am
#16
OK. Seems that you have already upload the result to your repo. Thanks.
tkaiser
November 16, 2018, 6:47am
#17
If you rebuilt the kernel (especially with CONFIG_HZ_250 ) and numbers differ I would be happy to include another result to the listing. So please share
numbqq
November 16, 2018, 2:14pm
#18
You can check the result here:
sbc-bench v0.6.2
Installing needed tools. This may take some time... Done.
Checking cpufreq OPP... Done.
Executing tinymembench. This will take a long time... Done.
Executing OpenSSL benchmark. This will take 3 minutes... Done.
Executing 7-zip benchmark. This will take a long time...+ Done.
Executing cpuminer. This will take 5 minutes... Done.
Checking cpufreq OPP... Done.
Memory performance (big.LITTLE cores measured individually):
memcpy: 1312.5 MB/s
memset: 4809.7 MB/s
memcpy: 2825.3 MB/s
memset: 4907.6 MB/s (0.5%)
Cpuminer total scores (5 minutes execution): 10.61,10.60,10.55,10.53,10.49,10.44,10.41,10.38,10.37,10.36,10.35,10.33,10.32,10.31,10.30,10.29,10.28,10.26 kH/s
7-zip total scores (3 consecutive runs): 6444,6468,6465
OpenSSL results (big.LITTLE cores measured individually):
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes
aes-128-cbc 135585.72k 400632.06k 765320.11k 1021268.65k 1131615.57k 1138731.69k
aes-128-cbc 403032.81k 906390.44k 1292418.65k 1423920.13k 1495247.53k 1495979.35k
aes-192-cbc 96385.22k 286495.02k 558528.51k 753120.60k 839611.73k 848134.14k
aes-192-cbc 376892.10k 829469.82k 1097246.98k 1259299.16k 1310932.99k 1320004.27k
aes-256-cbc 125190.33k 327254.49k 540443.73k 658430.63k 703100.25k 705779.03k
aes-256-cbc 370017.61k 753589.61k 998980.01k 1088732.16k 1125856.60k 1126438.23k
Full results uploaded to http://ix.io/1s6C. Please check the log for anomalies (e.g. swapping
or throttling happenend) and otherwise share this URL.
2 Likes
tkaiser
November 17, 2018, 2:03am
#19
Thank you. But that’s still with CONFIG_HZ_1000 , right?
numbqq
November 17, 2018, 2:07am
#20
I have set CONFIG_HZ=250.
tkaiser
November 17, 2018, 10:36pm
#21
numbqq:
I have set CONFIG_HZ=250
That’s really strange since when I tested with CONFIG_HZ=1000 I got similar results wrt memory performance: https://github.com/ThomasKaiser/sbc-bench/blob/master/Results.md
According to ayufan the way lower memory performance with Rockchip’s BSP kernel compared to mainline is related to CONFIG_HZ and switching from 1000 to 250 fixed it for him. Hmm… I need to test with my RockPro64 tomorrow…
numbqq
November 22, 2018, 2:29am
#22
So the result in https://github.com/ThomasKaiser/sbc-bench/blob/master/Results.md for RockPro64 4.4 kernel is CONFIG_HZ=250 ? I found the memory performance is lower then mainline kernel.
tkaiser
November 22, 2018, 12:29pm
#23
Nope, that was also with CONFIG_HZ=1000 – I need to retest soon.
Now that @balbes150 pushed his Armbian changes (using 250Hz and the other tweaks already) I’ll test with his images.
3 Likes
tkaiser
November 22, 2018, 4:26pm
#24
New result with latest ayufan image where he switched to CONFIG_HZ=250 but still at 1.8/1.4GHz:
Memory performance (big.LITTLE cores measured individually):
memcpy: 1800.4 MB/s
memset: 8313.6 MB/s
memcpy: 3513.9 MB/s
memset: 8410.0 MB/s (0.5%)
Vs. Edge/Captain also with CONFIG_HZ=250 but at 2.0/1.5GHz:
Memory performance (big.LITTLE cores measured individually):
memcpy: 1391.1 MB/s (0.2%)
memset: 4821.1 MB/s
memcpy: 2856.5 MB/s (0.3%)
memset: 4882.2 MB/s (0.6%)
Significant change was this and afterwards memory performance was reported at same level as mainline kernel.
Maybe CONFIG_ARM_ROCKCHIP_DMC_DEVFREQ is the culprit? https://github.com/ayufan-rock64/linux-kernel/blob/ed3ce4d15ec1c8721b55fbc6860b44907ea9eba5/arch/arm64/configs/rockchip_linux_defconfig#L1146-L1159 vs. https://github.com/armbian/build/blob/master/config/kernel/linux-rk3399-default.config#L5174-L5179
Interestingly on ayufan’s image /sys/bus/platform/drivers/rockchip-dmc/dmc/ is missing completely…
