tkaiser
November 13, 2018, 4:01pm
5
Hmm… Average load is constantly around 1.0 which prevents my sbc-bench from starting (checks average load and when above 0.1 then the system is too busy to be considered for benchmarking)
kworker seems to be the culprit, unfortunately your kernel lacks the necessary settings to diagnose with iotop:
root@Khadas:~# iotop
Could not run iotop as some of the requirements are not met:
- Linux >= 2.6.20 with
- I/O accounting support (CONFIG_TASKSTATS, CONFIG_TASK_DELAY_ACCT, CONFIG_TASK_IO_ACCOUNTING)
Same with perf – no appropriate linux-tools version available.
The task sits on cpu1 and consumes around 10% of the CPU (overall reported by top is 7-7.5% CPU utilization). Do you have an idea what could be causing this?
Gouwa
November 13, 2018, 5:16pm
6
@tkaiser it’s mid-might here in China now, @numbqq will follow up and update here tomorrow
1 Like
numbqq
November 14, 2018, 6:13am
7
Hi tkaiser,
When I enable the configurations iotop works:
I found some errors in /var/log/syslog
Nov 14 06:19:38 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:39 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:39 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:40 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:40 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:41 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:41 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:42 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:42 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:43 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:43 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:44 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:44 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:45 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:45 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:46 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:46 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:47 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:47 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:48 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:48 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:49 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:49 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:51 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:51 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:52 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:52 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:53 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:53 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:54 localhost lircd[781]: lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Nov 14 06:19:54 localhost lircd-0.10.0[781]: Error: Cannot glob /sys/class/rc/rc0/input[0-9]*/event[0-9]*
Maybe this is the cause? I will look into this…
tkaiser
November 14, 2018, 8:27am
8
numbqq:
Maybe this is the cause?
Seem not related. I stopped everything with lirc in its name but still kworker is consuming CPU cycles:
root@Khadas:~# systemctl list-unit-files | grep lirc
lircd-setup.service disabled
lircd-uinput.service disabled
lircd.service disabled
lircmd.service disabled
lircd.socket disabled
numbqq
November 15, 2018, 2:51am
9
Hello tkaiser,
I build a new version V181115 , it seems that the CPU usage is normal, can you check it?
top - 02:51:37 up 2 min, 1 user, load average: 0.19, 0.15, 0.06
Tasks: 198 total, 1 running, 196 sleeping, 0 stopped, 1 zombie
%Cpu(s): 1.7 us, 1.7 sy, 0.0 ni, 94.0 id, 2.5 wa, 0.0 hi, 0.1 si, 0.0 st
KiB Mem : 1945096 total, 1552188 free, 154052 used, 238856 buff/cache
KiB Swap: 972544 total, 972544 free, 0 used. 1743788 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1258 root 20 0 7700 3500 2848 R 19.0 0.2 0:00.11 top
1 root 20 0 161020 7988 5728 S 0.0 0.4 0:03.56 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:00.06 ksoftirqd/0
4 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:+
6 root 20 0 0 0 0 S 0.0 0.0 0:00.60 kworker/u1+
7 root 20 0 0 0 0 S 0.0 0.0 0:00.21 rcu_sched
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
9 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
10 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
11 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
12 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/1
13 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1
14 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/1:0
15 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:+
16 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/2
Thanks.
1 Like
tkaiser
November 15, 2018, 12:11pm
10
Yes, the latest version has fixed the issue. Now starting to benchmark after bringing rock64_fix_performance.sh into place and adjusting DT (allowing for 2.0/1.5GHz and higher throttling tresholds):
root@Khadas:/boot/dtb# diff *.dts
864c864
< temperature = <0x11170>;
---
> temperature = <0x14c08>;
871c871
< temperature = <0x14c08>;
---
> temperature = <0x17318>;
3662a3663,3672
>
> opp-1512000000 {
> opp-hz = <0x0 0x5a1f4a00>;
> opp-microvolt = <0x124f80>;
> opp-microvolt-L0 = <0x124f80>;
> opp-microvolt-L1 = <0x11edd8>;
> opp-microvolt-L2 = <0x118c30>;
> opp-microvolt-L3 = <0x112a88>;
> clock-latency-ns = <0x9c40>;
> };
3764a3775,3785
>
> opp-1992000000 {
> opp-hz = <0x0 0x76bb8200>;
> opp-microvolt = <0x13d620>;
> opp-microvolt-L0 = <0x13d620>;
> opp-microvolt-L1 = <0x137478>;
> opp-microvolt-L2 = <0x1312d0>;
> opp-microvolt-L3 = <0x12b128>;
> clock-latency-ns = <0x9c40>;
> };
>
tkaiser
November 15, 2018, 12:58pm
11
BTW: IMO you should switch to CONFIG_HZ_250 (see reasons ).
Gouwa
November 15, 2018, 1:56pm
12
If I don’t get wrong, RK3399 SoC can only run 1.8GHz and 1.5GHz, only the Chromebook version RK3399C can up to 2.0GHz of bigger cluster.
Surely, you can still find out a few RK3399 SoC can run 2.0GHz, as both RK3399 and RK3399C are same thing, the only different is that when Rockchip filter the SoC, good ones label as RK3399C, and not so good one with RK3399.
Good day!
1 Like
tkaiser
November 15, 2018, 2:05pm
13
I understood something different: RK filters out those SoCs as RK3399C/OP1 that run reliably at lower DVFS operating points so they are able to clock them up to 2GHz in their Chromebook without wasting too much energy and generating too much heat. An RK3399C at 2.0GHz will stay as cool or even cooler as an RK3399 at 1.8GHz.
I simply used unlock-temperature.patch and overclock-rk3399-to-1.5-2.0.patch and the latter of course uses higher DVFS operating points therefore leading to higher consumption and temperatures under load.
Please note overclock-rk3399-to-1.8-2.2.patch.disabled some users activate but I don’t consider running RK3399 with A53 cores at 1.8GHz and A72 at 2.2GHz a sane default. But with appropriate cooling it also works: https://twitter.com/hjc4869/status/1006849763067621378
2 Likes
tkaiser
November 15, 2018, 2:55pm
14
@numbqq and I would also suggest applying this patch (see reasons )
1 Like
numbqq
November 15, 2018, 3:07pm
15
Hello tkaiser,
You can check the result here:
Retest with -c option. Will update here tomorrow.
numbqq
November 16, 2018, 3:23am
16
OK. Seems that you have already upload the result to your repo. Thanks.
tkaiser
November 16, 2018, 6:47am
17
If you rebuilt the kernel (especially with CONFIG_HZ_250 ) and numbers differ I would be happy to include another result to the listing. So please share
numbqq
November 16, 2018, 2:14pm
18
You can check the result here:
sbc-bench v0.6.2
Installing needed tools. This may take some time... Done.
Checking cpufreq OPP... Done.
Executing tinymembench. This will take a long time... Done.
Executing OpenSSL benchmark. This will take 3 minutes... Done.
Executing 7-zip benchmark. This will take a long time...+ Done.
Executing cpuminer. This will take 5 minutes... Done.
Checking cpufreq OPP... Done.
Memory performance (big.LITTLE cores measured individually):
memcpy: 1312.5 MB/s
memset: 4809.7 MB/s
memcpy: 2825.3 MB/s
memset: 4907.6 MB/s (0.5%)
Cpuminer total scores (5 minutes execution): 10.61,10.60,10.55,10.53,10.49,10.44,10.41,10.38,10.37,10.36,10.35,10.33,10.32,10.31,10.30,10.29,10.28,10.26 kH/s
7-zip total scores (3 consecutive runs): 6444,6468,6465
OpenSSL results (big.LITTLE cores measured individually):
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes
aes-128-cbc 135585.72k 400632.06k 765320.11k 1021268.65k 1131615.57k 1138731.69k
aes-128-cbc 403032.81k 906390.44k 1292418.65k 1423920.13k 1495247.53k 1495979.35k
aes-192-cbc 96385.22k 286495.02k 558528.51k 753120.60k 839611.73k 848134.14k
aes-192-cbc 376892.10k 829469.82k 1097246.98k 1259299.16k 1310932.99k 1320004.27k
aes-256-cbc 125190.33k 327254.49k 540443.73k 658430.63k 703100.25k 705779.03k
aes-256-cbc 370017.61k 753589.61k 998980.01k 1088732.16k 1125856.60k 1126438.23k
Full results uploaded to http://ix.io/1s6C. Please check the log for anomalies (e.g. swapping
or throttling happenend) and otherwise share this URL.
2 Likes
tkaiser
November 17, 2018, 2:03am
19
Thank you. But that’s still with CONFIG_HZ_1000 , right?
numbqq
November 17, 2018, 2:07am
20
I have set CONFIG_HZ=250.
tkaiser
November 17, 2018, 10:36pm
21
numbqq:
I have set CONFIG_HZ=250
That’s really strange since when I tested with CONFIG_HZ=1000 I got similar results wrt memory performance: sbc-bench/Results.md at master · ThomasKaiser/sbc-bench · GitHub
According to ayufan the way lower memory performance with Rockchip’s BSP kernel compared to mainline is related to CONFIG_HZ and switching from 1000 to 250 fixed it for him. Hmm… I need to test with my RockPro64 tomorrow…
numbqq
November 22, 2018, 2:29am
22
So the result in https://github.com/ThomasKaiser/sbc-bench/blob/master/Results.md for RockPro64 4.4 kernel is CONFIG_HZ=250 ? I found the memory performance is lower then mainline kernel.
tkaiser
November 22, 2018, 12:29pm
23
Nope, that was also with CONFIG_HZ=1000 – I need to retest soon.
Now that @balbes150 pushed his Armbian changes (using 250Hz and the other tweaks already) I’ll test with his images.
3 Likes
tkaiser
November 22, 2018, 4:26pm
24
New result with latest ayufan image where he switched to CONFIG_HZ=250 but still at 1.8/1.4GHz:
Memory performance (big.LITTLE cores measured individually):
memcpy: 1800.4 MB/s
memset: 8313.6 MB/s
memcpy: 3513.9 MB/s
memset: 8410.0 MB/s (0.5%)
Vs. Edge/Captain also with CONFIG_HZ=250 but at 2.0/1.5GHz:
Memory performance (big.LITTLE cores measured individually):
memcpy: 1391.1 MB/s (0.2%)
memset: 4821.1 MB/s
memcpy: 2856.5 MB/s (0.3%)
memset: 4882.2 MB/s (0.6%)
Significant change was this and afterwards memory performance was reported at same level as mainline kernel.
Maybe CONFIG_ARM_ROCKCHIP_DMC_DEVFREQ is the culprit? linux-kernel/arch/arm64/configs/rockchip_linux_defconfig at ed3ce4d15ec1c8721b55fbc6860b44907ea9eba5 · ayufan-rock64/linux-kernel · GitHub vs. https://github.com/armbian/build/blob/master/config/kernel/linux-rk3399-default.config#L5174-L5179
Interestingly on ayufan’s image /sys/bus/platform/drivers/rockchip-dmc/dmc/ is missing completely…
