Dear all,
I’m really having troubles to run my VIM3 stable for now. At least every 1 or 2 days the VIM3 crashes or freezes completely.
Two days ago I was able to gather the following Kernel message dump when it froze:
[21900.298170] Unable to handle kernel NULL pointer dereference at virtual address 00000000000000a8
[21900.301313] Mem abort info:
[21900.304072] ESR = 0x96000004
[21900.307092] EC = 0x25: DABT (current EL), IL = 32 bits
[21900.312352] SET = 0, FnV = 0
[21900.315370] EA = 0, S1PTW = 0
[21900.318474] Data abort info:
[21900.321322] ISV = 0, ISS = 0x00000004
[21900.325116] CM = 0, WnR = 0
[21900.328052] user pgtable: 4k pages, 48-bit VAs, pgdp=00000000920d5000
[21900.334432] [00000000000000a8] pgd=0000000000000000
[21900.339267] Internal error: Oops: 96000004 [#1] SMP
[21900.344093] Modules linked in: binfmt_misc(E) veth(E) xt_nat(E) xt_tcpudp(E) xt_conntrack(E) xt_MASQUERADE(E) nf_conntrack_netlink(E) nfnetlink(E) xfrm_user(E) xfrm_algo(E) xt_addrtype(E) iptable_filter(E) iptable_nat(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) br_netfilter(E) bridge(E) stp(E) llc(E) cpufreq_conservative(E) cpufreq_userspace(E) cpufreq_ondemand(E) cpufreq_powersave(E) macvlan(E) xfs(E) btsdio(E) hci_uart(E) btqca(E) btrtl(E) brcmfmac(E) btbcm(E) btintel(E) brcmutil(E) bluetooth(E) cfg80211(E) ecdh_generic(E) ecc(E) nvmem_meson_efuse(E) ip_tables(E) x_tables(E) meson_mx_sdio(E) rtc_meson_vrtc(E) meson_rng(E) rng_core(E) dwmac_generic(E) [last unloaded: reset_meson_audio_arb]
[21900.406631] CPU: 4 PID: 25622 Comm: kworker/4:2 Tainted: G E 5.7.0 #1
[21900.414386] Hardware name: amlogic w400/w400, BIOS 2020.04 08/03/2020
[21900.420784] Workqueue: events dbs_work_handler
[21900.425170] pstate: 60000085 (nZCv daIf -PAN -UAO)
[21900.429919] pc : regmap_update_bits_base+0x70/0x9c
[21900.434658] lr : regmap_update_bits_base+0x6c/0x9c
[21900.439398] sp : ffff8000128bba50
[21900.442675] x29: ffff8000128bba50 x28: 0000000000000000
[21900.447936] x27: ffff00007ff16d00 x26: ffff00007ff14de0
[21900.453198] x25: 0000000000000000 x24: 0000000000000000
[21900.458459] x23: 0000000000000000 x22: 0000000000000090
[21900.463720] x21: 00000000040003f0 x20: 0000000000000208
[21900.468981] x19: ffff0000d9ebf400 x18: 0000000000000000
[21900.474242] x17: 0000000000000000 x16: 0000000000000000
[21900.479504] x15: 0000000000000000 x14: 0000000000b71b00
[21900.484765] x13: 00000000016e3600 x12: 0000000000000000
[21900.490026] x11: 0000000000000000 x10: 00000000ffffffc3
[21900.495288] x9 : ffff800010811584 x8 : 0000000000000006
[21900.500549] x7 : 0000000000000000 x6 : 0000000000000000
[21900.505810] x5 : 0000000000000000 x4 : 0000000000000000
[21900.511071] x3 : ffff8000108112c8 x2 : 245cb3377b54f800
[21900.516332] x1 : 0000000000000090 x0 : 0000000000000000
[21900.521594] Call trace:
[21900.524017] regmap_update_bits_base+0x70/0x9c
[21900.528416] meson_clk_cpu_dyndiv_set_rate+0xf8/0x110
[21900.533414] clk_change_rate+0x160/0x2d0
[21900.537294] clk_change_rate+0x260/0x2d0
[21900.541176] clk_core_set_rate_nolock+0x16c/0x19c
[21900.545833] clk_set_rate+0x44/0x78
[21900.549284] _generic_set_opp_clk_only+0x20/0x58
[21900.553854] dev_pm_opp_set_rate+0x450/0x484
[21900.558080] set_target+0x48/0x78
[21900.561361] __cpufreq_driver_target+0x220/0x2f4
[21900.565937] od_dbs_update+0xec/0x170 [cpufreq_ondemand]
[21900.571194] dbs_work_handler+0x48/0x80
[21900.574986] process_one_work+0x1b0/0x2b0
[21900.578952] worker_thread+0x1ec/0x284
[21900.582664] kthread+0xe0/0xf0
[21900.585682] ret_from_fork+0x10/0x30
[21900.589224] Code: 2a1403e1 aa1303e0 97fffbb0 2a0003f4 (a9428261)
[21900.595256] ---[ end trace 5557c55222dffdc6 ]---
It looks like it had something to do with the cpufreq driver (ondemand) crashing one of the CPUs and freeze the machine.
Just a few minutes ago, the machine again froze (hartbeart LED stopped working), luckily I had the Serial Console connected, which gave the following Kernel message dump:
[146491.652619] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[146491.653436] rcu: 2-...0: (5 ticks this GP) idle=96a/1/0x4000000000000000 softirq=712849/712849 fqs=2588
[146491.662989] rcu: 5-...0: (4 ticks this GP) idle=7e6/1/0x4000000000000000 softirq=652052/652053 fqs=2589
[146491.672531] (detected by 1, t=5256 jiffies, g=2398493, q=291)
[146491.678376] Task dump for CPU 2:
[146491.681652] kworker/2:2 R running task 0 3705 2 0x0000002a
[146491.688789] Workqueue: events dbs_work_handler
[146491.693203] Call trace:
[146491.695748] __switch_to+0xd0/0x124
[146491.699263] 0x0
[146491.701150] Task dump for CPU 5:
[146491.704419] node R running task 0 1325 810 0x00000002
[146491.711485] Call trace:
[146491.714025] __switch_to+0xd0/0x124
[146491.717544] 0xffff0000dad9e040
[146554.672647] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[146554.673468] rcu: 2-...0: (5 ticks this GP) idle=96a/1/0x4000000000000000 softirq=712849/712849 fqs=4219
[146554.683018] rcu: 5-...0: (4 ticks this GP) idle=7e6/1/0x4000000000000000 softirq=652052/652053 fqs=4219
[146554.692562] (detected by 0, t=21007 jiffies, g=2398493, q=494)
[146554.698490] Task dump for CPU 2:
[146554.701766] kworker/2:2 R running task 0 3705 2 0x0000002a
[146554.708900] Workqueue: events dbs_work_handler
[146554.713317] Call trace:
[146554.715861] __switch_to+0xd0/0x124
[146554.719376] 0x0
[146554.721262] Task dump for CPU 5:
[146554.724533] node R running task 0 1325 810 0x00000002
[146554.731598] Call trace:
[146554.734137] __switch_to+0xd0/0x124
[146554.737657] 0xffff0000dad9e040
[146554.740885] rcu: rcu_preempt kthread starved for 12372 jiffies! g2398493 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=4
[146554.751525] rcu: RCU grace-period kthread stack dump:
[146554.756616] rcu_preempt I 0 10 2 0x00000028
[146554.762130] Call trace:
[146554.764670] __switch_to+0xd0/0x124
[146554.768211] __schedule+0x398/0x444
[146554.771739] schedule+0x84/0xd4
[146554.774940] schedule_timeout+0xc8/0xf0
[146554.778820] rcu_gp_kthread+0x42c/0x808
[146554.782696] kthread+0xec/0xfc
[146554.785797] ret_from_fork+0x10/0x18
[146617.692607] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[146617.693426] rcu: 2-...0: (5 ticks this GP) idle=96a/1/0x4000000000000000 softirq=712849/712849 fqs=12083
[146617.703069] rcu: 5-...0: (4 ticks this GP) idle=7e6/1/0x4000000000000000 softirq=652052/652053 fqs=12084
[146617.712700] (detected by 1, t=36764 jiffies, g=2398493, q=512)
[146617.718631] Task dump for CPU 2:
[146617.721906] kworker/2:2 R running task 0 3705 2 0x0000002a
[146617.729036] Workqueue: events dbs_work_handler
[146617.733458] Call trace:
[146617.736000] __switch_to+0xd0/0x124
[146617.739516] 0x0
[146617.741402] Task dump for CPU 5:
[146617.744674] node R running task 0 1325 810 0x00000002
[146617.751739] Call trace:
[146617.754279] __switch_to+0xd0/0x124
[146617.757796] 0xffff0000dad9e040
[146680.712607] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[146680.713418] rcu: 2-...0: (5 ticks this GP) idle=96a/1/0x4000000000000000 softirq=712849/712849 fqs=19953
[146680.723061] rcu: 5-...0: (4 ticks this GP) idle=7e6/1/0x4000000000000000 softirq=652052/652053 fqs=19954
[146680.732693] (detected by 1, t=52518 jiffies, g=2398493, q=527)
[146680.738624] Task dump for CPU 2:
[146680.741899] kworker/2:2 R running task 0 3705 2 0x0000002a
[146680.749028] Workqueue: events dbs_work_handler
[146680.753451] Call trace:
[146680.755997] __switch_to+0xd0/0x124
[146680.759509] 0x0
[146680.761397] Task dump for CPU 5:
[146680.764666] node R running task 0 1325 810 0x00000002
[146680.771733] Call trace:
[146680.774273] __switch_to+0xd0/0x124
[146680.777790] 0xffff0000dad9e040
Again it looks like something to do with the CPU.
Running like that it really makes no sense for me to use the VIM3 in real production environment.
I’m using the U-Boot Version from here (https://github.com/hyphop/khadas-uboot/releases) and the Linux Kernel 5.7.0 from here (https://github.com/hyphop/khadas-linux-kernel) with a non-changed config (defconfig). System is a clean Debian Buster (debootstraped) running from the eMMC. Power Supply is a 5.1V 3A original Raspberry one.
Anyone has a clue what’s causing this freezes and fix them?
Thanks and best regards
mcbain