pci-E ubuntu server issues (google coral)

namvu · November 14, 2019, 2:59pm

Hi guys, one other thing that I like to share (we are working on documenting this):
If you are running a system with kernel version 4.19+ you’ll need to first disable built-in kernel module driver for apex device. It has a known bug which might prevent proper driver update.

Create a blacklist-apex.conf file in /etc/modprobe.d/ with two lines in there:
blacklist gasket
blacklist apex
Reboot system
Verify that apex/gasket modules aren’t loaded by doing:
lsmod|grep apex
Follow steps to install gasket-dkms package from

delete file /etc/modprobe.d/blacklist-apex.conf
reboot system

Then follow other steps from instruction.

Although I’m not sure if this is related to the issue with khadas devices because if lscpi not showing our devices, then it looks like a configuring issue.

larrylart · November 14, 2019, 8:34pm

@namvu I just tried that and made no difference. Based on the specs it should work from the interface point of view. https://coral.withgoogle.com/docs/m2/datasheet/

I had a quick look around and based on the logs it seems to be failing here at func amlogic_pcie_link_up https://github.com/khadas/linux/blob/khadas-vims-4.9.y/drivers/amlogic/pci/pcie-amlogic-v2.c

I tough for a second that coral was not gen 1 or 2 as in the driver but that is not the case. I notices some patches from sept/2019 which haven’t not been applied to pcie-amlogic-v2 (8 month old now) . I am not sure if any of these are relevant but maybe one of them could fix the problem.

https://lore.kernel.org/linux-amlogic/20190916125022.10754-7-narmstrong@baylibre.com/T/

namvu · November 15, 2019, 1:07am

I figured it’s not the same issue, if the wrong module were loaded, I’d imagine a runtime issue rather than not being able to detect at all.
So from the source code (thanks for pointing out the)
I believe this is an amlogic phy not linking to the PCIe module. I have no idea if that patch you mentioned would fix this.

Maybe move this to the amlogic forum?

Edit: Although that code is on the khadas-vims-4.9.y branch, is this the right build for the vim3 board?

numbqq · November 15, 2019, 2:23am

Does this different to the NVMe SSD ?

Yes, it the branch used for VIM3.

namvu · November 15, 2019, 3:44pm

AFAIK: NVMe is just a protocols for communicating via the PCIe interface. I believe this is an issue to the actual phy chip.

numbqq · November 16, 2019, 2:58am

No idea yet. If everything is OK, I will get a coral in the end of this month, I hope I can debug with it at that time.

rider · December 1, 2019, 7:46am

Have did you received the coral?

numbqq · December 2, 2019, 1:34am

Not yet…I can only get USB version for current stage, M.2 version still need to wait more time…

demps · February 3, 2020, 6:06am

I tried to connect a PCI-e sound card through an adapter. Kernel 4.9 produces a similar error in the logs. But on the mainline kernel (5.5 rc2), the sound card was determined. Apparently, you still need to backport the latest changes related to PCI-e.

kaishi · May 5, 2020, 11:07pm

@numbqq

Any update on this? Did you have a luck to get Coral M2 module to test it out?

Thanks

numbqq · May 6, 2020, 9:13am

I just checked, I will try to buy one and feedback here.

xlazom00 · May 6, 2020, 7:51pm

I made ubuntu 20.04 image with fenix with mainline linux
uname -a
Linux Khadas 5.6.0
cat /sys/class/mcu/usb_pcie_switch_mode
1
so it looks like I have pci-e mode

but
lspci -vvv
is empty and I don’t see any host bridge
and no sign of meson pci-e in dmesg
any idea?

I would like to connect pci-e dvb-t tunner

btw
http://lists.infradead.org/pipermail/linux-amlogic/2020-May/016641.html

srirammails · October 12, 2020, 8:19pm

Any progress on connecting coral M2 card to VIM3/L pcie slot?

adzio · March 10, 2021, 6:47am

Any updates on VIM3x and the Coral module? @numbqq

I just bought your VIM3L and tried on VIM3L_Ubuntu-server-focal_Linux-5.7-rc7_arm64_SD-USB_V0.9-20200530 : after installing the B+M module and the Coral driver lspci reports nothing, nothing relevant in dmesg.

numbqq · March 10, 2021, 12:41pm

I still have no luck to make it work…

adzio · March 10, 2021, 6:19pm

Thanks @numbqq for the update.

So you have the Coral M.2 B+M module purchased but you can’t make it work with VIM3x? Have you tried it with your other Khadas boards, especially the ones without built-in NPU, like Edge-V?

What power supply are you using? In my case, once I connect the M.2 module, install the driver and restart, my VIM3L doesn’t boot - I may have to try a beefier power supply (say 12V 5A) as the TPU module specs list current demand up to 3A.

adzio · March 12, 2021, 3:55am

After toggling USB 3.0 -> PCIe I see this in dmesg:

[    1.359614] ehci-pci: EHCI PCI platform driver
[    1.375263] ohci-pci: OHCI PCI platform driver
[    2.277208] dw-pcie fc000000.pcie: IRQ index 1 not found
[    2.282201] meson-pcie fc000000.pcie: host bridge /soc/pcie@fc000000 ranges:
[    2.286951] meson-pcie fc000000.pcie:       IO 0x00fc600000..0x00fc6fffff -> 0x0000000000
[    2.295051] meson-pcie fc000000.pcie:      MEM 0x00fc700000..0x00fdffffff -> 0x00fc700000
[    2.344942] meson-pcie fc000000.pcie: error: wait linkup timeout
[    2.495823] meson-pcie fc000000.pcie: error: wait linkup timeout
[    2.675384] meson-pcie fc000000.pcie: error: wait linkup timeout
[    2.850951] meson-pcie fc000000.pcie: error: wait linkup timeout
[    3.038993] meson-pcie fc000000.pcie: error: wait linkup timeout
[    3.220952] meson-pcie fc000000.pcie: error: wait linkup timeout
[    3.376090] meson-pcie fc000000.pcie: error: wait linkup timeout
[    3.540306] meson-pcie fc000000.pcie: error: wait linkup timeout
[    3.724689] meson-pcie fc000000.pcie: error: wait linkup timeout
[    3.900952] meson-pcie fc000000.pcie: error: wait linkup timeout
[    4.006469] meson-pcie fc000000.pcie: Phy link never came up
[    4.019777] meson-pcie fc000000.pcie: failed to initialize host
[    4.033097] meson-pcie fc000000.pcie: Add PCIe port failed, -110

$ uname -a
Linux Khadas 5.7.0-rc7 #0.9 SMP Sat May 30 14:37:25 CST 2020 aarch64 aarch64 aarch64 GNU/Linux

Someone reported on another forum that this -110 error had gone away with fixes in 5.8.x, can we count on updated Khadas images soon?

adzio · March 24, 2021, 1:55am

Same issue and syslog as OP on my VIM3L with Linux Khadas 4.9.241 #8 SMP and Coral TPU B+M. Using 12V 6A supply through the VIN connector.

lspci -x
(no output)

sudo dmesg | grep pci
[    0.557199] amlogic-pcie-v2 fc000000.pcieA: amlogic_pcie_probe!
[    0.559858] OF: PCI: host bridge /pcieA@fc000000 ranges:
[    0.561072] amlogic-pcie-v2 fc000000.pcieA: Set the RC Bus Master, Memory Space and I/O Space enables.
[    0.561107] amlogic-pcie-v2 fc000000.pcieA: normal gpio
[    0.561155] amlogic-pcie-v2 fc000000.pcieA: GPIO normal: amlogic_pcie_assert_reset
[    0.647563] amlogic-pcie-v2 fc000000.pcieA: Error: Wait linkup timeout.
[    0.819902] amlogic-pcie-v2 fc000000.pcieA: Error: Wait linkup timeout.
[    0.992109] amlogic-pcie-v2 fc000000.pcieA: Error: Wait linkup timeout.
[    1.164412] amlogic-pcie-v2 fc000000.pcieA: Error: Wait linkup timeout.
[    1.336667] amlogic-pcie-v2 fc000000.pcieA: Error: Wait linkup timeout.
[    1.508941] amlogic-pcie-v2 fc000000.pcieA: Error: Wait linkup timeout.
[    1.681217] amlogic-pcie-v2 fc000000.pcieA: Error: Wait linkup timeout.
[    1.853500] amlogic-pcie-v2 fc000000.pcieA: Error: Wait linkup timeout.
[    2.025769] amlogic-pcie-v2 fc000000.pcieA: Error: Wait linkup timeout.
[    2.198047] amlogic-pcie-v2 fc000000.pcieA: Error: Wait linkup timeout.
[    2.288849] amlogic-pcie-v2 fc000000.pcieA: phy link never came up
[    2.289047] amlogic-pcie-v2 fc000000.pcieA: PCI host bridge to bus 0000:00
[    2.289081] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.289105] pci_bus 0000:00: root bus resource [io  0x0000-0xfffff]
[    2.289129] pci_bus 0000:00: root bus resource [mem 0xfc700000-0xfdffffff]
[    2.289184] amlogic-pcie-v2 fc000000.pcieA: link timeout, disable PCIE PLL
[    2.289215] amlogic-pcie-v2 fc000000.pcieA: power down pcie phy
[    2.608198] ehci-pci: EHCI PCI platform driver
[    2.608328] ohci-pci: OHCI PCI platform driver
[    5.193860] amlogic-new-usb3-v2 ffe09080.usb3phy: pci-e driver probe, disable USB 3.0 function!!!

Message pci-e driver probe, disable USB 3.0 function is perplexing, since PCIe is enabled using factory Android on eMMC.

cat /sys/class/mcu/usb_pcie_switch_mode
1

Anything else can be checked to troubleshoot this @numbqq?

Frank · March 25, 2021, 1:47am

@adzio Sorry, we still haven’t let TPU run on VIM3/VIM3L …

David_Zou · August 20, 2021, 2:38am

What’s the status of this issue now? And any plan for this issue? (It lasts for almost 2 years.)
Following is the the FW running on my VIM3. PCIe Wi-Fi card inserting to M.2 slot still hit this issue.
Would you please share a working FW image (Ubuntu) that I can work as a WAR.

Thanks,

Linux Khadas 4.9.241 #20 SMP PREEMPT Fri Jun 25 14:26:34 CST 2021 aarch64 aarch64 aarch64 GNU/Linux