pci-E ubuntu server issues (google coral)

VIM3
1

Hi,

I’m trying to work with the pci-e (m.2) and I’ve encountered some issues:
My setup is Vim3 pro with Ubuntu 18.04 server image .

  1. trying to switch from usb 3.0 to pcie using this guide didn’t work as no such " sys/class/mcu/usb_pcie_switch_mode" location exists.
    I was able to find - “/sys/class/usb_pcie_switch/mode” location and set the bit to 1 using SU command (sudo didn’t work) ,but I don’t know if PCI-E was actually on - i assume it is, if so it might worth adding this to docs.

  2. pcie device inst recognized at all:
    khadas@Khadas:~$ lspci -x
    khadas@Khadas:~$

khadas@Khadas:~$ sudo dmesg | grep pci
[ 0.553733] amlogic-pcie-v2 fc000000.pcieA: amlogic_pcie_probe!
[ 0.555978] OF: PCI: host bridge /pcieA@fc000000 ranges:
[ 0.556845] amlogic-pcie-v2 fc000000.pcieA: Set the RC Bus Master, Memory Space and I/O Space enables.
[ 0.556873] amlogic-pcie-v2 fc000000.pcieA: normal gpio
[ 0.556911] amlogic-pcie-v2 fc000000.pcieA: GPIO normal: amlogic_pcie_assert_reset
[ 0.646528] amlogic-pcie-v2 fc000000.pcieA: Error: Wait linkup timeout.
[ 0.818685] amlogic-pcie-v2 fc000000.pcieA: Error: Wait linkup timeout.
[ 0.990850] amlogic-pcie-v2 fc000000.pcieA: Error: Wait linkup timeout.
[ 1.163030] amlogic-pcie-v2 fc000000.pcieA: Error: Wait linkup timeout.
[ 1.339171] amlogic-pcie-v2 fc000000.pcieA: Error: Wait linkup timeout.
[ 1.515324] amlogic-pcie-v2 fc000000.pcieA: Error: Wait linkup timeout.
[ 1.691477] amlogic-pcie-v2 fc000000.pcieA: Error: Wait linkup timeout.
[ 1.867628] amlogic-pcie-v2 fc000000.pcieA: Error: Wait linkup timeout.
[ 2.043778] amlogic-pcie-v2 fc000000.pcieA: Error: Wait linkup timeout.
[ 2.219930] amlogic-pcie-v2 fc000000.pcieA: Error: Wait linkup timeout.
[ 2.314396] amlogic-pcie-v2 fc000000.pcieA: phy link never came up
[ 2.314534] amlogic-pcie-v2 fc000000.pcieA: PCI host bridge to bus 0000:00
[ 2.314558] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 2.314577] pci_bus 0000:00: root bus resource [io 0x0000-0xfffff]
[ 2.314596] pci_bus 0000:00: root bus resource [mem 0xfc700000-0xfdffffff]
[ 2.314638] amlogic-pcie-v2 fc000000.pcieA: link timeout, disable PCIE PLL
[ 2.314662] amlogic-pcie-v2 fc000000.pcieA: power down pcie phy
[ 2.748498] ehci-pci: EHCI PCI platform driver
[ 2.748585] ohci-pci: OHCI PCI platform driver
[ 6.028101] amlogic-new-usb3-v2 ffe09080.usb3phy: pci-e driver probe, disable USB 3.0 function!!!
khadas@Khadas:~$

I would appreciate any help for 2nd issue.

2

@rider I am having the same issue, pci logs look the same. I am using ubuntu 18.04 kernel 4.9.190 desktop. And I used to switch both /sys/ and serial console/kbi. None made a difference.
Are trying to use pci b-m card (coral?) straight into the m.2 socket on VIM3 or do you use m.2 extension card?

3

@larrylart
You need to setup it by KBI on uboot interface.

4

What device you attached to the M.2 slot?

5

@Frank I did that in the KBI uboot interface as well after I tried the echo 1 >/sys/… method. I got the same problem
@numbqq I’m trying to attach the Coral tpu M.2 card https://coral.withgoogle.com/products/m2-accelerator-bm, however based on the logs it looks like the PCIe bus doesn’t get powered up ?

6

@larrylart im trying to add coral m.2 also straight into the m.2 socket , with no luck

7

Hi, any luck with this? It seems that PCIe does no get activated based on the logs?

[ 2.219930] amlogic-pcie-v2 fc000000.pcieA: Error: Wait linkup timeout.
[ 2.314396] amlogic-pcie-v2 fc000000.pcieA: phy link never came up
…
[ 2.314638] amlogic-pcie-v2 fc000000.pcieA: link timeout, disable PCIE PLL
[ 2.314662] amlogic-pcie-v2 fc000000.pcieA: power down pcie phy

8

I have no idea yet, I don’t have a coral, so I can’t debug on it.

Can you check whether the NVMe SSD can be recognized?

9

Unfortunately I don’t have any NVMe SSD to test with. Perhaps someone else here has one and willing to test to plug in directly in the board PCIe M.2 ?

10

I tested on my side, NVMe SSD works.

1 Like
11

I have both coral and Samsung 970 m.2 SSD.

SSD is recognized
dmesg:

khadas@Khadas:~$ sudo dmesg | grep pci
[sudo] password for khadas:
[    0.561873] amlogic-pcie-v2 fc000000.pcieA: amlogic_pcie_probe!
[    0.564125] OF: PCI: host bridge /pcieA@fc000000 ranges:
[    0.565013] amlogic-pcie-v2 fc000000.pcieA: Set the RC Bus Master, Memory Space and I/O Space enables.
[    0.565041] amlogic-pcie-v2 fc000000.pcieA: normal gpio
[    0.565076] amlogic-pcie-v2 fc000000.pcieA: GPIO normal: amlogic_pcie_assert_reset
[    0.592414] amlogic-pcie-v2 fc000000.pcieA: PCIE SPEED IS GEN2
[    0.592435] amlogic-pcie-v2 fc000000.pcieA: link up
[    0.592596] amlogic-pcie-v2 fc000000.pcieA: PCI host bridge to bus 0000:00
[    0.592620] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.592639] pci_bus 0000:00: root bus resource [io  0x0000-0xfffff]
[    0.592658] pci_bus 0000:00: root bus resource [mem 0xfc700000-0xfdffffff]
[    0.592692] amlogic-pcie-v2 fc000000.pcieA: the device class is not reported correctly from the register
[    0.592718] pci 0000:00:00.0: [16c3:abcd] type 01 class 0x060400
[    0.592747] pci 0000:00:00.0: reg 0x38: [mem 0x00000000-0x0000ffff pref]
[    0.592799] pci 0000:00:00.0: supports D1
[    0.592805] pci 0000:00:00.0: PME# supported from D0 D1 D3hot D3cold
[    0.593112] pci 0000:01:00.0: [144d:a808] type 00 class 0x010802
[    0.593197] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
[    0.593370] pci 0000:01:00.0: Max Payload Size set to 256 (was 128, max 256)
[    0.608684] pci 0000:00:00.0: BAR 8: assigned [mem 0xfc700000-0xfc7fffff]
[    0.608707] pci 0000:00:00.0: BAR 6: assigned [mem 0xfc800000-0xfc80ffff pref]
[    0.608737] pci 0000:01:00.0: BAR 0: assigned [mem 0xfc700000-0xfc703fff 64bit]
[    0.608802] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[    0.608822] pci 0000:00:00.0:   bridge window [mem 0xfc700000-0xfc7fffff]
[    1.007723] amlogic-pcie-v2 fc000000.pcieA: the device class is not reported correctly from the register
[    1.007916] aer 0000:00:00.0:pcie002: service driver aer loaded
[    1.007972] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
[    1.007994] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
[    1.008015] pcie_pme 0000:00:00.0:pcie001: service driver pcie_pme loaded
[    1.029907] nvme nvme0: pci function 0000:01:00.0
[    1.042825] ehci-pci: EHCI PCI platform driver
[    1.042916] ohci-pci: OHCI PCI platform driver
[    4.424738] amlogic-new-usb3-v2 ffe09080.usb3phy: pci-e driver probe, disable USB 3.0 function!!!
[  130.076194] amlogic-pcie-v2 fc000000.pcieA: the device class is not reported correctly from the register
[  138.980706] amlogic-pcie-v2 fc000000.pcieA: the device class is not reported correctly from the register
[  142.460720] amlogic-pcie-v2 fc000000.pcieA: the device class is not reported correctly from the register

lspci:

khadas@Khadas:~$ lspci
00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01)
01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981

but coral isent

1 Like
12

Hi are you Moshe by any chance?
It’s Nam here from the coral team, I’m checking out this thread for some further investigation (apologize, I don’t have access to my corp machine rt now so I can’t email you back directly).
Is the Samsung 970 m.2 SSD you tested communicating via the NMVe interface or PCIe?

Could we possibly test another PCIe device on the khadas vim3 board?

Khadas team, are there any ways to confirm when the m.2 slot is configured for PCIe?

1 Like
13

Yes, it is.

We don’t have other PCIe device except the NVMe SSD, and we don’t have a coral, I will buy some PCIe devices to check this issue.

Yes, it is configured for PCIe.

By the way, does the coral need other driver?

Thanks.

14

Hello @larrylart @rider

Where you buy the coral? It seems that I can’t buy a coral in China…

15

You can buy from Mouser, I see if you select china it says limited availability, so I selected Honk Kong and that seems to have it on stock.

https://www.mouser.hk/ProductDetail/Coral/G650-04686-01?qs=sGAEpiMZZMu3sxpa5v1qrr6044Dk0q4qeHP1PKY2XfI=

And there is a driver needed to install which I managed to compile install and loaded manually ok with ‘modprobe’.

16

@namvu
yes, its me.

currently i don’t have any other devices

17

Are you able to recognize the device using ‘modprobe’?

if so can you elaborate how?

18

No, just me testing the kernel modules for coral are loading successfully, after testing with mod probe I added them to /etc/modules. However the device is not sensed detected at boot as I am getting the same logs as you, even regardless if the device is plugged in the M2 socket or not.
However It does seem to get power as it gets a little bit warm, and I am sure is not malfunctioning as I got two of them and none works, so my guess will be that has something to do with pcie kernel driver not detecting it. Could it be because it’s an M.2 M+B key?

lsmod
Module Size Used by
apex 28672 0
gasket 106496 1 apex

19

Hello, For what it is worth, I have an NVMe SSD(Patriot PS128GPM280SSDR) with M+B key, it works in the VIM3’s M.2 socket(also works with M2X ext board).

1 Like
20

Thanks for that, and yes you do need to install some drivers:

Edit: I don’t believe the driver is the issue since lspci couldn’t detect the m2 key

1 Like