Which system do you use? Android, Ubuntu, OOWOW or others?
Ubuntu 22.02 Desktop
Which version of system do you use? Khadas official images, self built images, or others?
Official from Oowow installer
Please describe your issue below:
When the CPU is under heavy load (compiling software) I am seeing a lot of IO errors to the NVMe. My OS is installed to eMMC but use NVMe for my data. NVMe is WD 2TB WD Green SN350.
I have switched from using a generic USB-C power supply to using 12vdc 15A regulated power supply and the issue is still happening. The NVMe device tests fine in a PC.
I’ve installed a heatsink on the NVMe and also put a large 80mm fan on the VIM4 that blows across the top and bottom of the board. It seems I’m getting less write errors now but it’s still happening after the device is under full load for a while.
Continuing on, I still feel this is heat related so here is what I’ve done. I removed the heatsink, applied thermal pads to the RAM. I removed the M2X board and placed thermal pads on the 2 ICs that are on the bottom of the VIM4. I also put thermal pads under my NVMe so that the M2X is now being used as a heatsink. It’s much more stable now but still gets write errors after being under 100% load for a while.
Today I tried something different. I wanted to make an image of the NVMe. The system was under little load but after about 30 minutes of constant reading from the NVMe and the IO errors started. The NVMe (controller and actual memory chip) never went over 45c as I was monitoring it with an IR (FLIR) camera.
I have searched around and do not have surplus NVMe drives. I have lots of surplus SATA and SAS drives. I am still experimenting to see what might be the cause. I’ve ruled out heat and system load completely now. The issue happens after extended high IO to or from the NVMe. It seems that the command queue of the NVMe might be being exceeded. I’ve not been able to prove this. Maybe over this weekend I can put more time into it.
When I tried making an image to my Netac formatted as NTFS, it was pretty slow (~35 MB/sec) but managed about 12GB. When I tried the same thing with it formatted as ext4, it was fast (~125 MB/sec) but failed after only a few GB.
So, it seems that the system handles NTFS quite slowly and not just on NVME because SD cards formatted as NTFS are much slower than those formatted as ext4 as well.
For whatever reason, it seems there’s a bottleneck that allows more data to be written before it craps out entirely.
Not that what actually gets written is any good, as I discovered when all my dashcam videos got corrupted.
I mentioned above that I had added thermal pads to my NVMe device. While this did not help the issue, I still wanted to share this as NVMe devices do produce a decent amount of heat under load. The small controller chip, closest to the socket is typically where the majority of heat will be produced too. The M2X board has a lot of copper in it so it should work as a decent heat spreader.
I can corroborate this, with a 1TB Sabrent Rocket m.2 NVMe. I have a thermal pad between the SSD and the M2X expansion board, and a sizable heatsink thermally adhered to the drive itself. Operating temperature averages 47c, never rising above 50c. The M2X becomes noticeably warm to the touch, validating your observation that it is a good dissapator, without ever becoming hot.
Regardless, the NVMe data corruption, read-only lockups, and occasional system freezes still occur when external cooling drops op temp to 40c.
I have a couple aluminum heatsinks ready to adhear to the NVMe storage and controller chip but since putting the thermal pads, mine has not gone over 50c. After the io errors are addressed, I might need to add more cooling.
