Khadas Support Evidence — Mind Pro + Mind Graphics 2: PCIe stuck at Gen1 + crash under GPU load
Prepared: 2026-08-28 · Updated: 2026-09-03 (firmware assessment + Gen1 workaround) · Reporter OS: Ubuntu 24.04.4 LTS (Linux)
1. Summary of problem
The NVIDIA RTX 5060 Ti in the Mind Graphics 2 dock is recognised by the OS, but the
PCIe link cannot sustain any speed above Gen1 (2.5 GT/s). It idles at 2.5 GT/s,
throws physical-layer correctable errors (RxErr / BadDLLP) from boot, and the moment
real GPU traffic pushes the link up to 16 GT/s the GPU falls off the bus — Xid 79,
CUDA “device unavailable”, nvidia-smi in D-state, recoverable only by reboot.
Behaviour varies per boot: sometimes Gen1-usable, sometimes wedged, sometimes the GPU
does not enumerate at all.
The link is capable of negotiating Gen4 — it does so, and then fails. Holding the
link at Gen1 from the OS makes the machine completely stable (section 6d), which is the
clearest evidence that the problem lies in the link/firmware and not in software.
This matches these Khadas forum reports with identical symptoms:
- Mind 2s + Mind Graphics 2 : Driver patch loop, code 43 fixed but PCIe stuck at Gen1.1, extreme lag & system crash on load
- Mind 2 / Graphics 2 crash on 3d use
Khadas has stated that latest docks (Mind Graphics 2 / Mind xPlay) require a BIOS
upgrade for hardware compatibility. That advice does not resolve this case: this
machine already runs the newest BIOS Khadas has ever published for the Mind Pro
(v1.02, 2026-05-25 — verified against the download server, see section 2b), and the
fault persists on it. This report therefore asks for a fix, not for the existing update.
2. Hardware / firmware
| Item | Value |
|---|---|
| System | KHADAS Mind Pro |
| Board | MIND-PRO-K1016 |
| BIOS version | 1.02 — the newest Khadas publishes for this board |
| BIOS date | 2026-05-25 |
| Firmware GUID | DD2D4457-D89B-31B5-A76D-A41455790FAA |
| Board revision | V1.2 |
| Dock | Mind Graphics 2 (RTX 5060 Ti) |
| GPU PCI ID | 10de:2d04 (VGA) + 10de:22eb (audio) |
| Dock bridge | Intel 8086:e461 (PCIe bridge, port #9) |
| Kernel | 7.0.12-nopciequirk (custom; patches an unrelated older link-retrain quirk) |
| NVIDIA driver | 610.43.02 (open kernel module) |
2b. Firmware is already at the newest published version — verified
https://dl.khadas.com/products/mind-series/bios/mind-pro/ contains exactly two releases,
which appear to be the complete release history for this board:
| File | Version | Applicable here? |
|---|---|---|
mind-pro-bios-v1.01-260303.zip |
1.01 | No — older, and blocked (see below) |
mind-pro-bios-v1.02-260525.zip |
1.02 | Already installed |
Verified by identity, not just by version string:
Package mindfw.inf : FirmwareVersion 0x66 (= 102)
UEFI\RES_{DD2D4457-D89B-31B5-A76D-A41455790FAA}
Machine dmidecode : BIOS 1.02, date 05/25/2026,
Firmware ID DD2D4457-D89B-31B5-A76D-A41455790FAA
Machine UEFI ESRT : fw_class = dd2d4457-d89b-31b5-a76d-a41455790faa
fw_version = 102
last_attempt_version = 102, last_attempt_status = 0 (flash succeeded)
lowest_supported_fw_version = 102 (rollback to 1.01 refused)
Also checked and found unavailable for this board:
- No EC package for Mind Pro —
/products/mind-series/ec/contains onlymind-1/andmind-2/. - No separate Mind Graphics 2 dock firmware anywhere on the download server.
- For Mind Pro the single
mindfw.bincapsule is the whole firmware image.
For comparison, forum posts show Mind 2 already at BIOS 1.07 with Graphics 2 dock
firmware 1.09, while the Mind Pro has received only two BIOS releases in total. If the
Mind Graphics 2 compatibility work landed in those later Mind 2 releases, the equivalent
change appears never to have shipped for the Mind Pro.
3. Core evidence — PCIe link stuck at Gen1
Re-confirmed on a fresh boot, 2026-09-03 (kernel 7.0.12-nopciequirk, driver 610.43.02):
00:06.0 bridge LnkCap: Port #9, Speed 32GT/s, Width x4
LnkSta: Speed 2.5GT/s, Width x4
LnkCtl2: Target Link Speed: 32GT/s, SpeedDis-
01:00.0 GPU LnkCap: Port #0, Speed 32GT/s, Width x16
LnkSta: Speed 2.5GT/s (downgraded), Width x4 (downgraded)
nvidia-smi reports pcie.link.gen.current = 1 against pcie.link.gen.max = 5.
The bridge’s Target Link Speed is already set to 32 GT/s and speed change is not
disabled (SpeedDis-), yet the link still trains at 2.5 GT/s — i.e. the host asks for
Gen5 and the physical link will not come up above Gen1. Forcing a higher target speed
via setpci + retrain does NOT move it above Gen1.
4. Physical-layer / data-link PCIe errors (present from boot)
pcieport 0000:00:06.0: PCIe Bus Error: severity=Correctable, type=Data Link Layer, (Receiver ID)
first at t = 0.96 s, 16 such events during boot on 2026-09-03
AER counters on the dock bridge 0000:00:06.0 after 32 minutes of uptime
(/sys/bus/pci/devices/0000:00:06.0/aer_dev_correctable):
RxErr 1 BadTLP 11 BadDLLP 9 Timeout 1 TOTAL_ERR_COR 21
(aer_dev_fatal and aer_dev_nonfatal: all zero)
These are signal-integrity errors on the Mind Link connection. They appear even at the
reduced Gen1 speed, and under GPU load they escalate and the GPU drops off the bus.
5. Crash-under-load signature (observed when loading larger models / sustained load)
NVRM: GPU0 rpcSendMessage failed with status 0x0000000f ...
NVRM: GPU0 ... Assertion failed ... GPU_IN_FULLCHIP_RESET ...
=> CUDA error: CUDA-capable device(s) is/are busy or unavailable
=> nvidia-smi then hangs (uninterruptible D-state); requires reboot
Small models (≤7B) sometimes run at ~52 tok/s over Gen1; heavier transfer triggers the crash.
6. Workarounds already tried (NONE fixed Gen1 / crash)
- Physical re-dock of the magnetic Mind Link coupling — no change.
- Kernel
pcie_aspm=off— made it WORSE (GPU disappeared from the bus entirely). - Forced PCIe target Gen3 + link retrain — link refused to leave Gen1.
- Cold power-drain reboots — behaviour still varies boot-to-boot.
- NVIDIA driver already at 610.43.02.
- Deferred NVIDIA module load via udev (load only once
10de:2d04is enumerated) —
this stops the boot-time D-state hang, but has no effect on link speed.
Note on the kernel: the custom kernel only bypasses an unrelated Linux link-retrain
quirk (pcie_failed_link_retrain(), added in v6.5) that otherwise prevented the GPU from
enumerating at all. With that quirk bypassed the kernel logs zero “retraining failed”
messages — the Gen1 limitation is therefore not caused by that kernel quirk.
6b. Reproduced live on 2026-09-03 — Xid 79 “GPU has fallen off the bus”
Fresh boot, idle for 86 seconds, then one small inference job (a 1.5B-parameter model,
~1.8 GB) was started. The GPU died immediately:
[ 86.334916] NVRM: Xid (PCI:0000:01:00): 79, pid=2555, name=llama-server,
GPU has fallen off the bus.
[ 86.334925] NVRM: GPU 0000:01:00.0: GPU has fallen off the bus.
[ 86.334955] NVRM: GPU0 _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
... 179 x NVRM assertion "GPU_IN_FULLCHIP_RESET"
... ggml_cuda_init: failed to initialize CUDA: unknown error
Afterwards nvidia-smi reports “Unable to determine the device handle for GPU0:
0000:01:00.0: Unknown Error / No devices were found”, while the device is still
present in lspci and in power state D0. Unloading the nvidia modules hangs
(uninterruptible), so only a reboot recovers the GPU.
Correctable-error rate exploded at the same time
TOTAL_ERR_COR on bridge 0000:00:06.0, sampled after the crash:
t+0s 47
t+120s 158 (+111 in two minutes)
t+~5min 408
t+~6min 441 → 454 → 466 (~4 errors per second, sustained)
Nearly all are RxErr — Physical Layer on [8086:e461]. This is a continuous
signal-integrity failure on the Mind Link connection, not a software fault.
The link had just trained UP to Gen4
Critically, after the crash the bridge reports:
LnkSta: Speed 16GT/s, Width x4 (was 2.5GT/s while idle)
So the link is able to negotiate Gen4 — but as soon as it does, under actual GPU
traffic, the physical layer fails, and the GPU drops off the bus. This is consistent with
a marginal/incompatible Mind Link signal path rather than a link-training policy problem,
and it is why the machine appears “stable” only while pinned at Gen1 and idle.
This is reproducible: idle boot → single small inference → GPU lost.
6c. Controlled A/B test rules out the kernel (2026-09-03)
To be certain the fault is not caused by our custom kernel, the identical workload was
run on two builds of it that differ only in how the PCIe link-retrain quirk is bypassed
(one globally, one only for the dock root port). Each was a clean boot, idle, then one
small 1.5B-parameter model load:
| build #2 (quirk off globally) | build #3 (quirk off for 8086:e461 only) |
|
|---|---|---|
| Before load | GPU alive, Gen1 x4 | GPU alive, Gen1 x4 |
| During load | CUDA error: unspecified launch failure |
CUDA error: unknown error |
| Xid 79 “fallen off the bus” | yes | yes |
| Bridge link after crash | 16 GT/s | 16 GT/s |
| Result | GPU lost, reboot required | GPU lost, reboot required |
Both fail identically. The failure is therefore not a property of the kernel patch.
In both runs, the link had moved from 2.5 GT/s to 16 GT/s at the moment of the crash.
Note also that the crashing workload is the smallest model available (~1.8 GB). Earlier,
this machine could serve 7B models at ~52-62 tok/s over the Gen1 link; it can no longer
survive even a 1.5B load, so the behavior appears to be getting worse over time.
6d. Workaround found: capping the link at Gen1 makes it fully stable
Since the crash always coincided with the link moving to 16 GT/s, the bridge’s Target Link
Speed was pinned to 2.5 GT/s (LnkCtl2 low nibble = 1) before the NVIDIA driver binds
the GPU, via a udev rule on the GPU’s appearance. Result:
| Workload | Without cap | With Gen1 cap |
|---|---|---|
| 1.5B model, 400 tokens | GPU lost (Xid 79) | 178.7 tok/s |
| 7B model (4.7 GB) | not reached | 74.7 tok/s |
| 20B model (13.8 GB) | not reached | 76.2 tok/s, 27 s to load |
| 3 sustained requests | not reached | 73.8 / 74.0 / 74.3 tok/s |
| New correctable errors | +6 … +111 | 0 |
| Xid 79 / fullchip-reset | yes / 179 | 0 / 0 |
Zero new PCIe errors across all of it, GPU at 63 °C, 94% utilisation, models 100% on GPU.
This is diagnostically important: the GPU, the driver, the kernel and the software stack are
all healthy. The only thing that has to be given up is PCIe link speed — and once the link
is held at Gen1 the very same hardware runs faster and more reliably than it ever did while the
link was allowed to negotiate upward. That points squarely at the physical link / firmware
configuration of the Mind Link path, not at any software layer.
We would still very much like a BIOS that lets this dock run at its rated speed, since model
loading is currently bandwidth-limited (13.8 GB takes 27 s).
7. Current status
With the Gen1 cap in place the machine is stable and usable: models run 100% on the GPU
with no PCIe errors. The remaining cost is bandwidth — Gen1 x4 is ~1/16 of the link’s rated
capability (Gen5 x4), so loading a 13.8 GB model takes 27 s instead of ~2 s.
Without the cap the GPU is lost to Xid 79 on the first inference of any size.
8. Request to Khadas
We are not asking for the existing 1.02 update — it is installed, it flashed
successfully, and the fault persists on it. Concretely:
- Is there a Mind Pro BIOS newer than 1.02, including a beta or engineering build?
The Mind Graphics 2 compatibility fixes that reached Mind 2 (now at 1.07) do not appear
to have shipped for the Mind Pro (MIND-PRO-K1016), which has only ever had 1.01 and 1.02. - Is the Mind Pro officially supported with the Mind Graphics 2 dock? If it is not, please
say so plainly — we can plan around that. The product pages do not state a restriction. - Does the Mind Graphics 2 dock have its own updatable firmware? Forum posts reference
“Graphics 2 firmware 1.09”, but no dock firmware is published on the download server.
Our machine exposes a second ESRT entry,8d0f1282-8bbc-efc4-06c3-fbfe86040d4e,
device firmware version 1525 — please confirm what this is and whether it is current. - Is a PCIe link-speed setting exposed anywhere in BIOS setup? We currently have to pin
the link to Gen1 from the OS (section 6d) to keep the machine usable. A supported way to
set this, or a firmware default that trains reliably, would be a workable outcome. - If this is a hardware fault in the dock or the Mind Link connector, please advise on RMA.
Note the degradation described in section 6c: this machine previously served 7B models over
the link, and now loses the GPU on the smallest workload unless the link is capped.
What we have already ruled out
- Not the OS or driver stack — with the link pinned at Gen1 the same machine runs models
100% on the GPU at full speed with zero PCIe errors (section 6d). - Not the Linux kernel — a controlled A/B test between two kernel builds produced identical
failures (section 6c). - Not model size or memory pressure — the smallest model (1.8 GB) triggers the crash,
while a 13.8 GB model runs fine once the link is capped. - Not a failed or partial BIOS flash — ESRT reports
last_attempt_status = 0at version 102.