Mainline U-Boot works with LPDDR4 patch series

With a recently posted (not yet merged) patch series from Jagan Teki adding rank detection, LPDDR4 timings and so on, mainline U-Boot boots Khadas Edge.

The Rockchip miniloader (idbloader.img) is no longer needed and the ARM Trusted Firmware bl31 component is built into U-Boot itself. This results in a less ugly and noisy boot process, and is 100% mainline free software. (The entire system can be blob-free apart from the firmware for the Broadcom wifi module.)

Until all the changes are merged upstream, I have a pre-prepared tree of U-Boot mainline (as of this morning) with the patch series (99 patches), a suitable defconfig and device tree. This can be cloned with

  git clone https://github.com/arachsys/u-boot-edge.git
  cd u-boot-edge

Alternatively, you can reconstruct this tree yourself like this:

git clone https://gitlab.denx.de/u-boot/u-boot.git
cd u-boot

git fetch https://github.com/openedev/u-boot-amarula.git \
  rk3399-lpddr4:amarula-lpddr4 rk3399-build-target:amarula-parent
git cherry-pick amarula-parent..amarula-lpddr4
# There's a merge conflict to resolve; after fixing, remember to git add
# the resolved file before resuming with git cherry-pick --continue.

sed 's/rock-pi-4/khadas-edge/g' configs/rock-pi-4-rk3399_defconfig \
  >configs/khadas-edge-rk3399_defconfig
cp arch/arm/dts/rk3399-{rock-pi-4,khadas-edge}-u-boot.dtsi

# Put Edge device tree from Linux at arch/arm/dts/rk3399-khadas-edge.dts
# and add rk3399-khadas-edge.dtb to dtb-$(CONFIG_ROCKCHIP_RK3399) in
# arch/arm/dts/Makefile.

git add configs/khadas-edge-rk3399_defconfig
git add arch/arm/dts/{Makefile,rk3399-khadas-edge*.dts*}
git commit -m 'rockchip: rk3399: Add Khadas Edge board support'

Prerequisites to build U-Boot:

  • cross-compiling needs aarch64-linux-gnu (or -musl) gcc and binutils;
  • the Trusted Firmware Cortex-M0 code needs an arm-none-eabi toolchain;
  • python and pyelftools are used to generate a FIT image from bl31.elf.

Fetch and build ARM Trusted Firmware to get bl31 component:

git clone https://github.com/ARM-software/arm-trusted-firmware.git atf
make -C atf CROSS_COMPILE=aarch64-linux-gnu- PLAT=rk3399 bl31
cp atf/build/rk3399/release/bl31/bl31.elf bl31.elf

Build U-Boot:

make khadas-edge-rk3399_defconfig
make ARCH=arm CROSS_COMPILE=aarch64-linux-gnu-

TPL is needed with the default config as the built SPL is too large for the RK3399 boot ROM to load. Write TPL + SPL + U-Boot to the SD card:

tools/mkimage -n rk3399 -T rksd -d tpl/u-boot-tpl-dtb.bin tpl-spl.img
cat spl/u-boot-spl-dtb.bin >>tpl-spl.img
dd if=tpl-spl.img of=/dev/mmcblkN seek=64 conv=fsync
dd if=u-boot.itb of=/dev/mmcblkN seek=16384 conv=fsync

Assuming the SD card already has a suitable partition table, boot partition with extlinux.conf, kernel and dtb, e.g. by starting with an existing image, U-Boot will happily now boot it.

5 Likes

Here it is booting an Armbian image with a random 5.2-rcN kernel I happened to be testing at the time:

U-Boot TPL 2019.07-rc4-00457-gb45e4f05d6-dirty (Jul 06 2019 - 22:28:03 +0100)
Trying to boot from BOOTROM
Returning to boot ROM...

U-Boot SPL 2019.07-rc4-00457-gb45e4f05d6-dirty (Jul 06 2019 - 22:28:03 +0100)
Trying to boot from MMC1

U-Boot 2019.07-rc4-00457-gb45e4f05d6-dirty (Jul 06 2019 - 22:28:03 +0100)

Model: Khadas Edge
DRAM: 3.9 GiB
Cannot find regulator pwm init_voltage
MMC: dwmmc@fe310000: 2, dwmmc@fe320000: 1, sdhci@fe330000: 0
Loading Environment from MMC... *** Warning - bad CRC, using default environment

In: serial@ff1a0000
Out: serial@ff1a0000
Err: serial@ff1a0000
Model: Khadas Edge
Net: No ethernet found.
Hit any key to stop autoboot: 0
switch to partitions #0, OK
mmc0(part 0) is current device
** No partition table - mmc 0 **
switch to partitions #0, OK
mmc1 is current device
Scanning mmc 1:1...
Found /extlinux/extlinux.conf
Retrieving file: /extlinux/extlinux.conf
248 bytes read in 4 ms (60.5 KiB/s)
1: Armbian
Retrieving file: /testkernel
16517128 bytes read in 709 ms (22.2 MiB/s)
append: root=179:2 rootflags=data=writeback rw console=uart8250,mmio32,0xff1a0000 no_console_suspend consoleblank=0 fsck.fix=yes fsck.repair=yes net.ifnames=0
Retrieving file: /dtb/rk3399-khadas-edge.dtb
57536 bytes read in 8 ms (6.9 MiB/s)
## Flattened Device Tree blob at 01f00000
Booting using the fdt blob at 0x1f00000
Loading Device Tree to 00000000f5f15000, end 00000000f5f260bf ... OK

Starting kernel ...

[System then boots]

It will almost certainly be possible to skip the TPL and get the SPL down to a size where it would directly boot with a less ‘kitchen sink’ configuration of U-Boot than the defconfig.

I will continute upstream the code for Edge when this merged…

Current status: http://patchwork.ozlabs.org/patch/1117060/

2 Likes

Yes, I think that sounds sensible to me — mainline U-Boot won’t boot Edge on its own without the ddr4 support, so Jagan’s epic 99-patch lpddr4 series is effectively a dependency.

Nice to see that the existing ddr4 work is completely sufficient to get Edge booting though; there don’t seem to be any other ‘gotchas’ lurking in the shadows!

Given the working device trees in -next and therefore Linux 5.3, it looks like the next releases (Linux 5.3 and U-Boot 2019.10) will fully support Edge in mainline without further patches.

(I think the one remaining issue to debug is the problem with the bcm4359 wifi chipset on the brcmfmac driver. It’s on my list to play with soon, as it’d be nice to get that bug fixed too in time for 5.3.)

1 Like

I’ve updated my tree with an extra patch to remove the slightly annoying/obscure dependency on pyelftools when building u-boot for rockchip, in case it’s useful to anyone.

1 Like

Hello @ChrisW ,

I’m try to send patch to upsteam today.

I follow your instructions to creat the u-boot with SPL support, but I can’t bootup.

The instructions:

git clone https://github.com/arachsys/u-boot-edge.git
cd u-boot-edge
git clone https://github.com/ARM-software/arm-trusted-firmware.git atf
make -C atf CROSS_COMPILE=aarch64-linux-gnu- PLAT=rk3399 bl31
cp atf/build/rk3399/release/bl31/bl31.elf bl31.elf
make khadas-edge-rk3399_defconfig
make ARCH=arm CROSS_COMPILE=aarch64-linux-gnu-
tools/mkimage -n rk3399 -T rksd -d tpl/u-boot-tpl-dtb.bin tpl-spl.img
cat spl/u-boot-spl-dtb.bin >>tpl-spl.img
sudo dd if=tpl-spl.img of=/dev/sdX seek=64 conv=fsync
sudo dd if=u-boot.itb of=/dev/sdX seek=16384 conv=fsync

But can’t bootup, only output the SPL:

U-Boot TPL 2019.07-gd61ba1c (Jul 29 2019 - 17:10:45 +0800)

Do you heve any idea about this?

Thanks.

Hi Nick. That’s really strange, I’m not sure why it’s broken for you; I’ve been using this as my boot loader for the last couple of weeks.

I’m using an Edge with a Captain rather than Edge-V - perhaps there’s a slight different in their device trees that could be the issue? But I thought they were practically identical.

In fact, normal mainline U-Boot works fine now as well. I’ve just built afresh like this:

git clone https://gitlab.denx.de/u-boot/u-boot.git
cd u-boot
git apply ../khadas-edge.patch
git clone https://github.com/ARM-software/arm-trusted-firmware.git atf
make -C atf CROSS_COMPILE=aarch64-linux-gnu- PLAT=rk3399 bl31
cp atf/build/rk3399/release/bl31/bl31.elf bl31.elf
make khadas-edge-rk3399_defconfig
make ARCH=arm CROSS_COMPILE=aarch64-linux-gnu-
tools/mkimage -n rk3399 -T rksd -d tpl/u-boot-tpl-dtb.bin tpl-spl.img
cat spl/u-boot-spl-dtb.bin >>tpl-spl.img
dd if=tpl-spl.img of=/dev/mmcblk0 seek=64 conv=fsync
dd if=u-boot.itb of=/dev/mmcblk0 seek=16384 conv=fsync

and here’s the successful boot happening on the serial console:

U-Boot TPL 2019.07-00999-g75551c8bfc-dirty (Jul 29 2019 - 13:29:24)
Trying to boot from BOOTROM
Returning to boot ROM...

U-Boot SPL 2019.07-00999-g75551c8bfc-dirty (Jul 29 2019 - 13:29:24 +0100)
Trying to boot from MMC1


U-Boot 2019.07-00999-g75551c8bfc-dirty (Jul 29 2019 - 13:29:24 +0100)

Model: Khadas Edge
DRAM:  3.9 GiB
Cannot find regulator pwm init_voltage
MMC:   dwmmc@fe310000: 2, dwmmc@fe320000: 1, sdhci@fe330000: 0
Loading Environment from MMC... *** Warning - bad CRC, using default environment

In:    serial@ff1a0000
Out:   serial@ff1a0000
Err:   serial@ff1a0000
Model: Khadas Edge
rockchip_dnl_key_pressed: adc_channel_single_shot fail!
Net:   No ethernet found.
Hit any key to stop autoboot:  0
switch to partitions #0, OK
mmc0(part 0) is current device
** No partition table - mmc 0 **
switch to partitions #0, OK
mmc1 is current device
Scanning mmc 1:1...
Found /boot/extlinux/extlinux.conf
Retrieving file: /boot/extlinux/extlinux.conf
182 bytes read in 10 ms (17.6 KiB/s)
1:      Linux
Retrieving file: /boot/Image
17600520 bytes read in 749 ms (22.4 MiB/s)
append: root=179:1 rw console=uart8250,mmio32,0xff1a0000 init=/bin/bash
Retrieving file: /boot/rk3399-khadas-edge.dtb
55654 bytes read in 10 ms (5.3 MiB/s)
## Flattened Device Tree blob at 01f00000
   Booting using the fdt blob at 0x1f00000
   Loading Device Tree to 00000000f5f14000, end 00000000f5f24965 ... OK

Starting kernel ...

[    1.332989] rockchip-drm display-subsystem: failed to bind ff940000.hdmi (ops dw_hdmi_rockchip_ops): -517
[    1.342989] rk_gmac-dwmac fe300000.ethernet: phy regulator is not available yet, deferred probing
[    1.652343] rk_gmac-dwmac fe300000.ethernet: cannot get clock clk_mac_speed
bash: cannot set terminal process group (-1): Not a tty
bash: no job control in this shell
bash-4.4#

Unfortunately, the forum won’t allow me to attach the khadas-edge.patch I used to add the device tree and defconfig to U-Boot, but I’ve put it up as a gist at


and attached it to my email reply to you.

Does this work for you, or is it broken on your board in the same way as my patched tree?

Best wishes,
Chris.

Hello @ChrisW

I use your patch,but still can’t boot. It’s strange.

Here is the image I build: https://dl.khadas.com/test/edge-mainline-uboot/

Can you help me to test on your side?

And can you send me the image you build to me to test?

Thanks.

Your image boots my board fine too. (You can see it’s your build running not mine by the +0800 time zone in the U-Boot banners by the way.)

U-Boot TPL 2019.07-00999-g75551c8-dirty (Jul 29 2019 - 21:15:02)
Trying to boot from BOOTROM
Returning to boot ROM...

U-Boot SPL 2019.07-00999-g75551c8-dirty (Jul 29 2019 - 21:15:02 +0800)
Trying to boot from MMC1


U-Boot 2019.07-00999-g75551c8-dirty (Jul 29 2019 - 21:15:02 +0800)

Model: Khadas Edge
DRAM:  3.9 GiB
Cannot find regulator pwm init_voltage
MMC:   dwmmc@fe310000: 2, dwmmc@fe320000: 1, sdhci@fe330000: 0
Loading Environment from MMC... *** Warning - bad CRC, using default environment

In:    serial@ff1a0000
Out:   serial@ff1a0000
Err:   serial@ff1a0000
Model: Khadas Edge
rockchip_dnl_key_pressed: adc_channel_single_shot fail!
Net:   No ethernet found.
Hit any key to stop autoboot:  0 
switch to partitions #0, OK
mmc0(part 0) is current device
** No partition table - mmc 0 **
switch to partitions #0, OK
mmc1 is current device
Scanning mmc 1:1...
Found /boot/extlinux/extlinux.conf
Retrieving file: /boot/extlinux/extlinux.conf
182 bytes read in 9 ms (19.5 KiB/s)
1:      Linux
Retrieving file: /boot/Image
17600520 bytes read in 748 ms (22.4 MiB/s)
append: root=179:1 rw console=uart8250,mmio32,0xff1a0000 init=/bin/bash
Retrieving file: /boot/rk3399-khadas-edge.dtb
55654 bytes read in 10 ms (5.3 MiB/s)
## Flattened Device Tree blob at 01f00000
   Booting using the fdt blob at 0x1f00000
   Loading Device Tree to 00000000f5f14000, end 00000000f5f24965 ... OK

Starting kernel ...

[    1.333219] rockchip-drm display-subsystem: failed to bind ff940000.hdmi (ops dw_hdmi_rockchip_ops): -517
[    1.343176] rk_gmac-dwmac fe300000.ethernet: phy regulator is not available yet, deferred probing
[    1.652806] rk_gmac-dwmac fe300000.ethernet: cannot get clock clk_mac_speed
bash: cannot set terminal process group (-1): Not a tty
bash: no job control in this shell
bash-4.4# 

One thought: I’m booting from an SD card and have a complete blank eMMC on this board. In particular, there’s no boot loader on the onboard eMMC as I wiped the boot loader area and partition table on it with

dd if=/dev/zero of=/dev/mmcblk2 bs=1M count=16 conv=fsync

before I started working on the U-Boot on the SD card, to avoid interference between the two.

I wonder if the TPL is trying to start a valid (but not matching) SPL from the eMMC before the SD card, if you get a hang at the TPL stage? Whereas on my board, eMMC is completely blank and it can’t do that…

I’ve emailed my tpl-spl.img and u-boot.itb as I can’t attach them here.

I’ve erased the eMMC before.

It seems due to the DDR size.

4GB DDR can bootup well, but 2GB can’t. Your board is 4GB so it boots well. But the board I used to test is 2GB, can’t boot. When I switch to the board with 4GB DDR, it also boots up well.

Aha, I remember something like this cropping up on the list in relation to the 2GB vs 4GB version of another board. Is the 2GB board configured as single channel? If so, might be fixed by this patch which is not yet merged - although I remember a couple of 2GB vs 4GB cases that came up, so may be unrelated.

https://lists.denx.de/pipermail/u-boot/2019-July/377554.html

[EDIT: actually this patch was merged a little while back, so it’s something else. Sorry for the incorrect hunch.]

Does the mainline U-Boot work for you on eMMC as well as SD card booting? I’ve not actually got round to testing that yet.

1 Like

It also boots up well from eMMC on the 4GB version.

U-Boot TPL 2019.07-01002-g6ebba11-dirty (Jul 29 2019 - 18:36:09)
Trying to boot from BOOTROM
Returning to boot ROM...

U-Boot SPL 2019.07-01002-g6ebba11-dirty (Jul 29 2019 - 18:36:09 +0800)
Trying to boot from MMC2
Jumping to U-Boot via ARM Trusted Firmware
NOTICE:  BL31: v2.1(release):ac1adfd
NOTICE:  BL31: Built : 23:21:03, Jul 29 2019


U-Boot 2019.07-01002-g6ebba11-dirty (Jul 29 2019 - 23:21:55 +0800)

Model: Khadas Edge-Captain
DRAM:  3.9 GiB
Cannot find regulator pwm init_voltage
MMC:   dwmmc@fe310000: 2, dwmmc@fe320000: 1, sdhci@fe330000: 0
Loading Environment from MMC... Card did not respond to voltage select!
*** Warning - No block device, using default environment

In:    serial@ff1a0000
Out:   serial@ff1a0000
Err:   serial@ff1a0000
Model: Khadas Edge-Captain
rockchip_dnl_key_pressed: adc_channel_single_shot fail!
Net:   
Warning: ethernet@fe300000 (eth0) using random MAC address - 92:1a:4a:3b:ac:54
eth0: ethernet@fe300000
Hit any key to stop autoboot:  0 
kedge# 
kedge# 
kedge# 

This is interesting. Do you see these extra bl31 messages when you boot by SD card too? I’ve been trying to suppress as much of the boot ‘noise’ as possible, just prune it down to errors, so maybe I’ve configured something off compared to your build.

Yes, it is printed from the ATF, so you disable the print message?

Not as far as I know, no. I’m using the defconfig and an unmodified clone of mainline for testing here, although I have been working on a (separate) patch series to make U-Boot less noisy elsewhere.

The relevant bit of trusted firmware (the PT_LOAD segments of bl31.elf) are extracted and linked into U-Boot during the SPL build process, but they don’t seem to be noisy when they’re part of an SPL image as far as I know.

Your U-Boot build doesn’t print those three lines for me either. Maybe you have log level set differently in your environment? Just curious, really, as it’s the only thing I can see that’s different when you run the boot process vs when I do now.

Final update: all the required support is now in mainline u-boot, including the device trees from Nick @numbqq . Build and install with

git clone https://gitlab.denx.de/u-boot/u-boot.git
cd u-boot
git clone https://github.com/ARM-software/arm-trusted-firmware.git atf
make -C atf CROSS_COMPILE=aarch64-linux-gnu- PLAT=rk3399 bl31
cp atf/build/rk3399/release/bl31/bl31.elf bl31.elf
make khadas-edge-rk3399_defconfig
make ARCH=arm CROSS_COMPILE=aarch64-linux-gnu-
tools/mkimage -n rk3399 -T rksd -d tpl/u-boot-tpl-dtb.bin tpl-spl.img
cat spl/u-boot-spl-dtb.bin >>tpl-spl.img
dd if=tpl-spl.img of=/dev/mmcblk0 seek=64 conv=fsync
dd if=u-boot.itb of=/dev/mmcblk0 seek=16384 conv=fsync

Here’s a log of completely unpatched, mainline u-boot successfully booting an Edge from uSD:

U-Boot TPL 2019.10-rc1-00159-g163bc1e4da (Aug 10 2019 - 14:44:38)
Trying to boot from BOOTROM
Returning to boot ROM...

U-Boot SPL 2019.10-rc1-00159-g163bc1e4da (Aug 10 2019 - 14:44:38 +0100)
Trying to boot from MMC1


U-Boot 2019.10-rc1-00159-g163bc1e4da (Aug 10 2019 - 14:44:38 +0100)

Model: Khadas Edge
DRAM:  3.9 GiB
Cannot find regulator pwm init_voltage
MMC:   dwmmc@fe310000: 2, dwmmc@fe320000: 1, sdhci@fe330000: 0
Loading Environment from MMC... *** Warning - bad CRC, using default environment

In:    serial@ff1a0000
Out:   serial@ff1a0000
Err:   serial@ff1a0000
Model: Khadas Edge
rockchip_dnl_key_pressed: adc_channel_single_shot fail!
Net:   No ethernet found.
Hit any key to stop autoboot:  0 
switch to partitions #0, OK
mmc0(part 0) is current device
** No partition table - mmc 0 **
switch to partitions #0, OK
mmc1 is current device
Scanning mmc 1:1...
Found /boot/extlinux/extlinux.conf
Retrieving file: /boot/extlinux/extlinux.conf
182 bytes read in 9 ms (19.5 KiB/s)
1:      Linux
Retrieving file: /boot/Image
17600520 bytes read in 748 ms (22.4 MiB/s)
append: root=179:1 rw console=uart8250,mmio32,0xff1a0000 init=/bin/bash
Retrieving file: /boot/rk3399-khadas-edge.dtb
55654 bytes read in 10 ms (5.3 MiB/s)
## Flattened Device Tree blob at 01f00000
   Booting using the fdt blob at 0x1f00000
   Loading Device Tree to 00000000f5f14000, end 00000000f5f24965 ... OK

Starting kernel ...

[    1.332916] rockchip-drm display-subsystem: failed to bind ff940000.hdmi (ops dw_hdmi_rockchip_ops): -517
[    1.342949] rk_gmac-dwmac fe300000.ethernet: phy regulator is not available yet, deferred probing
[    1.649012] rk_gmac-dwmac fe300000.ethernet: cannot get clock clk_mac_speed
bash: cannot set terminal process group (-1): Not a tty
bash: no job control in this shell
bash-4.4# 

Only remaining issue is that there may still be a problem with the 2GB Edge boards — presumably a slightly different lpddr4 configuration to the 4GB models?

1 Like

Finally it merged to the master brabch :wink:

1 Like

No idea yet, but I will stay concerned about this issue…