| View previous topic :: View next topic |
| Author |
Message |
jpsollie
Guru
Joined: 17 Aug 2013
Posts: 323
|
 Posted: Thu Feb 25, 2021 5:19 am Post subject: porting gentoo to a new system (RK3328) [SOLVED] |
 |
|
Hi everyone,
In order to reduce power consumption, I bought myself a nanoPI neo 3, to which I would like to offload most DNS/DHCP/etc... task, which are currently done by a power-hungry threadripper device.
It contains a RK3328 chip, and no GUI (but I ordered a serial port output, so at least I got something)
It''s not "supported" by Gentoo yet, so I'd like some advice:
1) The people on the site say only SD card booting is available, so I thought it would be a good idea to:
- Install GRUB on a SD card, chainload to PXE booting
- put a PXE kernel onto the tftp server I currently have, and point dhcpd to provide that one, so it could boot from LAN, and provide the stage3 image via NFS
- remove the SD card, and run the normal installation procedure, via telnet / ssh
2) Due to the limited amout of CPU power, is it a good idea to offload gcc compiling to AMD64 distcc servers?
3) What about kernel functionality? Does it need additional patches?
thank you for your advice!
_________________
The power of Gentoo optimization (not overclocked): [img]https://www.passmark.com/baselines/V10/images/503714802842.png[/img]
Last edited by jpsollie on Sun Apr 18, 2021 6:26 am; edited 1 time in total |
|
| Back to top |
|
 |
joanandk
Apprentice
Joined: 12 Feb 2017
Posts: 169
|
| Posted: Thu Feb 25, 2021 7:15 am Post subject: Re: porting gentoo to a new system (RK3328) |
|
|
| jpsollie wrote: | | In order to reduce power consumption, I bought myself a nanoPI neo 3, to which I would like to offload most DNS/DHCP/etc... task, which are currently done by a power-hungry threadripper device. |
| jpsollie wrote: | | put a PXE kernel onto the tftp server I currently have |
If your tftp server is running on the threadripper, there will be no reduction in power consumption. There would be even more energy consumption as you have an additional Pi running.
| jpsollie wrote: | 1) The people on the site say only SD card booting is available, so I thought it would be a good idea to:
- Install GRUB on a SD card, chainload to PXE booting
- put a PXE kernel onto the tftp server I currently have, and point dhcpd to provide that one, so it could boot from LAN, and provide the stage3 image via NFS
- remove the SD card, and run the normal installation procedure, via telnet / ssh |
Depends on how good your Pi can handle a short or long interruption of access to the TFTP server. I have been experimenting with a read only system with the home on a NFS server. This setup works very well as long as there is no longer interruption of the network access. As your Pi is handling vital network configuration (DNS, DHCP), if it goes down, your entire network could get unstable or inaccessible.
You could use an usb external ssd instead of TFTP.
BR |
|
| Back to top |
|
|
jpsollie
Guru
Joined: 17 Aug 2013
Posts: 323
|
| Posted: Thu Feb 25, 2021 9:36 am Post subject: Re: porting gentoo to a new system (RK3328) |
|
|
| joanandk wrote: | | jpsollie wrote: | | In order to reduce power consumption, I bought myself a nanoPI neo 3, to which I would like to offload most DNS/DHCP/etc... task, which are currently done by a power-hungry threadripper device. |
| jpsollie wrote: | | put a PXE kernel onto the tftp server I currently have |
If your tftp server is running on the threadripper, there will be no reduction in power consumption. There would be even more energy consumption as you have an additional Pi running.
| jpsollie wrote: | 1) The people on the site say only SD card booting is available, so I thought it would be a good idea to:
- Install GRUB on a SD card, chainload to PXE booting
- put a PXE kernel onto the tftp server I currently have, and point dhcpd to provide that one, so it could boot from LAN, and provide the stage3 image via NFS
- remove the SD card, and run the normal installation procedure, via telnet / ssh |
Depends on how good your Pi can handle a short or long interruption of access to the TFTP server. I have been experimenting with a read only system with the home on a NFS server. This setup works very well as long as there is no longer interruption of the network access. As your Pi is handling vital network configuration (DNS, DHCP), if it goes down, your entire network could get unstable or inaccessible.
You could use an usb external ssd instead of TFTP.
BR |
No, I must have explained it the wrong way: the idea was: load stage3 only over network + install gentoo, not daily run!
the SD card would first be written with grub, and grub would need to load the PXE image from tftp, so the booting process can start.
When everything is loaded & up and running, I would install gentoo on the SD card.
_________________
The power of Gentoo optimization (not overclocked): [img]https://www.passmark.com/baselines/V10/images/503714802842.png[/img] |
|
| Back to top |
|
|
Hu
Administrator
Joined: 06 Mar 2007
Posts: 22655
|
| Posted: Thu Feb 25, 2021 5:11 pm Post subject: |
|
|
| Beware that part of the gcc build cannot be distributed. If CPU speed is a concern, you will need to arrange a more elaborate way to avoid building gcc on the device, such as building a compatible gcc on another system and delivering it as a prebuilt binary package. You could choose, at least initially, to build gcc on the device with distcc help, and only look at other options if the native build is too slow. |
|
| Back to top |
|
|
jpsollie
Guru
Joined: 17 Aug 2013
Posts: 323
|
| Posted: Thu Feb 25, 2021 6:09 pm Post subject: |
|
|
| Hu wrote: | | Beware that part of the gcc build cannot be distributed. If CPU speed is a concern, you will need to arrange a more elaborate way to avoid building gcc on the device, such as building a compatible gcc on another system and delivering it as a prebuilt binary package. You could choose, at least initially, to build gcc on the device with distcc help, and only look at other options if the native build is too slow. |
Good point. But fortunately, gcc only releases a new version once 6 months, so if I renice that one, it may be a boost for all other packages 
_________________
The power of Gentoo optimization (not overclocked): [img]https://www.passmark.com/baselines/V10/images/503714802842.png[/img] |
|
| Back to top |
|
|
Ralphred
l33t
Joined: 31 Dec 2013
Posts: 653
|
| Posted: Sat Feb 27, 2021 6:55 am Post subject: Re: porting gentoo to a new system (RK3328) |
|
|
| jpsollie wrote: | | I would like to offload most DNS/DHCP/etc... task! |
I'm not sure about offloading DNS to a Pi, depends what your network traffic looks like, and if you are going to have it just forward requests or look them up.
The reason I say this, when my sister visits with her kids and they are browsing nasty sites (pintrest facebook or whatever the kids like these days) they used to bring named to it's knees with DNS requests for dead domains, and that runs on a dual core Xeon.
In the end I split the DHCP pool, so only the local machines get results from named now.
| jpsollie wrote: |
It''s not "supported" by Gentoo yet, so I'd like some advice:
1) The people on the site say only SD card booting is available, so I thought it would be a good idea to:
- Install GRUB on a SD card, chainload to PXE booting
- put a PXE kernel onto the tftp server I currently have, and point dhcpd to provide that one, so it could boot from LAN, and provide the stage3 image via NFS
- remove the SD card, and run the normal installation procedure, via telnet / ssh
|
This seems over complicated, last time I built something like this I used qemu so I could basically install an aarch64 stage-3 chroot on my desktop, and finish the entire disk from inside there, not as fast as native compilation but still quicker than native on the Pi. |
|
| Back to top |
|
|
jpsollie
Guru
Joined: 17 Aug 2013
Posts: 323
|
| Posted: Sat Feb 27, 2021 3:50 pm Post subject: Re: porting gentoo to a new system (RK3328) |
|
|
| Ralphred wrote: | | jpsollie wrote: | | I would like to offload most DNS/DHCP/etc... task! |
I'm not sure about offloading DNS to a Pi, depends what your network traffic looks like, and if you are going to have it just forward requests or look them up.
The reason I say this, when my sister visits with her kids and they are browsing nasty sites (pintrest facebook or whatever the kids like these days) they used to bring named to it's knees with DNS requests for dead domains, and that runs on a dual core Xeon.
In the end I split the DHCP pool, so only the local machines get results from named now.
| jpsollie wrote: |
It''s not "supported" by Gentoo yet, so I'd like some advice:
1) The people on the site say only SD card booting is available, so I thought it would be a good idea to:
- Install GRUB on a SD card, chainload to PXE booting
- put a PXE kernel onto the tftp server I currently have, and point dhcpd to provide that one, so it could boot from LAN, and provide the stage3 image via NFS
- remove the SD card, and run the normal installation procedure, via telnet / ssh
|
This seems over complicated, last time I built something like this I used qemu so I could basically install an aarch64 stage-3 chroot on my desktop, and finish the entire disk from inside there, not as fast as native compilation but still quicker than native on the Pi. |
Good to know! Any idea whether a 4xA53 core cpu could be overloaded with 10 clients on a system running recursive bind + dhcp only for a 100mbit connection? I *think* ik should be fine, no?
_________________
The power of Gentoo optimization (not overclocked): [img]https://www.passmark.com/baselines/V10/images/503714802842.png[/img] |
|
| Back to top |
|
|
jpsollie
Guru
Joined: 17 Aug 2013
Posts: 323
|
| Posted: Mon Mar 22, 2021 6:28 am Post subject: |
|
|
Small update:
I *almost* made it work,
but I have an issue booting the kernel via u-boot:
it reports filesystem error 500000 when trying to access /dev/mmcblk0p4 and is unable to load the /Image, while it can read the /extlinux/extlinux.conf config file.
What is this filesystem error? I tried both fat16 and fat32, and fat filesystem support is enabled in U-BOOT, but I have no idea what goes wrong.
it should load the 'Image' vanilla kernel, but guess I'm doing something wrong ...
Any hints? My knowledge of U-BOOT (and bootloaders in general) is quite limited, even on X86-64 I load my linux images directly via EFI and not grub.
_________________
The power of Gentoo optimization (not overclocked): [img]https://www.passmark.com/baselines/V10/images/503714802842.png[/img] |
|
| Back to top |
|
|
jpsollie
Guru
Joined: 17 Aug 2013
Posts: 323
|
| Posted: Mon Mar 22, 2021 2:39 pm Post subject: |
|
|
skipping EFI and going to booti instead solved the problem ...
At least, this part:
I can't boot the kernel ...
Could someone help me what I *possibly* could do to get at least some info from the device?
The info from U-boot:
| Code: |
nanopi-r2# load mmc 1:4 ${fdt_addr_r} rockchip/rk3328-nanopi-r2s.dtb
35450 bytes read in 6 ms (5.6 MiB/s)
nanopi-r2# confinfo
Unknown command 'confinfo' - try 'help'
nanopi-r2# coninfo
List of available devices:
serial@ff130000 00000007 IO stdin stdout stderr
serial 00000003 IO
nc 00000003 IO
nanopi-r2# load mmc 1:4 ${kernel_addr_r} efi/boot/bootaa64.efi
11173896 bytes read in 508 ms (21 MiB/s)
nanopi-r2# load mmc 1:4 ${fdt_addr_r} rockchip/rk3328-nanopi-r2s.dtb
52081 bytes read in 7 ms (7.1 MiB/s)
nanopi-r2# fdt addr ${fdt_addr_r}
nanopi-r2# fdt list
/ {
compatible = "friendlyarm,nanopi-r2s", "rockchip,rk3328";
interrupt-parent = <0x00000001>;
#address-cells = <0x00000002>;
#size-cells = <0x00000002>;
model = "FriendlyElec NanoPi R2S";
aliases {
};
cpus {
};
opp_table0 {
};
bus {
};
analog-sound {
};
arm-pmu {
};
display-subsystem {
};
hdmi-sound {
};
psci {
};
timer {
};
xin24m {
};
i2s@ff000000 {
};
i2s@ff010000 {
};
i2s@ff020000 {
};
spdif@ff030000 {
};
pdm@ff040000 {
};
syscon@ff100000 {
};
serial@ff110000 {
};
serial@ff120000 {
};
serial@ff130000 {
};
i2c@ff150000 {
};
i2c@ff160000 {
};
i2c@ff170000 {
};
i2c@ff180000 {
};
spi@ff190000 {
};
watchdog@ff1a0000 {
};
pwm@ff1b0000 {
};
pwm@ff1b0010 {
};
pwm@ff1b0020 {
};
pwm@ff1b0030 {
};
thermal-zones {
};
tsadc@ff250000 {
};
efuse@ff260000 {
};
adc@ff280000 {
};
gpu@ff300000 {
};
iommu@ff330200 {
};
iommu@ff340800 {
};
video-codec@ff350000 {
};
iommu@ff350800 {
};
iommu@ff360480 {
};
vop@ff370000 {
};
iommu@ff373f00 {
};
hdmi@ff3c0000 {
};
codec@ff410000 {
};
phy@ff430000 {
};
clock-controller@ff440000 {
};
syscon@ff450000 {
};
mmc@ff500000 {
};
mmc@ff510000 {
};
mmc@ff520000 {
};
ethernet@ff540000 {
};
ethernet@ff550000 {
};
usb@ff5c0000 {
};
usb@ff5d0000 {
};
usb@ff580000 {
nanopi-r2# load mmc 1:4 ${kernel_addr_r} efi/boot/bootaa64.efi
11173896 bytes read in 508 ms (21 MiB/s)
nanopi-r2# load mmc 1:4 ${fdt_addr_r} rockchip/rk3328-nanopi-r2s.dtb
52081 bytes read in 7 ms (7.1 MiB/s)
nanopi-r2# booti ${kernel_addr_r} ${fdt_addr_r}
Moving Image from 0x2080000 to 0x2200000, end=2d00000
## Loading ramdisk from FIT Image at 01f00000 ...
Bad FIT ramdisk image format! (err=-42)
Ramdisk image is corrupt or invalid
nanopi-r2# bootz ${kernel_addr_r} ${fdt_addr_r}
nanopi-r2# mii device
MII devices: 'ethernet@ff540000'
Current device: 'ethernet@ff540000'
nanopi-r2# booti ${kernel_addr_r} ${fdt_addr_r}
Moving Image from 0x2080000 to 0x2200000, end=2d00000
## Loading ramdisk from FIT Image at 01f00000 ...
Bad FIT ramdisk image format! (err=-42)
Ramdisk image is corrupt or invalid
nanopi-r2# env print
arch=arm
baudrate=115200
board=nanopi_r2
board_name=nanopi_r2
boot_a_script=load ${devtype} ${devnum}:${distro_bootpart} ${scriptaddr} ${prefix}${script}; source ${scriptaddr}
boot_efi_binary=load ${devtype} ${devnum}:${distro_bootpart} ${kernel_addr_r} efi/boot/bootaa64.efi; if fdt addr ${fdt_addr_r};
then bootefi ${kernel_addr_r} ${fdt_addr_r};else bootefi ${kernel_addr_r} ${fdtcontroladdr};fi
boot_efi_bootmgr=if fdt addr ${fdt_addr_r}; then bootefi bootmgr ${fdt_addr_r};else bootefi bootmgr;fi
boot_extlinux=sysboot ${devtype} ${devnum}:${distro_bootpart} any ${scriptaddr} ${prefix}${boot_syslinux_conf}
boot_net_usb_start=usb start
boot_prefixes=/ /boot/
boot_script_dhcp=boot.scr.uimg
boot_scripts=boot.scr.uimg boot.scr
boot_syslinux_conf=extlinux/extlinux.conf
boot_targets=mmc0 mmc1 usb0 pxe dhcp
bootcmd=bootrkp; run distro_bootcmd
bootcmd_dhcp=setenv devtype dhcp; run boot_net_usb_start; if dhcp ${scriptaddr} ${boot_script_dhcp}; then source ${scriptaddr};
fi;setenv efi_fdtfile ${fdtfile}; setenv efi_old_vci ${bootp_vci};setenv efi_old_arch ${bootp_arch};setenv bootp_vci
PXEClient:Arch:00011:UNDI:003000;setenv bootp_arch 0xb;if dhcp ${kernel_addr_r}; then tftpboot ${fdt_addr_r} dtb/${efi_fdtfile};
if fdt addr ${fdt_addr_r}; then bootefi ${kernel_addr_r} ${fdt_addr_r}; else bootefi ${kernel_addr_r} ${fdtcontroladdr};fi;
fi;setenv bootp_vci ${efi_old_vci};setenv bootp_arch ${efi_old_arch};setenv efi_fdtfile;setenv efi_old_arch;setenv efi_old_vci;
bootcmd_mmc0=devnum=0; run mmc_boot
bootcmd_mmc1=devnum=1; run mmc_boot
bootcmd_pxe=run boot_net_usb_start; dhcp; if pxe get; then pxe boot; fi
bootcmd_usb0=devnum=0; run usb_boot
bootdelay=8
cpu=armv8
distro_bootcmd=for target in ${boot_targets}; do run bootcmd_${target}; done
efi_dtb_prefixes=/ /dtb/ /dtb/current/
fdt_addr_r=0x01f00000
fdtcontroladdr=7df241b0
fdtfile=rockchip/rk3328-nanopi-r2s.dtb
fileaddr=1f00000
filesize=cb71
kernel_addr_r=0x02080000
load_efi_dtb=load ${devtype} ${devnum}:${distro_bootpart} ${fdt_addr_r} ${prefix}${efi_fdtfile}
mmc_boot=if mmc dev ${devnum}; then devtype=mmc; run scan_dev_for_boot_part; fi
partitions=uuid_disk=${uuid_gpt_disk};name=loader1,start=32K,size=4000K,uuid=${uuid_gpt_loader1};name=loader2,start=8MB,size=4MB,
uuid=${uuid_gpt_loader2};name=trust,size=4M,uuid=${uuid_gpt_atf};name=boot,size=112M,bootable,uuid=${uuid_gpt_boot};
name=rootfs,size=-,uuid=B921B045-1DF0-41C3-AF44-4C6F280D3FAE;
pxefile_addr_r=0x00600000
ramdisk_addr_r=0x06000000
scan_dev_for_boot=echo Scanning ${devtype} ${devnum}:${distro_bootpart}...; for prefix in ${boot_prefixes}; do run scan_dev_for_extlinux;
run scan_dev_for_scripts; done;run scan_dev_for_efi;
scan_dev_for_boot_part=part list ${devtype} ${devnum} -bootable devplist; env exists devplist || setenv devplist 1; for distro_bootpart in ${devplist};
do if fstype ${devtype} ${devnum}:${distro_bootpart} bootfstype; then run scan_dev_for_boot; fi; done; setenv devplist
scan_dev_for_efi=setenv efi_fdtfile ${fdtfile}; for prefix in ${efi_dtb_prefixes}; do if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${efi_fdtfile};
then run load_efi_dtb; fi;done;run boot_efi_bootmgr;if test -e ${devtype} ${devnum}:${distro_bootpart} efi/boot/bootaa64.efi;
then echo Found EFI removable media binary efi/boot/bootaa64.efi; run boot_efi_binary; echo EFI LOAD FAILED: continuing...; fi; setenv efi_fdtfile
scan_dev_for_extlinux=if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${boot_syslinux_conf}; then echo Found ${prefix}${boot_syslinux_conf};
run boot_extlinux; echo SCRIPT FAILED: continuing...; fi
scan_dev_for_scripts=for script in ${boot_scripts}; do if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${script}; then echo Found U-Boot script ${prefix}${script};
run boot_a_script; echo SCRIPT FAILED: continuing...; fi; done
scriptaddr=0x00500000
soc=rk3328
stderr=serial@ff130000
stdin=serial@ff130000
stdout=serial@ff130000
usb_boot=usb start; if usb dev ${devnum}; then devtype=usb; run scan_dev_for_boot_part; fi
vendor=rockchip
Environment size: 4021/32764 bytes
nanopi-r2# booti ${kernel_addr_r}
Moving Image from 0x2080000 to 0x2200000, end=2d00000
|
... and then it hangs.
I tried adding the following arguments directly into kernel, to make sure they would not be passed improperly:
| Quote: |
CONFIG_CMDLINE="earlycon=uart8250,mmio,0xff130000,115200n8 console=uart8250,mmio32,0xff130000,115200n8 root=/dev/mmcblk0p5 rootwait rootfstype=ext4 init=/sbin/init debug ignore_loglevel netconsole=6666@192.168.177.30/eth0,6666@192.168.177.29/e0:d5:5e:2a:d6:3c"
|
but netcat doesn't see anything on port 6666 on the pc with IP 29, and I do not see anything on UART either!
any idea where I can get any info? at least something? I've really nothing to think of any longer 
Thank you!!
Mod edit: Long lines split to preserve Forum formatting. — JRG
_________________
The power of Gentoo optimization (not overclocked): [img]https://www.passmark.com/baselines/V10/images/503714802842.png[/img] |
|
| Back to top |
|
|
jpsollie
Guru
Joined: 17 Aug 2013
Posts: 323
|
| Posted: Sun Apr 18, 2021 6:25 am Post subject: |
|
|
small update here:
netconsole did not work as I accidentially forgot to include it in the kernel
init did not work as busybox took over /sbin/init (which requires a different inittab file)
I made it work! I'll dump the gained knowledge into the wiki
_________________
The power of Gentoo optimization (not overclocked): [img]https://www.passmark.com/baselines/V10/images/503714802842.png[/img] |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Gentoo on ARM
