My killer optimizations for Gentoo on Raspberry Pi 3 B+.

[crocket]

Reducing Writes

• Disable anacron USE flag in cronie because anacron writes to disk. Cronie doesn't write to disk.
  
• Don't run an SMTP server because it keeps its states on disk. An SMTP client like msmtp doesn't keep its runtime states on disk.
  
• Redirect non-essential writes to RAM through /etc/fstab
  

  Code:

  tmpfs /tmp tmpfs mode=1777,noatime 0 0 tmpfs /var/tmp tmpfs mode=1777,noatime 0 0 tmpfs /var/log tmpfs mode=0755,noatime 0 0 tmpfs /var/cache tmpfs mode=0755,noatime 0 0 tmpfs /var/lib/dhcpcd tmpfs mode=0755,noatime 0 0 tmpfs /var/lib/ntp tmpfs mode=0755,uid=ntp,gid=ntp,noatime 0 0 tmpfs /var/lib/misc tmpfs mode=0755,noatime 0 0 tmpfs /var/lib/run tmpfs mode=0755,noatime 0 0 tmpfs /var/lib/syslog-ng tmpfs mode=0700,noatime 0 0

  
  
• Set sync=disabled and logbias=throughput in the root ZFS dataset so that ZIL is disabled. ZIL doubles write amplification factor.
  
• Keep /usr/portage empty on raspberry pi because updating portage tree on raspberry pi consumes precious microSD write cycles. I can build everything in an arm64 chroot.
  
• You can batch writes by updating everything in a ZFS arm64 chroot, taking a ZFS snapshot, and sending the ZFS snapshot to Raspberry Pi.
  
• Set SAVE_ON_STOP="no" in /etc/conf.d/alsasound to prevent /etc/init.d/alsasound from writing to disk during shutdown.
  
• Disable less history logging in ~/.bash_profile. With loksh, it's ~/.profile
  

  Code:

  export LESSHISTFILE="-"

  
  
• Disable bash history logging in ~/.bashrc. With loksh, you don't need to unset HISTFILE.
  

  Code:

  unset HISTFILE

  
  
• Disable shada and swapfile in ~/.config/nvim/init.vim
  

  Code:

  set shada="" set noswapfile

  
  

Other optimizations

• /etc/local.d/turn-off-wifi-power-saving.start disables wifi power saving which can lead to frequent WiFi disconnections. Install net-wireless/iw to get iw.
  

  Code:

  #!/bin/sh iw dev wlan0 set power_save off

  
  
• Use net-misc/ntp instead of net-misc/openntpd
  
  • Because Raspberry Pi doesn't save time in hardware clock, time needs to be set after boot. net-misc/openntpd waits for only 15 seconds for an internet connection. After 15 seconds, openntpd fails to set the time automatically forever. It can often take more than 15 seconds for dhcpcd to fetch an IP address. On the other hand, net-misc/ntp waits for an internet connection in the background and sets the time properly in a few minutes.
  
  
• /boot/cmdline.txt
  

  Code:

  rootwait rootfstype=zfs root=ZFS=rpi/ROOT/gentoo zfs.zfs_arc_max=67108864 dwc_otg.lpm_enable=0 console=tty1 console=ttyAMA0,115200

  
  
  • zfs.zfs_arc_max=67108864 limits the maximum size of ZFS ARC to 64MB because Raspberry Pi 3 B+ has only 1GB of RAM.
    
  • dwc_otg.lpm_enable=0 disables USB controller power management which can lead to unexpected problems.
    
  • console=tty1 console=ttyAMA0,115200 displays kernel messages on both tty1 and UART console and shows OpenRC messages only on UART console. UART console is a lot more convenient than an HDMI cable and a USB keyboard.
    
  
  
• /boot/config.txt
  

  Code:

  # have a properly sized image disable_overscan=1 # Disable sound over HDMI hdmi_drive=2 # gpu_mem is for closed-source driver only; since we are only using the # open-source driver here, set low gpu_mem=16 initramfs initramfs-4.19.30-v8+.img followkernel dtparam=audio=on dtoverlay=pi3-disable-bt

  
  
  • dtoverlay=pi3-disable-bt disables bluetooth and assigns PL011 module to GPIO UART pins because miniUART and bluetooth suck.
    
  • dtparam=audio=on enables audio.
    
  • initramfs initramfs-4.19.30-v8+.img followkernel makes linux kernel use initramfs-4.19.30-v8+.img as the initial ramdisk. This initial ramdisk contains ZFS kernel modules needed for booting ZFS root.
    
  • gpu_mem=16 because I don't need a graphical user interface from Raspberry Pi.
    
  
  

[erm67]

I don't know if ZFS on linux now has finally a working zero copy mechanism between linux file cache and ARC, once upon a time it didn't and copying stuff between the 2 caches wasn't a great idea ...
I guess you can make the ARC even smaller, I remember that 40mb ARC was actually faster and worked, you can't turn it off completely however.
Maybe you have read on ZFS documentation that the ARC is a lot more efficient than LRU caches, but they forgot to tell you that linux doesn't use LRU caches since version 2.2 and the algorithms used by recent linux are more efficient than the ARC.

zfs has a property called volblocksize and you should probably tune it carefully for an sd card since often there is an optimal blocksize to use. I don't know what is the equivalent of the stripe size for ZFS but other filesystem let specify an optimal size to be used for io and setting it to the value of the erase block size, or the best performing block size found by flashbench, will improve performance and reduce write amplification.
https://raspberrypi.stackexchange.com/questions/32884/why-does-the-sd-card-random-write-performance-for-record-size-8-128-kb-drops-bel
The internals of SD cards are top secret and not disclosed by manufacturer so the kernel uses fixed values for sd cards that are often not optimal:
https://www.kernel.org/doc/Documentation/mmc/mmc-dev-attrs.txt

linaro has developed a tool to benchmark sd cards using different block sizes to find the optimal block size and the erase block size for your card:
https://github.com/bradfa/flashbench

There is also a good tool called sdinfo
https://github.com/johnlane/sdinfo

that will decode some registers provided by the card to print some useful information.

Also the readahead to be tuned to the best performing block size (the erase block size) determined by flashbench.

For example on my card:

[crocket]

[erm67]

[crocket]

[erm67]

Sorry for the delay but I was busy ..

It looks like it is more or less like mine ... different cards same values, I am starting to think it is useless or filled with the same values for all sd cards ...

Don't forget to tune for your card /sys/block/mmcblk0/queue/max_sectors_kb in theory it should be equal to the erase sector size but linux uses the same values for all cards. Your CCD suggest to use 64kb (erase sector size) but again it's the same as mine, and maybe the same for all sd cards .....

[crocket]

[erm67]