FreeBSD geli zfs mirror add drives boot problem solution

I was using zfs mirrored zroot pool on top of geli encrypted drives on
FreeBSD 13.

I needed to add two more drives to the pool, to get more space.

After reboot, efi loader didn't ask for password for a new drives and since
zfs pool was already spanned on both 4 drives, loader threw only a chain of
"zio_read error: 5" and the system didn't start.

-------

After booting from a usbstick, manually attaching geli drives
(geli attach ada0; ...) importing the zpool (zpool import zroot), I felt
relief that I didn't loose any data at least.

After two nights of trial-and-failures, conslusion is:

* for efi geli password prompt, not only -b parameter used with geli init is
  neccessary, but also you need -g parameter, from some strange reason
  so use:
  geli init -b -g -l 256 partitionname
  to create it

* furthermore, from some strange reason, efi geli attacher need also a small 
  freebsd-boot partition on each drive it got to mount

* correct sequence to add a drive to zfs pool zroot is therefore (ada3 as an
  example):

  # wipe out gpt table
  gpart destroy -F ada3

  #create a gpt table
  gpart create -s gpt ada3

  #create a boot partition
  gpart add -t freebsd-boot -s 512k ada3

  #insert bootcode to partition 1
  gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada3

  #create a partition that will host zfs over geli
  gpart add -t freebsd-zfs ada3

  #check
  gpart show ada3

  #init geli with 256 keylength 
  geli init -b -g -l 256 ada3p2

  #attach
  geli attach ada3p2

  #if booted from a stick, import zroot
  zfspool import zroot

  #add new geli to a pool (assuming ada2p2 already there, otherwise somehow
  #else)
  zfspool attach zroot ada2p2.eli ada3p2.eli

  #wait for resilver
  zfspool status

  #export if booted from a stick
  zpool export zroot

  #say a prayer and
  reboot


EOF

Comments requested

~~~~~~~~~
Binary Sxizophreny - index of comp related stuff
Kangaroo's Homepage (czech)