Update Alpine, Arch, Fedora and RHEL root on ZFS guides

Signed-off-by: Maurice Zhou <jasper@apvc.uk>

docs/Getting Started/Alpine Linux/Root on ZFS/1-preparation.rst

@@ -6,9 +6,12 @@ Preparation
 .. contents:: Table of Contents
    :local:
 
-#. Disable Secure Boot. ZFS modules can not be loaded if Secure Boot is enabled.
+#. Disable Secure Boot. ZFS modules can not be loaded if
-#. Download latest extended variant of `Alpine Linux live image
+   Secure Boot is enabled.
-   <https://dl-cdn.alpinelinux.org/alpine/latest-stable/releases/x86_64/>`__ and boot from it.
+#. Download latest extended variant of `Alpine Linux
+   live image
+   <https://dl-cdn.alpinelinux.org/alpine/latest-stable/releases/x86_64/>`__
+   and boot from it.
 #. Login as root user.  There is no password.
 #. Configure Internet::
 
@@ -44,15 +47,15 @@ Preparation
    and add it to package manager configuration::
 
      tee -a /etc/apk/repositories <<EOF
-     https://dl-5.alpinelinux.org/alpine/latest-stable/community/
+     https://dl-cdn.alpinelinux.org/alpine/latest-stable/community/
-     https://dl-5.alpinelinux.org/alpine/latest-stable/main/
+     https://dl-cdn.alpinelinux.org/alpine/latest-stable/main/
      EOF
 
 #. Throughout this guide, we use predictable disk names generated by udev::
 
      apk update
      apk add eudev
-     setup-udev
+     setup-devd udev
 
    It can be removed after reboot with ``setup-devd mdev && apk del eudev``.
 
@@ -60,9 +63,9 @@ Preparation
 
    List available disks with::
 
-    ls /dev/disk/by-id/*
+    find /dev/disk/by-id/
 
-   If using virtio as disk bus, use ``/dev/disk/by-path/*``.
+   If using virtio as disk bus, use ``/dev/disk/by-path/``.
 
    Declare disk array::
 
@@ -72,12 +75,24 @@ Preparation
 
     DISK='/dev/disk/by-id/disk1'
 
+#. Set partition size:
+
+   Set swap size, set to 1 if you don't want swap to
+   take up too much space::
+
+    INST_PARTSIZE_SWAP=4
+
+   Root pool size, use all remaining disk space if not set::
+
+    INST_PARTSIZE_RPOOL=
+
 #. Install ZFS support and partition tool::
 
-    apk add zfs zfs-lts sgdisk e2fsprogs
+    apk add zfs zfs-lts sgdisk e2fsprogs cryptsetup util-linux
     modprobe zfs
 
-   Many errors about firmware will appear.  They are safe to ignore.
+   Many errors about firmware will appear.  They are
+   safe to be ignored.
 
 #. Install bootloader for both legacy boot and UEFI::
 

docs/Getting Started/Alpine Linux/Root on ZFS/2-system-installation.rst

@@ -10,13 +10,18 @@ System Installation
 
      for i in ${DISK}; do
 
+     # wipe flash-based storage device to improve
+     # performance.
+     # ALL DATA WILL BE LOST
+     # blkdiscard -f $i
+
      sgdisk --zap-all $i
 
      sgdisk -n1:1M:+1G -t1:EF00 $i
 
      sgdisk -n2:0:+4G -t2:BE00 $i
 
-     test -z $INST_PARTSIZE_SWAP || sgdisk -n4:0:+${INST_PARTSIZE_SWAP}G -t4:8200 $i
+     sgdisk -n4:0:+${INST_PARTSIZE_SWAP}G -t4:8200 $i
 
      if test -z $INST_PARTSIZE_RPOOL; then
          sgdisk -n3:0:0   -t3:BF00 $i
@@ -25,17 +30,15 @@ System Installation
      fi
 
      sgdisk -a1 -n5:24K:+1000K -t5:EF02 $i
+
+     sync && udevadm settle && sleep 3 
+
+     cryptsetup open --type plain --key-file /dev/random $i-part4 ${i##*/}-part4
+     mkswap /dev/mapper/${i##*/}-part4
+     swapon /dev/mapper/${i##*/}-part4 
      done
 
-#. Probe new partitions::
+#. Create boot pool::
-
-    for i in ${DISK}; do
-      partprobe $i
-    done
-    udevadm settle
-    sync
-
-#. Create boot partition::
 
      tee -a /root/grub2 <<EOF
      # Features which are supported by GRUB2
@@ -73,6 +76,18 @@ System Installation
   
    If not using a multi-disk setup, remove ``mirror``.
 
+   You should not need to customize any of the options for the boot pool.
+
+   GRUB does not support all of the zpool features. See ``spa_feature_names``
+   in `grub-core/fs/zfs/zfs.c
+   <http://git.savannah.gnu.org/cgit/grub.git/tree/grub-core/fs/zfs/zfs.c#n276>`__.
+   This step creates a separate boot pool for ``/boot`` with the features
+   limited to only those that GRUB supports, allowing the root pool to use
+   any/all features.
+
+   Features enabled with ``-o compatibility=grub2`` can be seen
+   `here <https://github.com/openzfs/zfs/blob/master/cmd/zpool/compatibility.d/grub2>`__.
+
 #. Create root pool::
 
        zpool create \
@@ -95,18 +110,19 @@ System Installation
 
    If not using a multi-disk setup, remove ``mirror``.
 
-#. This section implements dataset layout as described in `overview <1-preparation.html>`__.
+#. Create root system container:
-
-   Create root system container:
 
    - Unencrypted::
 
       zfs create \
        -o canmount=off \
        -o mountpoint=none \
-       rpool/alpine
+       rpool/alpinelinux
 
-   - Encrypted::
+   - Encrypted:
+
+     Pick a strong password. Once compromised, changing password will not keep your
+     data safe. See ``zfs-change-key(8)`` for more info::
 
       zfs create \
        -o canmount=off \
@@ -114,17 +130,39 @@ System Installation
        -o encryption=on \
        -o keylocation=prompt \
        -o keyformat=passphrase \
-       rpool/alpine
+       rpool/alpinelinux
 
-#. Create datasets::
+   You can automate this step (insecure) with: ``echo POOLPASS | zfs create ...``.
 
-     zfs create -o canmount=on -o mountpoint=/     rpool/alpine/root
+   Create system datasets, let Alpinelinux declaratively
-     zfs create -o canmount=on -o mountpoint=/home rpool/alpine/home
+   manage mountpoints with ``mountpoint=legacy``::
-     zfs create -o canmount=off -o mountpoint=/var  rpool/alpine/var
+
-     zfs create -o canmount=on  rpool/alpine/var/lib
+      zfs create -o mountpoint=legacy     rpool/alpinelinux/root
-     zfs create -o canmount=on  rpool/alpine/var/log
+      mount -t zfs rpool/alpinelinux/root /mnt/
-     zfs create -o canmount=off  -o mountpoint=none bpool/alpine
+      zfs create -o mountpoint=legacy rpool/alpinelinux/home
-     zfs create -o canmount=on  -o mountpoint=/boot bpool/alpine/root
+      mkdir /mnt/home
+      mount -t zfs  rpool/alpinelinux/home /mnt/home
+      mkdir -p /mnt/var/lib
+      mkdir -p /mnt/var/log
+      zfs create -o mountpoint=legacy  rpool/alpinelinux/var
+      zfs create -o mountpoint=legacy rpool/alpinelinux/var/lib
+      zfs create -o mountpoint=legacy rpool/alpinelinux/var/log
+      zfs create -o mountpoint=none bpool/alpinelinux
+      zfs create -o mountpoint=legacy bpool/alpinelinux/root
+      mkdir /mnt/boot
+      mount -t zfs bpool/alpinelinux/root /mnt/boot
+
+#. mkinitfs requires root dataset to have a mountpoint
+   other than legacy::
+
+      umount -Rl /mnt
+      zfs set canmount=noauto  rpool/alpinelinux/root
+      zfs set mountpoint=/     rpool/alpinelinux/root
+      mount -t zfs -o zfsutil  rpool/alpinelinux/root /mnt
+      mount -t zfs  rpool/alpinelinux/home /mnt/home
+      mount -t zfs bpool/alpinelinux/root /mnt/boot
+      mount -t zfs rpool/alpinelinux/var/lib /mnt/var/lib
+      mount -t zfs rpool/alpinelinux/var/log /mnt/var/log
 
 #. Format and mount ESP::
 
@@ -152,6 +190,10 @@ System Installation
 
    GRUB installation will fail and will be reinstalled later.
 
+#. Allow EFI system partition to fail at boot::
+
+    sed -i "s|vfat.*rw|vfat rw,nofail|" /mnt/etc/fstab
+
 #. Chroot::
 
     m='/dev /proc /sys'
@@ -161,36 +203,12 @@ System Installation
 
 #. Rebuild initrd::
 
-    mkdir -p /etc/zfs
-    rm -f /etc/zfs/zpool.cache
-    touch /etc/zfs/zpool.cache
-    chmod a-w /etc/zfs/zpool.cache
-    chattr +i /etc/zfs/zpool.cache
-
     sed -i 's|zfs|nvme zfs|' /etc/mkinitfs/mkinitfs.conf
     for directory in /lib/modules/*; do
       kernel_version=$(basename $directory)
       mkinitfs $kernel_version
     done
 
-#. Enable dataset mounting at boot::
-
-     rc-update add zfs-mount sysinit
-
-#. Replace predictable disk path with traditional disk path:
-
-   For SATA drives::
-
-     sed -i 's|/dev/disk/by-id/ata-.*-part|/dev/sda|' /etc/fstab
-
-   For NVMe drives::
-
-     sed -i 's|/dev/disk/by-id/nvme-.*-part|/dev/nvme0n1p|' /etc/fstab
-
-#. Mount datasets with zfsutil option::
-
-     sed -i 's|,posixacl|,zfsutil,posixacl|' /etc/fstab
-
 #. Apply GRUB workaround::
 
      echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile.d/zpool_vdev_name_path.sh
@@ -210,7 +228,6 @@ System Installation
 
 #. Install GRUB::
 
-      export ZPOOL_VDEV_NAME_PATH=YES
       mkdir -p /boot/efi/alpine/grub-bootdir/i386-pc/
       mkdir -p /boot/efi/alpine/grub-bootdir/x86_64-efi/
       for i in ${DISK}; do
@@ -243,15 +260,15 @@ System Installation
 
 #. Reboot::
 
-     poweroff
+     reboot
 
-   Disconnect the live media and other non-boot storage devices.
+Post installaion
-   Due to missing support of predictable device names in initrd,
+~~~~~~~~~~~~~~~~
-   Alpine Linux will mount whichever disk appears to be /dev/sda or /dev/nvme0
-   at /boot/efi at boot.
 
-   Root filesystem at / and /boot are ZFS and imported via pool name thus not affected by the above restriction.
+#. Setup graphical desktop::
 
-#. Post-install:
+     setup-desktop
 
-   #. Setup swap.
+#. You can create a snapshot of the newly installed
+   system for later rollback,
+   see `this page <https://openzfs.github.io/openzfs-docs/Getting%20Started/Arch%20Linux/Root%20on%20ZFS/6-create-boot-environment.html>`__.

docs/Getting Started/Arch Linux/Root on ZFS/1-preparation.rst

@@ -26,9 +26,9 @@ Preparation
 
    List available disks with::
 
-    ls /dev/disk/by-id/*
+    find /dev/disk/by-id/
 
-   If using virtio as disk bus, use ``/dev/disk/by-path/*``.
+   If using virtio as disk bus, use ``/dev/disk/by-path/``.
 
    Declare disk array::
 
@@ -40,11 +40,10 @@ Preparation
 
 #. Set partition size:
 
-   Set swap size. It's `recommended <https://chrisdown.name/2018/01/02/in-defence-of-swap.html>`__
+   Set swap size, set to 1 if you don't want swap to
-   to setup a swap partition. If you intend to use hibernation,
+   take up too much space::
-   the minimum should be no less than RAM size. Skip if swap is not needed::
 
-    INST_PARTSIZE_SWAP=8
+    INST_PARTSIZE_SWAP=4
 
    Root pool size, use all remaining disk space if not set::
 
@@ -54,7 +53,7 @@ Preparation
 
      curl -L https://archzfs.com/archzfs.gpg |  pacman-key -a -
      pacman-key --lsign-key $(curl -L https://git.io/JsfVS)
-     curl -L https://git.io/Jsfw2 > /etc/pacman.d/mirrorlist-archzfs
+     curl -L https://raw.githubusercontent.com/openzfs/openzfs-docs/master/docs/Getting%20Started/Arch%20Linux/archzfs-repo/mirrorlist-archzfs > /etc/pacman.d/mirrorlist-archzfs
 
      tee -a /etc/pacman.conf <<- 'EOF'
 
@@ -77,12 +76,26 @@ Preparation
    * https://archzfs.com/archive_archzfs/
    * https://archzfs.com/archzfs/x86_64/
 
+   ::
+      curl -L https://archzfs.com/archive_archzfs/ \
+      | grep zfs-linux-[0-9] \
+      | grep -v src.tar \
+      | grep "5.18.7"
+      # ...<a href="zfs-linux-2.1.5_5.18.7.arch1.1-1-x86_64.pkg.tar.zst">...
+
    Result: https/.../archive_archzfs/zfs-linux-2.1.5_5.18.7.arch1.1-1-x86_64.pkg.tar.zst
 
 #. Find compatible zfs-utils package:
 
    Search ZFS version string (e.g. 2.1.5) in both pages above.
 
+   ::
+      curl -L https://archzfs.com/archzfs/x86_64/ \
+      | grep zfs-utils-2.1.5 \
+      | grep -v src.tar
+      # ...<a href="zfs-utils-2.1.5-1-x86_64.pkg.tar.zst">...
+
+
    Result: https/.../archzfs/x86_64/zfs-utils-2.1.5-2-x86_64.pkg.tar.zst
 
 #. Download both then install::

docs/Getting Started/Arch Linux/Root on ZFS/2-system-installation.rst

@@ -10,13 +10,18 @@ System Installation
 
      for i in ${DISK}; do
 
+     # wipe flash-based storage device to improve
+     # performance.
+     # ALL DATA WILL BE LOST
+     # blkdiscard -f $i
+
      sgdisk --zap-all $i
 
      sgdisk -n1:1M:+1G -t1:EF00 $i
 
      sgdisk -n2:0:+4G -t2:BE00 $i
 
-     test -z $INST_PARTSIZE_SWAP || sgdisk -n4:0:+${INST_PARTSIZE_SWAP}G -t4:8200 $i
+     sgdisk -n4:0:+${INST_PARTSIZE_SWAP}G -t4:8200 $i
 
      if test -z $INST_PARTSIZE_RPOOL; then
          sgdisk -n3:0:0   -t3:BF00 $i
@@ -25,6 +30,12 @@ System Installation
      fi
 
      sgdisk -a1 -n5:24K:+1000K -t5:EF02 $i
+
+     sync && udevadm settle && sleep 3 
+
+     cryptsetup open --type plain --key-file /dev/random $i-part4 ${i##*/}-part4
+     mkswap /dev/mapper/${i##*/}-part4
+     swapon /dev/mapper/${i##*/}-part4 
      done
 
 #. Create boot pool::
@@ -84,9 +95,7 @@ System Installation
 
    If not using a multi-disk setup, remove ``mirror``.
 
-#. This section implements dataset layout as described in `overview <1-preparation.html>`__.
+#. Create root system container:
-
-   Create root system container:
 
    - Unencrypted::
 
@@ -108,18 +117,34 @@ System Installation
        -o keyformat=passphrase \
        rpool/archlinux
 
-   Create system datasets::
+   You can automate this step (insecure) with: ``echo POOLPASS | zfs create ...``.
 
-      zfs create -o canmount=on -o mountpoint=/     rpool/archlinux/root
+   Create system datasets, let Archlinux declaratively
-      zfs create -o canmount=on -o mountpoint=/home rpool/archlinux/home
+   manage mountpoints with ``mountpoint=legacy``::
-      zfs create -o canmount=off -o mountpoint=/var  rpool/archlinux/var
-      zfs create -o canmount=on  rpool/archlinux/var/lib
-      zfs create -o canmount=on  rpool/archlinux/var/log
 
-   Create boot dataset::
+      zfs create -o mountpoint=legacy     rpool/archlinux/root
+      mount -t zfs rpool/archlinux/root /mnt/
+      zfs create -o mountpoint=legacy rpool/archlinux/home
+      mkdir /mnt/home
+      mount -t zfs  rpool/archlinux/home /mnt/home
+      zfs create -o mountpoint=legacy  rpool/archlinux/var
+      zfs create -o mountpoint=legacy rpool/archlinux/var/lib
+      zfs create -o mountpoint=legacy rpool/archlinux/var/log
+      zfs create -o mountpoint=none bpool/archlinux
+      zfs create -o mountpoint=legacy bpool/archlinux/root
+      mkdir /mnt/boot
+      mount -t zfs bpool/archlinux/root /mnt/boot
+
+#. zfs-mkinitcpio requires root dataset to have a mountpoint
+   other than legacy::
+
+      umount -Rl /mnt
+      zfs set canmount=noauto  rpool/archlinux/root
+      zfs set mountpoint=/     rpool/archlinux/root
+      mount -t zfs -o zfsutil rpool/archlinux/root /mnt
+      mount -t zfs  rpool/archlinux/home /mnt/home
+      mount -t zfs bpool/archlinux/root /mnt/boot
 
-     zfs create -o canmount=off -o mountpoint=none bpool/archlinux
-     zfs create -o canmount=on -o mountpoint=/boot bpool/archlinux/root
 
 #. Format and mount ESP::
 
@@ -131,14 +156,3 @@ System Installation
 
     mkdir -p /mnt/boot/efi
     mount -t vfat $(echo $DISK | cut -f1 -d\ )-part1 /mnt/boot/efi
-
-#. Install packages::
-
-     pacstrap /mnt base vi mandoc grub efibootmgr mkinitcpio
-
-     CompatibleVer=$(pacman -Si zfs-linux | grep 'Depends On' | sed "s|.*linux=||" | awk '{ print $1 }')
-     pacstrap -U /mnt https://archive.archlinux.org/packages/l/linux/linux-${CompatibleVer}-x86_64.pkg.tar.zst
-
-     pacstrap /mnt zfs-linux zfs-utils
-
-     pacstrap /mnt linux-firmware intel-ucode amd-ucode

docs/Getting Started/Arch Linux/Root on ZFS/3-system-configuration.rst

@@ -8,13 +8,28 @@ System Configuration
 
 #. Generate fstab::
 
+    mkdir -p /mnt/var/log
+    mkdir -p /mnt/var/lib
+    mount -t zfs rpool/archlinux/var/lib /mnt/var/lib
+    mount -t zfs rpool/archlinux/var/log /mnt/var/log
     mkdir -p /mnt/etc/
-    for i in ${DISK}; do
+    genfstab -t PARTUUID /mnt | grep -v swap > /mnt/etc/fstab
-       echo UUID=$(blkid -s UUID -o value ${i}-part1) /boot/efis/${i##*/}-part1 vfat \
+    sed -i "s|vfat.*rw|vfat rw,x-systemd.idle-timeout=1min,x-systemd.automount,noauto,nofail|" /mnt/etc/fstab
-       umask=0022,fmask=0022,dmask=0022 0 1 >> /mnt/etc/fstab
+
-    done
+#. Install packages::
-    echo $(echo $DISK | cut -f1 -d\ )-part1 /boot/efi vfat \
+
-       noauto,umask=0022,fmask=0022,dmask=0022 0 1 >> /mnt/etc/fstab
+     pacstrap /mnt base mg mandoc grub efibootmgr mkinitcpio
+
+     CompatibleVer=$(pacman -Si zfs-linux \
+     | grep 'Depends On' \
+     | sed "s|.*linux=||" \
+     | awk '{ print $1 }')
+
+     pacstrap -U /mnt https://archive.archlinux.org/packages/l/linux/linux-${CompatibleVer}-x86_64.pkg.tar.zst
+
+     pacstrap /mnt zfs-linux zfs-utils
+
+     pacstrap /mnt linux-firmware intel-ucode amd-ucode
 
 #. Configure mkinitcpio::
 
@@ -28,19 +43,10 @@ System Configuration
      hwclock --systohc
      systemctl enable systemd-timesyncd --root=/mnt
 
-#. Set locale, keymap, timezone, hostname and root password::
-
-    rm -f /mnt/etc/localtime
-    systemd-firstboot --root=/mnt --prompt --force
-
 #. Generate host id::
 
     zgenhostid -f -o /mnt/etc/hostid
 
-#. Enable ZFS services::
-
-    systemctl enable zfs-import-scan.service zfs-mount zfs-import.target zfs-zed zfs.target --root=/mnt
-
 #. Add archzfs repo::
 
      curl -L https://archzfs.com/archzfs.gpg |  pacman-key -a - --gpgdir /mnt/etc/pacman.d/gnupg
@@ -66,3 +72,16 @@ System Configuration
 
     echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen
     locale-gen
+
+#. Generate initrd::
+
+    mkinitcpio -P
+
+#. Import from by-id::
+
+     echo GRUB_CMDLINE_LINUX=\"zfs_import_dir=/dev/disk/by-id/\" >> /etc/default/grub
+
+#. Set locale, keymap, timezone, hostname and root password::
+
+    rm -f /etc/localtime
+    systemd-firstboot --prompt --force

docs/Getting Started/Arch Linux/Root on ZFS/5-bootloader.rst

@@ -6,15 +6,6 @@ Bootloader
 .. contents:: Table of Contents
    :local:
 
-#. Create empty cache file and generate initrd::
-
-    rm -f /etc/zfs/zpool.cache
-    touch /etc/zfs/zpool.cache
-    chmod a-w /etc/zfs/zpool.cache
-    chattr +i /etc/zfs/zpool.cache
-
-    mkinitcpio -P
-
 #. Apply GRUB workaround::
 
      echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile.d/zpool_vdev_name_path.sh
@@ -23,14 +14,11 @@ Bootloader
      # GRUB fails to detect rpool name, hard code as "rpool"
      sed -i "s|rpool=.*|rpool=rpool|"  /etc/grub.d/10_linux
 
-     echo GRUB_CMDLINE_LINUX=\"zfs_import_dir=/dev/disk/by-id/\" >> /etc/default/grub
-
    This workaround needs to be applied for every GRUB update, as the
    update will overwrite the changes.
 
 #. Install GRUB::
 
-      export ZPOOL_VDEV_NAME_PATH=YES
       mkdir -p /boot/efi/arch/grub-bootdir/i386-pc/
       mkdir -p /boot/efi/arch/grub-bootdir/x86_64-efi/
       for i in ${DISK}; do
@@ -40,9 +28,6 @@ Bootloader
       grub-install --target x86_64-efi --boot-directory \
           /boot/efi/arch/grub-bootdir/x86_64-efi/ --efi-directory \
 	  /boot/efi --bootloader-id arch --removable
-
-#. Generate GRUB menu::
-
       grub-mkconfig -o /boot/efi/arch/grub-bootdir/x86_64-efi/grub/grub.cfg
       grub-mkconfig -o /boot/efi/arch/grub-bootdir/i386-pc/grub/grub.cfg
 
@@ -70,3 +55,7 @@ Finish Installation
 #. Reboot::
 
     reboot
+
+#. You can create a snapshot of the newly installed
+   system for later rollback,
+   see `this page <https://openzfs.github.io/openzfs-docs/Getting%20Started/Arch%20Linux/Root%20on%20ZFS/6-create-boot-environment.html>`__.

docs/Getting Started/Arch Linux/Root on ZFS/6-create-boot-environment.rst

@@ -0,0 +1,172 @@
+.. highlight:: sh
+
+Create a Boot Environment
+==========================
+
+This page is tested for Alpine, Arch Linux, Fedora and
+RHEL guides.  Not useful for NixOS since system rollback
+is already managed by Nix.
+
+With Root on ZFS, it is possible to take snapshots of
+existing root file systems, which is a read-only copy of
+that file system.  A new, full-fledged file system --
+clones -- can be then created from this snapshot.  This
+bootable clone of the current system is then called a
+"Boot Environment".
+
+This could be useful if you are performing a major
+system upgrade and wish to have the option to go back to
+a previous state if the upgrade fails.
+
+#. Identify which dataset is currently mounted as root
+   ``/`` and boot ``/boot``::
+
+     findmnt /
+     # output
+     TARGET SOURCE               FSTYPE OPTIONS
+     /      rpool/archlinux/root zfs    rw,relatime,xattr,posixacl
+
+     findmnt /boot
+     # output
+     TARGET SOURCE               FSTYPE OPTIONS
+     /boot  bpool/archlinux/root zfs    rw,relatime,xattr,posixacl
+
+#. Identify your distribution in the dataset (file system) path::
+
+     DISTRO=archlinux # or `fedora', `alma', `alpinelinux'
+
+#. Choose a name for the new boot environment::
+
+     BE_NAME=backup
+
+#. Take snapshots of the ``/`` and ``/boot`` datasets::
+
+     zfs snapshot rpool/$DISTRO/root@$BE_NAME
+     zfs snapshot bpool/$DISTRO/root@$BE_NAME
+
+#. Create clones from read-only snapshots::
+
+     zfs clone -o canmount=noauto \
+       -o mountpoint=/ \
+       rpool/$DISTRO/root@$BE_NAME \
+       rpool/$DISTRO/$BE_NAME
+
+     zfs clone -o canmount=noauto \
+       -o mountpoint=legacy \
+       bpool/$DISTRO/root@$BE_NAME \
+       bpool/$DISTRO/$BE_NAME
+
+#. Mount clone and update file system table (fstab) ::
+
+     mount -t zfs -o zfsutil rpool/$DISTRO/$BE_NAME /mnt
+     mount -t zfs  bpool/$DISTRO/$BE_NAME /mnt/boot
+
+     sed -i "s|rpool/$DISTRO/root|rpool/$DISTRO/$BE_NAME|g" /mnt/etc/fstab
+     sed -i "s|bpool/$DISTRO/root|bpool/$DISTRO/$BE_NAME|g" /mnt/etc/fstab
+
+   If legacy mountpoint is used, omit ``-o zfsutil``
+   from mount command.
+
+#. Create GRUB menu for new clone::
+
+     m='/dev /proc /sys'
+     for i in $m; do mount --rbind $i /mnt/$i; done
+     chroot /mnt /usr/bin/env sh <<EOF
+       if which grub-mkconfig; then
+         grub-mkconfig -o /boot/grub.cfg
+       else
+         grub2-mkconfig -o /boot/grub.cfg
+       fi
+     EOF
+
+   GRUB menu contains information on kernel version and initramfs.
+
+#. Unmount clone::
+
+     umount -Rl /mnt
+
+#. Add new boot environment as GRUB menu entry::
+
+     tee -a new_entry <<EOF
+     menuentry 'ZFS Clone of ${DISTRO}: ${BE_NAME}' { configfile (hd0,gpt2)/${DISTRO}/${BE_NAME}@/grub.cfg }
+     EOF
+
+     find /boot/efis/ -name "grub.cfg" \
+     | while read i; do
+         if grep -q "${DISTRO}" $i; then
+           cat $i new_entry > grub.cfg
+	   cp grub.cfg $i
+	 fi
+       done
+
+     rm new_entry
+
+#. After reboot, select boot environment entry from GRUB
+   menu to boot from the clone.  Press ESC inside
+   submenu to return to the previous menu.
+
+#. Steps above can also be used to create a new clone
+   from an existing snapshot.
+
+#. To set a boot environment as default, replace
+   existing ``grub.cfg`` inside EFI system partition
+   with the one from the boot environment::
+
+     mount -t zfs  bpool/$DISTRO/$BE_NAME /mnt
+
+     # backup existing grub.cfg inside EFI
+     # then replace it with menu from clone
+
+     mkdir -p /mnt/grub_menu_backup
+     menu_counter=1
+     find /boot/efis/ -name "grub.cfg" \
+     | while read i; do
+         if grep -q "${DISTRO}" $i; then
+	   cp $i /mnt/grub_menu_backup/grub_${menu_counter}.cfg
+           echo $i > /mnt/grub_menu_backup/grub_${menu_counter}_path.txt
+	   cp /mnt/grub.cfg $i
+	   menu_counter=$(($menu_counter + 1))
+	 fi
+       done
+
+     umount -Rl /mnt
+
+#. To delete the boot environment, check with
+   ``findmnt`` that the boot environment is not
+   currently used::
+
+     findmnt /
+     findmnt /boot
+
+   Set variables::
+
+     DISTRO=archlinux
+
+   Then check the origin snapshot::
+
+     zfs get origin rpool/archlinux/backup
+     # rpool/archlinux/root@backup
+     zfs get origin bpool/archlinux/backup
+     # bpool/archlinux/root@backup
+
+     RM_BE=backup
+     RM_SNAPSHOT=root@backup
+
+   Finally, destroy clone (boot environment) and its
+   origin snapshot::
+
+     zfs destroy rpool/${DISTRO}/${RM_BE}
+     zfs destroy rpool/${DISTRO}/${RM_SNAPSHOT}
+
+     zfs destroy bpool/${DISTRO}/${RM_BE}
+     zfs destroy bpool/${DISTRO}/${RM_SNAPSHOT}
+
+   Remove GRUB entry::
+
+     find /boot/efis/ -name "grub.cfg" \
+     | while read i; do
+         if grep -q "${DISTRO}/${RM_BE}@/grub.cfg" $i; then
+             head -n -1 $i > grub.cfg
+	     cp grub.cfg $i
+	 fi
+       done

docs/Getting Started/Fedora/Root on ZFS/1-preparation.rst

@@ -6,9 +6,12 @@ Preparation
 .. contents:: Table of Contents
    :local:
 
-#. Disable Secure Boot. ZFS modules can not be loaded if Secure Boot is enabled.
+#. Disable Secure Boot. ZFS modules can not be loaded if
-#. Download a variant of `Fedora live image
+   Secure Boot is enabled.
-   <https://download.fedoraproject.org/pub/fedora/linux/releases/>`__ and boot from it.
+#. Download live Fedora media, such as this `LXQt spin
+   <https://spins.fedoraproject.org/lxqt/download/index.html>`__.
+   The installed system is the same regardless of live
+   media used.
 #. Connect to the Internet.
 #. Set root password or ``/root/.ssh/authorized_keys``.
 #. Start SSH server::
@@ -24,9 +27,9 @@ Preparation
 
    List available disks with::
 
-    ls /dev/disk/by-id/*
+    find /dev/disk/by-id/
 
-   If using virtio as disk bus, use ``/dev/disk/by-path/*``.
+   If using virtio as disk bus, use ``/dev/disk/by-path/``.
 
    Declare disk array::
 
@@ -38,11 +41,10 @@ Preparation
 
 #. Set partition size:
 
-   Set swap size. It's `recommended <https://chrisdown.name/2018/01/02/in-defence-of-swap.html>`__
+   Set swap size, set to 1 if you don't want swap to
-   to setup a swap partition. If you intend to use hibernation,
+   take up too much space::
-   the minimum should be no less than RAM size. Skip if swap is not needed::
 
-    INST_PARTSIZE_SWAP=8
+    INST_PARTSIZE_SWAP=4
 
    Root pool size, use all remaining disk space if not set::
 
@@ -54,24 +56,16 @@ Preparation
 
    SELinux will be enabled on the installed system.
 
-#. Add ZFS repo::
+#. Add ZFS repo and install ZFS inside live system::
 
     dnf install -y https://zfsonlinux.org/fedora/zfs-release-2-2$(rpm --eval "%{dist}").noarch.rpm
-
+    rpm -e --nodeps zfs-fuse || true
-#. Check available repos::
+    source /etc/os-release
-
+    export VERSION_ID
-     dnf repolist --all
+    dnf install -y https://dl.fedoraproject.org/pub/fedora/linux/releases/${VERSION_ID}/Everything/x86_64/os/Packages/k/kernel-devel-$(uname -r).rpm
-
-#. Install ZFS packages::
-
-    rpm -e --nodeps zfs-fuse
-    dnf install -y https://dl.fedoraproject.org/pub/fedora/linux/releases/$(source /etc/os-release; echo $VERSION_ID)/Everything/x86_64/os/Packages/k/kernel-devel-$(uname -r).rpm
     dnf install -y zfs
-
-#. Load kernel modules::
-
     modprobe zfs
 
-#. Install partition tool::
+#. Install partition tool and arch-install-scripts::
 
-    dnf install -y gdisk dosfstools
+    dnf install -y gdisk dosfstools arch-install-scripts

docs/Getting Started/Fedora/Root on ZFS/2-system-installation.rst

@@ -10,13 +10,18 @@ System Installation
 
      for i in ${DISK}; do
 
+     # wipe flash-based storage device to improve
+     # performance.
+     # ALL DATA WILL BE LOST
+     # blkdiscard -f $i
+
      sgdisk --zap-all $i
 
      sgdisk -n1:1M:+1G -t1:EF00 $i
 
      sgdisk -n2:0:+4G -t2:BE00 $i
 
-     test -z $INST_PARTSIZE_SWAP || sgdisk -n4:0:+${INST_PARTSIZE_SWAP}G -t4:8200 $i
+     sgdisk -n4:0:+${INST_PARTSIZE_SWAP}G -t4:8200 $i
 
      if test -z $INST_PARTSIZE_RPOOL; then
          sgdisk -n3:0:0   -t3:BF00 $i
@@ -25,6 +30,12 @@ System Installation
      fi
 
      sgdisk -a1 -n5:24K:+1000K -t5:EF02 $i
+
+     sync && udevadm settle && sleep 3 
+
+     cryptsetup open --type plain --key-file /dev/random $i-part4 ${i##*/}-part4
+     mkswap /dev/mapper/${i##*/}-part4
+     swapon /dev/mapper/${i##*/}-part4 
      done
 
 #. Create boot pool::
@@ -84,16 +95,14 @@ System Installation
 
    If not using a multi-disk setup, remove ``mirror``.
 
-#. This section implements dataset layout as described in `overview <1-preparation.html>`__.
+#. Create root system container:
-
-   Create root system container:
 
    - Unencrypted::
 
       zfs create \
        -o canmount=off \
        -o mountpoint=none \
-       rpool/redhat
+       rpool/fedora
 
    - Encrypted:
 
@@ -106,20 +115,36 @@ System Installation
        -o encryption=on \
        -o keylocation=prompt \
        -o keyformat=passphrase \
-       rpool/redhat
+       rpool/fedora
 
-   Create system datasets::
+   You can automate this step (insecure) with: ``echo POOLPASS | zfs create ...``.
 
-      zfs create -o canmount=on -o mountpoint=/     rpool/redhat/root
+   Create system datasets, let Fedora declaratively
-      zfs create -o canmount=on -o mountpoint=/home rpool/redhat/home
+   manage mountpoints with ``mountpoint=legacy``::
-      zfs create -o canmount=off -o mountpoint=/var  rpool/redhat/var
-      zfs create -o canmount=on  rpool/redhat/var/lib
-      zfs create -o canmount=on  rpool/redhat/var/log
 
-   Create boot dataset::
+      zfs create -o mountpoint=legacy     rpool/fedora/root
+      mount -t zfs rpool/fedora/root /mnt/
+      zfs create -o mountpoint=legacy rpool/fedora/home
+      mkdir /mnt/home
+      mount -t zfs  rpool/fedora/home /mnt/home
+      zfs create -o mountpoint=legacy  rpool/fedora/var
+      zfs create -o mountpoint=legacy rpool/fedora/var/lib
+      zfs create -o mountpoint=legacy rpool/fedora/var/log
+      zfs create -o mountpoint=none bpool/fedora
+      zfs create -o mountpoint=legacy bpool/fedora/root
+      mkdir /mnt/boot
+      mount -t zfs bpool/fedora/root /mnt/boot
+
+#. zfs-dracut requires root dataset to have a mountpoint
+   other than legacy::
+
+      umount -Rl /mnt
+      zfs set canmount=noauto  rpool/fedora/root
+      zfs set mountpoint=/     rpool/fedora/root
+      mount -t zfs -o zfsutil rpool/fedora/root /mnt
+      mount -t zfs  rpool/fedora/home /mnt/home
+      mount -t zfs bpool/fedora/root /mnt/boot
 
-     zfs create -o canmount=off -o mountpoint=none bpool/redhat
-     zfs create -o canmount=on -o mountpoint=/boot bpool/redhat/root
 
 #. Format and mount ESP::
 
@@ -131,14 +156,3 @@ System Installation
 
     mkdir -p /mnt/boot/efi
     mount -t vfat $(echo $DISK | cut -f1 -d\ )-part1 /mnt/boot/efi
-
-#. Install packages::
-
-    dnf --installroot=/mnt   --releasever=$(source /etc/os-release ; echo $VERSION_ID) -y install \
-    @core  grub2-efi-x64 grub2-pc-modules grub2-efi-x64-modules shim-x64 efibootmgr kernel \
-    kernel-devel
-
-    dnf --installroot=/mnt   --releasever=$(source /etc/os-release ; echo $VERSION_ID) -y install \
-    https://zfsonlinux.org/fedora/zfs-release-2-2$(rpm --eval "%{dist}").noarch.rpm
-
-    dnf --installroot=/mnt   --releasever=$(source /etc/os-release ; echo $VERSION_ID) -y install zfs zfs-dracut

docs/Getting Started/Fedora/Root on ZFS/3-system-configuration.rst

@@ -6,30 +6,42 @@ System Configuration
 .. contents:: Table of Contents
    :local:
 
+
 #. Generate fstab::
 
+    mkdir -p /mnt/var/log
+    mkdir -p /mnt/var/lib
+    mount -t zfs rpool/fedora/var/lib /mnt/var/lib
+    mount -t zfs rpool/fedora/var/log /mnt/var/log
     mkdir -p /mnt/etc/
-    for i in ${DISK}; do
+    genfstab -t PARTUUID /mnt | grep -v swap > /mnt/etc/fstab
-       echo UUID=$(blkid -s UUID -o value ${i}-part1) /boot/efis/${i##*/}-part1 vfat \
+    sed -i "s|vfat.*rw|vfat rw,x-systemd.idle-timeout=1min,x-systemd.automount,noauto,nofail|" /mnt/etc/fstab
-       umask=0022,fmask=0022,dmask=0022 0 1 >> /mnt/etc/fstab
+
-    done
+#. Install basic system packages::
-    echo $(echo $DISK | cut -f1 -d\ )-part1 /boot/efi vfat \
+
-       noauto,umask=0022,fmask=0022,dmask=0022 0 1 >> /mnt/etc/fstab
+    dnf --installroot=/mnt \
+    --releasever=$VERSION_ID -y install \
+    @core  grub2-efi-x64 \
+    grub2-pc-modules grub2-efi-x64-modules \
+    shim-x64  efibootmgr \
+    kernel kernel-devel
+
+    dnf --installroot=/mnt \
+    --releasever=$VERSION_ID -y install \
+    https://zfsonlinux.org/fedora/zfs-release-2-2$(rpm --eval "%{dist}").noarch.rpm
+
+    dnf --installroot=/mnt --releasever=$VERSION_ID \
+    -y install zfs zfs-dracut
 
 #. Configure dracut::
 
     echo 'add_dracutmodules+=" zfs "' > /mnt/etc/dracut.conf.d/zfs.conf
-
-#. Force load mpt3sas module if used::
-
     if grep mpt3sas /proc/modules; then
       echo 'forced_drivers+=" mpt3sas "'  >> /mnt/etc/dracut.conf.d/zfs.conf
     fi
-
+    if grep virtio_blk /proc/modules; then
-#. Set locale, keymap, timezone, hostname and root password::
+      echo 'filesystems+=" virtio_blk "' >> /mnt/etc/dracut.conf.d/fs.conf
-
+    fi
-    rm -f /mnt/etc/localtime
-    systemd-firstboot --root=/mnt --prompt --root-password=PASSWORD --force
 
 #. Generate host id::
 
@@ -39,10 +51,6 @@ System Configuration
 
     dnf --installroot=/mnt install -y glibc-minimal-langpack glibc-langpack-en
 
-#. Enable ZFS services::
-
-    systemctl enable zfs-import-scan.service zfs-mount zfs-import.target zfs-zed zfs.target --root=/mnt
-
 #. By default SSH server is enabled, allowing root login by password,
    disable SSH server::
 
@@ -51,23 +59,33 @@ System Configuration
 
 #. Chroot::
 
-    m='/dev /proc /sys'
-    for i in $m; do mount --rbind $i /mnt/$i; done
-
      history -w /mnt/home/sys-install-pre-chroot.txt
-    chroot /mnt /usr/bin/env DISK="$DISK" bash --login
+     arch-chroot /mnt /usr/bin/env DISK="$DISK" bash --login
 
 #. For SELinux, relabel filesystem on reboot::
 
     fixfiles -F onboot
 
-#. Set root password, the password set earlier does not work due to SELinux::
-
-    passwd
-
 #. Build ZFS modules::
 
     for directory in /lib/modules/*; do
       kernel_version=$(basename $directory)
       dkms autoinstall -k $kernel_version
     done
+
+
+#. Generate initrd::
+
+    for directory in /lib/modules/*; do
+      kernel_version=$(basename $directory)
+      dracut --force --kver $kernel_version
+    done
+
+#. Set locale, keymap, timezone, hostname and root password::
+
+    rm -f /etc/localtime
+    systemd-firstboot --prompt --root-password=PASSWORD --force
+
+#. Set root password, the password set earlier does not work due to SELinux::
+
+    passwd

docs/Getting Started/Fedora/Root on ZFS/5-bootloader.rst

@@ -6,27 +6,6 @@ Bootloader
 .. contents:: Table of Contents
    :local:
 
-
-#. If using virtio disk, add driver to initrd::
-
-    echo 'filesystems+=" virtio_blk "' >> /etc/dracut.conf.d/fs.conf
-
-#. Create empty cache file and generate initrd::
-
-    rm -f /etc/zfs/zpool.cache
-    touch /etc/zfs/zpool.cache
-    chmod a-w /etc/zfs/zpool.cache
-    chattr +i /etc/zfs/zpool.cache
-
-    for directory in /lib/modules/*; do
-      kernel_version=$(basename $directory)
-      dracut --force --kver $kernel_version
-    done
-
-#. Load ZFS modules and disable BLS::
-
-    echo 'GRUB_ENABLE_BLSCFG=false' >> /etc/default/grub
-
 #. Apply GRUB workaround::
 
      echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile.d/zpool_vdev_name_path.sh
@@ -35,41 +14,39 @@ Bootloader
      # GRUB fails to detect rpool name, hard code as "rpool"
      sed -i "s|rpool=.*|rpool=rpool|"  /etc/grub.d/10_linux
 
-   This workaround needs to be applied for every GRUB update, as the
+   This ``sed`` workaround needs to be applied for every
-   update will overwrite the changes.
+   GRUB update, as the update will overwrite the
+   changes.
 
 #. Install GRUB::
 
-      export ZPOOL_VDEV_NAME_PATH=YES
+      echo 'GRUB_ENABLE_BLSCFG=false' >> /etc/default/grub
       mkdir -p /boot/efi/fedora/grub-bootdir/i386-pc/
       mkdir -p /boot/efi/fedora/grub-bootdir/x86_64-efi/
       for i in ${DISK}; do
        grub2-install --target=i386-pc --boot-directory \
            /boot/efi/fedora/grub-bootdir/i386-pc/  $i
       done
-
       cp -r /usr/lib/grub/x86_64-efi/ /boot/efi/EFI/fedora/
-
-#. Generate GRUB menu::
-
       grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
       grub2-mkconfig -o /boot/efi/fedora/grub-bootdir/i386-pc/grub2/grub.cfg
 
 #. For both legacy and EFI booting: mirror ESP content::
 
-    ESP_MIRROR=$(mktemp -d)
     unalias -a
+    ESP_MIRROR=$(mktemp -d)
     cp -r /boot/efi/EFI $ESP_MIRROR
     for i in /boot/efis/*; do
      cp -r $ESP_MIRROR/EFI $i
     done
     rm -rf $ESP_MIRROR
 
-#. Notes for GRUB on RHEL
+#. Note: you need to regenerate GRUB menu after kernel
+   updates, otherwise computer will still boot old
+   kernel on reboot::
 
-   As bls is disabled, you will need to regenerate GRUB menu after each kernel upgrade.
+      grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
-   Or else the new kernel will not be recognized and system will boot the old kernel
+      grub2-mkconfig -o /boot/efi/fedora/grub-bootdir/i386-pc/grub2/grub.cfg
-   on reboot.
 
 Finish Installation
 ~~~~~~~~~~~~~~~~~~~~
@@ -87,11 +64,6 @@ Finish Installation
 
     reboot
 
-#. On first reboot, the boot process will fail, with failure messages such
-   as "You are in Emergency Mode...Press Ctrl-D to continue".
-
-   Wait for the computer to automatically reboot and the problem will be resolved.
-
 Post installaion
 ~~~~~~~~~~~~~~~~
 #. Install package groups::
@@ -100,3 +72,7 @@ Post installaion
     dnf group install gnome-desktop
 
 #. Add new user, configure swap.
+
+#. You can create a snapshot of the newly installed
+   system for later rollback,
+   see `this page <https://openzfs.github.io/openzfs-docs/Getting%20Started/Arch%20Linux/Root%20on%20ZFS/6-create-boot-environment.html>`__.

docs/Getting Started/RHEL-based distro/RHEL-based distro Root on ZFS/1-preparation.rst

@@ -8,7 +8,8 @@ Preparation
 
 #. Disable Secure Boot. ZFS modules can not be loaded if Secure Boot is enabled.
 #. Download a variant of `AlmaLinux Minimal Live ISO
-   <https://repo.almalinux.org/almalinux/9/live/x86_64/>`__ and boot from it.
+   <https://repo.almalinux.org/almalinux/9.1/live/x86_64/>`__
+   and boot from it.
 #. Connect to the Internet.
 #. Set root password or ``/root/.ssh/authorized_keys``.
 #. Start SSH server::
@@ -24,9 +25,9 @@ Preparation
 
    List available disks with::
 
-    ls /dev/disk/by-id/*
+    find /dev/disk/by-id/
 
-   If using virtio as disk bus, use ``/dev/disk/by-path/*``.
+   If using virtio as disk bus, use ``/dev/disk/by-path/``.
 
    Declare disk array::
 
@@ -38,11 +39,10 @@ Preparation
 
 #. Set partition size:
 
-   Set swap size. It's `recommended <https://chrisdown.name/2018/01/02/in-defence-of-swap.html>`__
+   Set swap size, set to 1 if you don't want swap to
-   to setup a swap partition. If you intend to use hibernation,
+   take up too much space::
-   the minimum should be no less than RAM size. Skip if swap is not needed::
 
-    INST_PARTSIZE_SWAP=8
+    INST_PARTSIZE_SWAP=4
 
    Root pool size, use all remaining disk space if not set::
 
@@ -54,24 +54,21 @@ Preparation
 
    SELinux will be enabled on the installed system.
 
-#. Add ZFS repo::
+#. Add ZFS repo and install ZFS inside live system::
 
     dnf install -y https://zfsonlinux.org/epel/zfs-release-2-2$(rpm --eval "%{dist}").noarch.rpm
-
+    rpm -e --nodeps zfs-fuse || true
-#. Check available repos::
+    source /etc/os-release
-
+    export VERSION_ID
-     dnf repolist --all
-
-#. Install ZFS packages::
-
     dnf config-manager --disable zfs
     dnf config-manager --enable zfs-kmod
     dnf install -y zfs
-
-#. Load kernel modules::
-
     modprobe zfs
 
-#. Install partition tool::
+#. Install partition tool and arch-install-scripts::
 
-    dnf install -y gdisk dosfstools
+    dnf install -y epel-release
+    dnf install -y gdisk dosfstools arch-install-scripts
+    dnf download arch-install-scripts
+    rpm -i --nodeps arch-install-scripts*.rpm
+    dnf remove -y epel-release

docs/Getting Started/RHEL-based distro/RHEL-based distro Root on ZFS/2-system-installation.rst

@@ -10,13 +10,18 @@ System Installation
 
      for i in ${DISK}; do
 
+     # wipe flash-based storage device to improve
+     # performance.
+     # ALL DATA WILL BE LOST
+     # blkdiscard -f $i
+
      sgdisk --zap-all $i
 
      sgdisk -n1:1M:+1G -t1:EF00 $i
 
      sgdisk -n2:0:+4G -t2:BE00 $i
 
-     test -z $INST_PARTSIZE_SWAP || sgdisk -n4:0:+${INST_PARTSIZE_SWAP}G -t4:8200 $i
+     sgdisk -n4:0:+${INST_PARTSIZE_SWAP}G -t4:8200 $i
 
      if test -z $INST_PARTSIZE_RPOOL; then
          sgdisk -n3:0:0   -t3:BF00 $i
@@ -25,6 +30,12 @@ System Installation
      fi
 
      sgdisk -a1 -n5:24K:+1000K -t5:EF02 $i
+
+     sync && udevadm settle && sleep 3 
+
+     cryptsetup open --type plain --key-file /dev/random $i-part4 ${i##*/}-part4
+     mkswap /dev/mapper/${i##*/}-part4
+     swapon /dev/mapper/${i##*/}-part4 
      done
 
 #. Create boot pool::
@@ -84,16 +95,14 @@ System Installation
 
    If not using a multi-disk setup, remove ``mirror``.
 
-#. This section implements dataset layout as described in `overview <1-preparation.html>`__.
+#. Create root system container:
-
-   Create root system container:
 
    - Unencrypted::
 
       zfs create \
        -o canmount=off \
        -o mountpoint=none \
-       rpool/redhat
+       rpool/alma
 
    - Encrypted:
 
@@ -106,20 +115,35 @@ System Installation
        -o encryption=on \
        -o keylocation=prompt \
        -o keyformat=passphrase \
-       rpool/redhat
+       rpool/alma
 
-   Create system datasets::
+   You can automate this step (insecure) with: ``echo POOLPASS | zfs create ...``.
 
-      zfs create -o canmount=on -o mountpoint=/     rpool/redhat/root
+   Create system datasets, let Alma declaratively
-      zfs create -o canmount=on -o mountpoint=/home rpool/redhat/home
+   manage mountpoints with ``mountpoint=legacy``::
-      zfs create -o canmount=off -o mountpoint=/var  rpool/redhat/var
-      zfs create -o canmount=on  rpool/redhat/var/lib
-      zfs create -o canmount=on  rpool/redhat/var/log
 
-   Create boot dataset::
+      zfs create -o mountpoint=legacy     rpool/alma/root
+      mount -t zfs rpool/alma/root /mnt/
+      zfs create -o mountpoint=legacy rpool/alma/home
+      mkdir /mnt/home
+      mount -t zfs -o zfsutil rpool/alma/home /mnt/home
+      zfs create -o mountpoint=legacy  rpool/alma/var
+      zfs create -o mountpoint=legacy rpool/alma/var/lib
+      zfs create -o mountpoint=legacy rpool/alma/var/log
+      zfs create -o mountpoint=none bpool/alma
+      zfs create -o mountpoint=legacy bpool/alma/root
+      mkdir /mnt/boot
+      mount -t zfs bpool/alma/root /mnt/boot
 
-     zfs create -o canmount=off -o mountpoint=none bpool/redhat
+#. zfs-dracut requires root dataset to have a mountpoint
-     zfs create -o canmount=on -o mountpoint=/boot bpool/redhat/root
+   other than legacy::
+
+      umount -Rl /mnt
+      zfs set canmount=noauto  rpool/alma/root
+      zfs set mountpoint=/     rpool/alma/root
+      mount -t zfs  rpool/alma/root /mnt
+      mount -t zfs  rpool/alma/home /mnt/home
+      mount -t zfs bpool/alma/root /mnt/boot
 
 #. Format and mount ESP::
 

docs/Getting Started/RHEL-based distro/RHEL-based distro Root on ZFS/3-system-configuration.rst

@@ -8,28 +8,39 @@ System Configuration
 
 #. Generate fstab::
 
+    mkdir -p /mnt/var/log
+    mkdir -p /mnt/var/lib
+    mount -t zfs rpool/alma/var/lib /mnt/var/lib
+    mount -t zfs rpool/alma/var/log /mnt/var/log
     mkdir -p /mnt/etc/
-    for i in ${DISK}; do
+    genfstab -t PARTUUID /mnt | grep -v swap > /mnt/etc/fstab
-       echo UUID=$(blkid -s UUID -o value ${i}-part1) /boot/efis/${i##*/}-part1 vfat \
+    sed -i "s|vfat.*rw|vfat rw,x-systemd.idle-timeout=1min,x-systemd.automount,noauto,nofail|" /mnt/etc/fstab
-       umask=0022,fmask=0022,dmask=0022 0 1 >> /mnt/etc/fstab
+
-    done
+#. Install basic system packages::
-    echo $(echo $DISK | cut -f1 -d\ )-part1 /boot/efi vfat \
+
-       noauto,umask=0022,fmask=0022,dmask=0022 0 1 >> /mnt/etc/fstab
+    dnf --installroot=/mnt \
+    --releasever=$VERSION_ID -y install \
+    @core  grub2-efi-x64 \
+    grub2-pc-modules grub2-efi-x64-modules \
+    shim-x64  efibootmgr \
+    kernel
+    dnf --installroot=/mnt \
+    --releasever=$VERSION_ID -y install \
+    https://zfsonlinux.org/epel/zfs-release-2-2$(rpm --eval "%{dist}").noarch.rpm
+    dnf config-manager --installroot=/mnt --disable zfs
+    dnf config-manager --installroot=/mnt --enable zfs-kmod
+    dnf --installroot=/mnt --releasever=$VERSION_ID \
+    -y install zfs zfs-dracut
 
 #. Configure dracut::
 
     echo 'add_dracutmodules+=" zfs "' > /mnt/etc/dracut.conf.d/zfs.conf
-
-#. Force load mpt3sas module if used::
-
     if grep mpt3sas /proc/modules; then
       echo 'forced_drivers+=" mpt3sas "'  >> /mnt/etc/dracut.conf.d/zfs.conf
     fi
-
+    if grep virtio_blk /proc/modules; then
-#. Set locale, keymap, timezone, hostname and root password::
+      echo 'filesystems+=" virtio_blk "' >> /mnt/etc/dracut.conf.d/fs.conf
-
+    fi
-    rm -f /mnt/etc/localtime
-    systemd-firstboot --root=/mnt --prompt --root-password=PASSWORD --force
 
 #. Generate host id::
 
@@ -39,10 +50,6 @@ System Configuration
 
     dnf --installroot=/mnt install -y glibc-minimal-langpack glibc-langpack-en
 
-#. Enable ZFS services::
-
-    systemctl enable zfs-import-scan.service zfs-mount zfs-import.target zfs-zed zfs.target --root=/mnt
-
 #. By default SSH server is enabled, allowing root login by password,
    disable SSH server::
 
@@ -51,16 +58,25 @@ System Configuration
 
 #. Chroot::
 
-    m='/dev /proc /sys'
-    for i in $m; do mount --rbind $i /mnt/$i; done
-
      history -w /mnt/home/sys-install-pre-chroot.txt
-    chroot /mnt /usr/bin/env DISK="$DISK" bash --login
+     arch-chroot /mnt /usr/bin/env DISK="$DISK" bash --login
 
 #. For SELinux, relabel filesystem on reboot::
 
     fixfiles -F onboot
 
+#. Generate initrd::
+
+    for directory in /lib/modules/*; do
+      kernel_version=$(basename $directory)
+      dracut --force --kver $kernel_version
+    done
+
+#. Set locale, keymap, timezone, hostname and root password::
+
+    rm -f /etc/localtime
+    systemd-firstboot --prompt --root-password=PASSWORD --force
+
 #. Set root password, the password set earlier does not work due to SELinux::
 
     passwd

docs/Getting Started/RHEL-based distro/RHEL-based distro Root on ZFS/5-bootloader.rst

@@ -6,27 +6,6 @@ Bootloader
 .. contents:: Table of Contents
    :local:
 
-
-#. If using virtio disk, add driver to initrd::
-
-    echo 'filesystems+=" virtio_blk "' >> /etc/dracut.conf.d/fs.conf
-
-#. Create empty cache file and generate initrd::
-
-    rm -f /etc/zfs/zpool.cache
-    touch /etc/zfs/zpool.cache
-    chmod a-w /etc/zfs/zpool.cache
-    chattr +i /etc/zfs/zpool.cache
-
-    for directory in /lib/modules/*; do
-      kernel_version=$(basename $directory)
-      dracut --force --kver $kernel_version
-    done
-
-#. Load ZFS modules and disable BLS::
-
-    echo 'GRUB_ENABLE_BLSCFG=false' >> /etc/default/grub
-
 #. Apply GRUB workaround::
 
      echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile.d/zpool_vdev_name_path.sh
@@ -35,41 +14,39 @@ Bootloader
      # GRUB fails to detect rpool name, hard code as "rpool"
      sed -i "s|rpool=.*|rpool=rpool|"  /etc/grub.d/10_linux
 
-   This workaround needs to be applied for every GRUB update, as the
+   This ``sed`` workaround needs to be applied for every
-   update will overwrite the changes.
+   GRUB update, as the update will overwrite the
+   changes.
 
 #. Install GRUB::
 
-      export ZPOOL_VDEV_NAME_PATH=YES
+      echo 'GRUB_ENABLE_BLSCFG=false' >> /etc/default/grub
       mkdir -p /boot/efi/almalinux/grub-bootdir/i386-pc/
       mkdir -p /boot/efi/almalinux/grub-bootdir/x86_64-efi/
       for i in ${DISK}; do
        grub2-install --target=i386-pc --boot-directory \
            /boot/efi/almalinux/grub-bootdir/i386-pc/  $i
       done
-
       cp -r /usr/lib/grub/x86_64-efi/ /boot/efi/EFI/almalinux/
-
-#. Generate GRUB menu::
-
       grub2-mkconfig -o /boot/efi/EFI/almalinux/grub.cfg
       grub2-mkconfig -o /boot/efi/almalinux/grub-bootdir/i386-pc/grub2/grub.cfg
 
 #. For both legacy and EFI booting: mirror ESP content::
 
-    ESP_MIRROR=$(mktemp -d)
     unalias -a
+    ESP_MIRROR=$(mktemp -d)
     cp -r /boot/efi/EFI $ESP_MIRROR
     for i in /boot/efis/*; do
      cp -r $ESP_MIRROR/EFI $i
     done
     rm -rf $ESP_MIRROR
 
-#. Notes for GRUB on RHEL
+#. Note: you need to regenerate GRUB menu after kernel
+   updates, otherwise computer will still boot old
+   kernel on reboot::
 
-   As bls is disabled, you will need to regenerate GRUB menu after each kernel upgrade.
+      grub2-mkconfig -o /boot/efi/EFI/almalinux/grub.cfg
-   Or else the new kernel will not be recognized and system will boot the old kernel
+      grub2-mkconfig -o /boot/efi/almalinux/grub-bootdir/i386-pc/grub2/grub.cfg
-   on reboot.
 
 Finish Installation
 ~~~~~~~~~~~~~~~~~~~~
@@ -87,16 +64,15 @@ Finish Installation
 
     reboot
 
-#. On first reboot, the boot process will fail, with failure messages such
-   as "You are in Emergency Mode...Press Ctrl-D to continue".
-
-   Wait for the computer to automatically reboot and the problem will be resolved.
-
 Post installaion
 ~~~~~~~~~~~~~~~~
 #. Install package groups::
 
     dnf group list --hidden -v       # query package groups
-    dnf group install @gnome-desktop
+    dnf group install gnome-desktop
 
 #. Add new user, configure swap.
+
+#. You can create a snapshot of the newly installed
+   system for later rollback,
+   see `this page <https://openzfs.github.io/openzfs-docs/Getting%20Started/Arch%20Linux/Root%20on%20ZFS/6-create-boot-environment.html>`__.

docs/Getting Started/RHEL-based distro/index.rst

@@ -26,19 +26,16 @@ encouraged to verify the authenticity of the ZFS on Linux public key using
 the fingerprint listed here.
 
 | **Location:** /etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
-| **EL6-8 Packages:** `zfs-release-el-2-1.noarch.rpm`
+| **Archived Repositories:** `see repo page <https://github.com/zfsonlinux/zfsonlinux.github.com/tree/master/epel>`__
-| **Archived Repositories:** `el6`, `el7_5`_, `el7_6`_, `el7_7`_, `el7_8`_, `el8_0`_, `el8_1`_, `el8_2`_, `el8_3`_, `el8_4`_, `el8_5`_, `el8_6`_
-| **List of Repositories** `see repo page <https://github.com/zfsonlinux/zfsonlinux.github.com/tree/master/epel>`__
 
 | **Signing key1 (Centos 8 and older, Fedora 36 and older)**
-  `pgp.mit.edu <https://pgp.mit.edu/pks/lookup?search=0xF14AB620&op=index&fingerprint=on>`__
+  `direct link <https://raw.githubusercontent.com/zfsonlinux/zfsonlinux.github.com/master/zfs-release/RPM-GPG-KEY-openzfs-key1>`__
 | **Fingerprint:** C93A FFFD 9F3F 7B03 C310 CEB6 A9D5 A1C0 F14A B620
 
 | **Signing key2 (Centos 9+, Fedora 37+)**
-  `pgp.mit.edu <https://pgp.mit.edu/pks/lookup?search=0xA599FD5E9DB84141&op=index&fingerprint=on>`__
+  `direct link <https://raw.githubusercontent.com/zfsonlinux/zfsonlinux.github.com/master/zfs-release/RPM-GPG-KEY-openzfs-key2>`__
 | **Fingerprint:** 7DC7 299D CF7C 7FD9 CD87 701B A599 FD5E 9DB8 4141
 
-
 For RHEL/CentOS versions 6 and 7 run::
 
  yum install https://zfsonlinux.org/epel/zfs-release-2-2$(rpm --eval "%{dist}").noarch.rpm
@@ -161,19 +158,5 @@ Start from "Preparation".
 
 .. _kABI-tracking kmod: https://elrepoproject.blogspot.com/2016/02/kabi-tracking-kmod-packages.html
 .. _DKMS: https://en.wikipedia.org/wiki/Dynamic_Kernel_Module_Support
-.. _zfs-release-el-2-1.noarch.rpm: https://zfsonlinux.org/epel/zfs-release-el-2-1.noarch.rpm
-.. _el6: https://zfsonlinux.org/epel/zfs-release.el6.noarch.rpm
-.. _el7_5: https://zfsonlinux.org/epel/zfs-release.el7_5.noarch.rpm
-.. _el7_6: https://zfsonlinux.org/epel/zfs-release.el7_6.noarch.rpm
-.. _el7_7: https://zfsonlinux.org/epel/zfs-release.el7_7.noarch.rpm
-.. _el7_8: https://zfsonlinux.org/epel/zfs-release.el7_8.noarch.rpm
-.. _el7_9: https://zfsonlinux.org/epel/zfs-release.el7_9.noarch.rpm
-.. _el8_0: https://zfsonlinux.org/epel/zfs-release.el8_0.noarch.rpm
-.. _el8_1: https://zfsonlinux.org/epel/zfs-release.el8_1.noarch.rpm
-.. _el8_2: https://zfsonlinux.org/epel/zfs-release.el8_2.noarch.rpm
-.. _el8_3: https://zfsonlinux.org/epel/zfs-release.el8_3.noarch.rpm
-.. _el8_4: https://zfsonlinux.org/epel/zfs-release.el8_4.noarch.rpm
-.. _el8_5: https://zfsonlinux.org/epel/zfs-release.el8_5.noarch.rpm
-.. _el8_6: https://zfsonlinux.org/epel/zfs-release.el8_6.noarch.rpm
 
 .. _EPEL repository: https://fedoraproject.org/wiki/EPEL