Add support for NixOS + Fedora, Arch, RH minor changes (#195)

* Add support for NixOS

Signed-off-by: Maurice Zhou <jasper@apvc.uk>

* updated files for immutable root fs

Signed-off-by: Maurice Zhou <jasper@apvc.uk>

* update ssh key location

Signed-off-by: Maurice Zhou <jasper@apvc.uk>

* change initrd host key name

Signed-off-by: Maurice Zhou <jasper@apvc.uk>

* add missing -p switch

Signed-off-by: Maurice Zhou <jasper@apvc.uk>

* comments about SSH and encrypted bpool

Signed-off-by: Maurice Zhou <jasper@apvc.uk>

* backup hardware-configuration.nix file

Signed-off-by: Maurice Zhou <jasper@apvc.uk>

* clearer wording for backup

Signed-off-by: Maurice Zhou <jasper@apvc.uk>

* global useDHCP flag is deprecated

Signed-off-by: Maurice Zhou <jasper@apvc.uk>

* point user to configuration.nix

Signed-off-by: Maurice Zhou <jasper@apvc.uk>

* fixes for systemd-autofs; network; typo

Signed-off-by: Maurice Zhou <jasper@apvc.uk>

* force command to return 0

Signed-off-by: Maurice Zhou <jasper@apvc.uk>

* fix command to postMount

Signed-off-by: Maurice Zhou <jasper@apvc.uk>

* fix mount command

Signed-off-by: Maurice Zhou <jasper@apvc.uk>

* escape $i

Signed-off-by: Maurice Zhou <jasper@apvc.uk>

* create mountpoints before booting

Signed-off-by: Maurice Zhou <jasper@apvc.uk>

* replace postDevice,preMount command

Signed-off-by: Maurice Zhou <jasper@apvc.uk>

* disable zfs-mount

Signed-off-by: Maurice Zhou <jasper@apvc.uk>

* generate machine-id

Signed-off-by: Maurice Zhou <jasper@apvc.uk>

docs/Getting Started/Arch Linux/Root on ZFS/0-overview.rst

@@ -62,12 +62,12 @@ Dataset layout
 |                           |                      |                      |                                     | datasets", "persistent datasets"; also    |
 |                           |                      |                      |                                     | include /var/lib, /srv, ...               |
 +---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
-|   bpool/sys/BOOT/default  | noauto               | legacy /boot         | no                                  | noauto is used to switch BE. because of   |
+|   bpool/sys/BOOT/default  | noauto               |        /boot         | no                                  | noauto is used to switch BE. because of   |
 |                           |                      |                      |                                     | noauto, must use fstab to mount           |
 +---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
 |   rpool/sys/ROOT/default  | noauto               | /                    | no                                  | mounted by initrd zfs hook                |
 +---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
-|   bpool/sys/BOOT/be1      | noauto               | legacy /boot         | no                                  | see bpool/sys/BOOT/default                |
+|   bpool/sys/BOOT/be1      | noauto               |        /boot         | no                                  | see bpool/sys/BOOT/default                |
 +---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
 |   rpool/sys/ROOT/be1      | noauto               | /                    | no                                  | see rpool/sys/ROOT/default                |
 +---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
@@ -120,3 +120,32 @@ Encryption
 
   As enabling Secure Boot is device specific, this is not
   covered in detail.
+
+Booting with disk failure
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This guide is written with disk failure in mind.
+
+If disks used in Root on ZFS pool failed, but
+sufficient redundancy for both root pool and boot pool
+still exists, the system will still boot normally.
+
+Swap partition on the failed disk will fail to mount,
+after an 1m30s timeout.
+
+This feature is useful for use cases such
+as an unattended remote server.
+
+Example:
+
+ - System has disks ``n>1``
+
+ - Installed with mirrored setup
+
+ - Mirrored setup can tolerate up to ``n-1`` disk failures
+
+ - Disconnect one or more disks, keep at least
+   one disk connected
+
+ - System still boots, but fails to mount swap and
+   EFI partition

docs/Getting Started/Arch Linux/Root on ZFS/1-preparation.rst

@@ -73,7 +73,7 @@ Preparation
     ls /dev/disk/by-id/*
 
    If using virtio as disk bus, use
-   ``/dev/disk/by-path/*`` or ``/dev/vd*``.
+   ``/dev/disk/by-path/*``.
 
    Declare disk array::
 

docs/Getting Started/Arch Linux/Root on ZFS/2-system-installation.rst

@@ -169,12 +169,11 @@ System Installation
     zfs create -o canmount=off -o mountpoint=none bpool_$INST_UUID/$INST_ID/BOOT
     zfs create -o canmount=off -o mountpoint=none rpool_$INST_UUID/$INST_ID/ROOT
     zfs create -o canmount=off -o mountpoint=none rpool_$INST_UUID/$INST_ID/DATA
-    zfs create -o mountpoint=legacy -o canmount=noauto bpool_$INST_UUID/$INST_ID/BOOT/default
+    zfs create -o mountpoint=/boot -o canmount=noauto bpool_$INST_UUID/$INST_ID/BOOT/default
     zfs create -o mountpoint=/ -o canmount=off    rpool_$INST_UUID/$INST_ID/DATA/default
     zfs create -o mountpoint=/ -o canmount=noauto rpool_$INST_UUID/$INST_ID/ROOT/default
     zfs mount rpool_$INST_UUID/$INST_ID/ROOT/default
-    mkdir /mnt/boot
+    zfs mount bpool_$INST_UUID/$INST_ID/BOOT/default
-    mount -t zfs bpool_$INST_UUID/$INST_ID/BOOT/default /mnt/boot
     for i in {usr,var,var/lib};
     do
         zfs create -o canmount=off rpool_$INST_UUID/$INST_ID/DATA/default/$i

docs/Getting Started/Arch Linux/Root on ZFS/3-system-configuration.rst

@@ -11,21 +11,9 @@ System Configuration
 
     echo GRUB_CMDLINE_LINUX=\"zfs_import_dir=${INST_PRIMARY_DISK%/*}\" >> /mnt/etc/default/grub
 
-#. Generate list of datasets for `zfs-mount-generator
-   <https://manpages.ubuntu.com/manpages/focal/man8/zfs-mount-generator.8.html>`__ to mount them at boot::
-
-    # tab-separated zfs properties
-    # see /etc/zfs/zed.d/history_event-zfs-list-cacher.sh
-    export \
-    PROPS="name,mountpoint,canmount,atime,relatime,devices,exec\
-    ,readonly,setuid,nbmand,encroot,keylocation"
-    mkdir -p /mnt/etc/zfs/zfs-list.cache
-    zfs list -H -t filesystem -o $PROPS -r rpool_$INST_UUID > /mnt/etc/zfs/zfs-list.cache/rpool_$INST_UUID
-    sed -Ei "s|/mnt/?|/|" /mnt/etc/zfs/zfs-list.cache/*
-
 #. Generate fstab::
 
-    echo bpool_$INST_UUID/$INST_ID/BOOT/default /boot zfs rw,xattr,posixacl 0 0 >> /mnt/etc/fstab
+    genfstab -U /mnt | sed 's;zfs[[:space:]]*;zfs zfsutil,;g' | grep "zfs zfsutil" >> /mnt/etc/fstab
     for i in ${DISK}; do
        echo UUID=$(blkid -s UUID -o value ${i}-part1) /boot/efis/${i##*/}-part1 vfat \
        x-systemd.idle-timeout=1min,x-systemd.automount,noauto,umask=0022,fmask=0022,dmask=0022 0 1 >> /mnt/etc/fstab
@@ -108,7 +96,7 @@ System Configuration
     systemctl enable zfs-import-scan.service zfs-import.target zfs-zed zfs.target --root=/mnt
     systemctl disable zfs-mount --root=/mnt
 
-   At boot, datasets on rpool are mounted with ``zfs-mount-generator``,
+   At boot, datasets on rpool are mounted with ``/etc/fstab``,
    which can control the mounting process more precisely than ``zfs-mount.service``.
 
 #. Chroot::

docs/Getting Started/Arch Linux/Root on ZFS/4-optional-configuration.rst

@@ -227,7 +227,7 @@ Persistent swap and hibernation
     resume=/dev/mapper/${INST_SWAPMAPPER}\"" \
     >> /etc/default/grub
 
-   Note that hibernation might not work with discrete graphics or
+   Note that hibernation might not work with discrete graphics, virtio graphics or
    AMD APU integrated graphics. This is not specific to this guide.
 
    Computer must resume from a continuous swap space, resume

docs/Getting Started/Fedora/Root on ZFS/0-overview.rst

@@ -63,12 +63,12 @@ Dataset layout
 |                           |                      |                      |                                     | datasets", "persistent datasets"; also    |
 |                           |                      |                      |                                     | include /var/lib, /srv, ...               |
 +---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
-|   bpool/sys/BOOT/default  | noauto               | legacy /boot         | no                                  | noauto is used to switch BE. because of   |
+|   bpool/sys/BOOT/default  | noauto               |        /boot         | no                                  | noauto is used to switch BE. because of   |
 |                           |                      |                      |                                     | noauto, must use fstab to mount           |
 +---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
 |   rpool/sys/ROOT/default  | noauto               | /                    | no                                  | mounted by initrd zfs hook                |
 +---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
-|   bpool/sys/BOOT/be1      | noauto               | legacy /boot         | no                                  | see bpool/sys/BOOT/default                |
+|   bpool/sys/BOOT/be1      | noauto               |        /boot         | no                                  | see bpool/sys/BOOT/default                |
 +---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
 |   rpool/sys/ROOT/be1      | noauto               | /                    | no                                  | see rpool/sys/ROOT/default                |
 +---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
@@ -110,3 +110,32 @@ Encryption
   which should be sufficient for most purposes.
 
   Secure Boot is not supported out-of-the-box due to ZFS module.
+
+Booting with disk failure
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This guide is written with disk failure in mind.
+
+If disks used in Root on ZFS pool failed, but
+sufficient redundancy for both root pool and boot pool
+still exists, the system will still boot normally.
+
+Swap partition on the failed disk will fail to mount,
+after an 1m30s timeout.
+
+This feature is useful for use cases such
+as an unattended remote server.
+
+Example:
+
+ - System has disks ``n>1``
+
+ - Installed with mirrored setup
+
+ - Mirrored setup can tolerate up to ``n-1`` disk failures
+
+ - Disconnect one or more disks, keep at least
+   one disk connected
+
+ - System still boots, but fails to mount swap and
+   EFI partition

docs/Getting Started/Fedora/Root on ZFS/1-preparation.rst

@@ -71,7 +71,7 @@ Preparation
     ls /dev/disk/by-id/*
 
    If using virtio as disk bus, use
-   ``/dev/disk/by-path/*`` or ``/dev/vd*``.
+   ``/dev/disk/by-path/*``.
 
    Declare disk array::
 

docs/Getting Started/Fedora/Root on ZFS/2-system-installation.rst

@@ -178,12 +178,11 @@ System Installation
     zfs create -o canmount=off -o mountpoint=none bpool_$INST_UUID/$INST_ID/BOOT
     zfs create -o canmount=off -o mountpoint=none rpool_$INST_UUID/$INST_ID/ROOT
     zfs create -o canmount=off -o mountpoint=none rpool_$INST_UUID/$INST_ID/DATA
-    zfs create -o mountpoint=legacy -o canmount=noauto bpool_$INST_UUID/$INST_ID/BOOT/default
+    zfs create -o mountpoint=/boot -o canmount=noauto bpool_$INST_UUID/$INST_ID/BOOT/default
     zfs create -o mountpoint=/ -o canmount=off    rpool_$INST_UUID/$INST_ID/DATA/default
     zfs create -o mountpoint=/ -o canmount=noauto rpool_$INST_UUID/$INST_ID/ROOT/default
     zfs mount rpool_$INST_UUID/$INST_ID/ROOT/default
-    mkdir /mnt/boot
+    zfs mount bpool_$INST_UUID/$INST_ID/BOOT/default
-    mount -t zfs bpool_$INST_UUID/$INST_ID/BOOT/default /mnt/boot
     for i in {usr,var,var/lib};
     do
         zfs create -o canmount=off rpool_$INST_UUID/$INST_ID/DATA/default/$i

docs/Getting Started/Fedora/Root on ZFS/3-system-configuration.rst

@@ -6,21 +6,9 @@ System Configuration
 .. contents:: Table of Contents
    :local:
 
-#. Generate list of datasets for `zfs-mount-generator
-   <https://manpages.ubuntu.com/manpages/focal/man8/zfs-mount-generator.8.html>`__ to mount them at boot::
-
-    # tab-separated zfs properties
-    # see /etc/zfs/zed.d/history_event-zfs-list-cacher.sh
-    export \
-    PROPS="name,mountpoint,canmount,atime,relatime,devices,exec\
-    ,readonly,setuid,nbmand,encroot,keylocation"
-    mkdir -p /mnt/etc/zfs/zfs-list.cache
-    zfs list -H -t filesystem -o $PROPS -r rpool_$INST_UUID > /mnt/etc/zfs/zfs-list.cache/rpool_$INST_UUID
-    sed -Ei "s|/mnt/?|/|" /mnt/etc/zfs/zfs-list.cache/*
-
 #. Generate fstab::
 
-    echo bpool_$INST_UUID/$INST_ID/BOOT/default /boot zfs rw,xattr,posixacl 0 0 >> /mnt/etc/fstab
+    genfstab -U /mnt | sed 's;zfs[[:space:]]*;zfs zfsutil,;g' | grep "zfs zfsutil" >> /mnt/etc/fstab
     for i in ${DISK}; do
        echo UUID=$(blkid -s UUID -o value ${i}-part1) /boot/efis/${i##*/}-part1 vfat \
        x-systemd.idle-timeout=1min,x-systemd.automount,noauto,umask=0022,fmask=0022,dmask=0022 0 1 >> /mnt/etc/fstab
@@ -78,7 +66,7 @@ System Configuration
     systemctl enable zfs-import-scan.service zfs-import.target zfs-zed zfs.target --root=/mnt
     systemctl disable zfs-mount --root=/mnt
 
-   At boot, datasets on rpool are mounted with ``zfs-mount-generator``,
+   At boot, datasets on rpool are mounted with ``/etc/fstab``,
    which can control the mounting process more precisely than ``zfs-mount.service``.
 
 

docs/Getting Started/NixOS/Root on ZFS.rst

@@ -0,0 +1,11 @@
+NixOS Root on ZFS
+=======================================
+`Start here <Root%20on%20ZFS/0-overview.html>`__.
+
+Contents
+--------
+.. toctree::
+  :maxdepth: 2
+  :glob:
+
+  Root on ZFS/*

docs/Getting Started/NixOS/Root on ZFS/0-overview.rst

@@ -0,0 +1,142 @@
+.. highlight:: sh
+
+Overview
+======================
+This document describes how to install NixOS with ZFS as root
+file system.
+
+Caution
+~~~~~~~
+- This guide wipes entire physical disks. Back up existing data.
+- `GRUB does not and
+  will not work on 4Kn drive with legacy (BIOS) booting.
+  <http://savannah.gnu.org/bugs/?46700>`__
+
+Partition layout
+~~~~~~~~~~~~~~~~
+
+GUID partition table (GPT) is used.
+EFI system partition will be referred to as **ESP** in this document.
+
++----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+
+| Name                 | legacy boot          | ESP                   | Boot pool            | swap                | root pool             | remaining space |
++======================+======================+=======================+======================+=====================+=======================+=================+
+| File system          |                      |  vfat                 | ZFS                  | swap                | ZFS                   |                 |
++----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+
+| Size                 |  1M                  |  2G                   | 4G                   | depends on RAM size |                       |                 |
++----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+
+| Optional encryption  |                      |  *Secure Boot*        |                      | plain dm-crypt      | ZFS native encryption |                 |
+|                      |                      |                       |                      |                     |                       |                 |
++----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+
+| Partition no.        | 5                    | 1                     | 2                    | 4                   | 3                     |                 |
++----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+
+| Mount point          |                      |                       | /boot                |                     | /                     |                 |
+|                      |                      | /boot/efis/disk-part1 |                      |                     |                       |                 |
++----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+
+
+Dataset layout
+~~~~~~~~~~~~~~
+
+The dataset layout used in this guide follows stardard
+mutable file positions (``/var``, ``/etc``, ...), but can
+still be modified to `a immutable root <https://grahamc.com/blog/erase-your-darlings>`__
+after installation.
+
++---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
+| Dataset                   | canmount             | mountpoint           | container                           | notes                                     |
++===========================+======================+======================+=====================================+===========================================+
+| bpool                     | off                  | /boot                | contains sys                        |                                           |
++---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
+| rpool                     | off                  | /                    | contains sys                        |                                           |
++---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
+| bpool/sys                 | off                  | none                 | contains BOOT                       |                                           |
++---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
+|      rpool/sys            | off                  | none                 | contains ROOT                       | sys is encryptionroot                     |
++---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
+|      bpool/sys/BOOT       | off                  | none                 | contains boot environments          |                                           |
++---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
+|      rpool/sys/ROOT       | off                  | none                 | contains boot environments          |                                           |
++---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
+|      rpool/sys/DATA       | off                  | none                 | contains placeholder "default"      |                                           |
++---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
+|  rpool/sys/DATA/default   | off                  | /                    | contains user datasets              | child datsets inherits mountpoint         |
++---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
+|  rpool/sys/DATA/local     | off                  | /                    | contains /nix datasets              | child datsets inherits mountpoint         |
++---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
+| rpool/sys/DATA/default/   | on                   |  /home (inherited)   | no                                  |                                           |
+| home                      |                      |                      |                                     | user datasets, also called "shared        |
+|                           |                      |                      |                                     | datasets", "persistent datasets"; also    |
+|                           |                      |                      |                                     | include /var/lib, /srv, ...               |
++---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
+|   bpool/sys/BOOT/default  | noauto               |        /boot         | no                                  | noauto is used to switch BE. because of   |
+|                           |                      |                      |                                     | noauto, must use fstab to mount           |
++---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
+|   rpool/sys/ROOT/default  | noauto               | /                    | no                                  | mounted by initrd zfs hook                |
++---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
+
+Encryption
+~~~~~~~~~~
+
+- Swap
+
+  Swap is always encrypted. By default, swap is encrypted
+  with plain dm-crypt with key generated from ``/dev/urandom``
+  at every boot. Swap content does not persist between reboots.
+
+- Root pool
+
+  ZFS native encryption can be optionally enabled for ``rpool/sys``
+  and child datasets.
+
+  User should be aware that, ZFS native encryption does not
+  encrypt some metadata of the datasets.
+  ZFS native encryption also does not change master key when ``zfs change-key`` is invoked.
+  Therefore, you should wipe the disk when password is compromised to protect confidentiality.
+  See `zfs-load-key.8 <https://openzfs.github.io/openzfs-docs/man/8/zfs-load-key.8.html>`__
+  and `zfs-change-key.8 <https://openzfs.github.io/openzfs-docs/man/8/zfs-change-key.8.html>`__
+  for more information regarding ZFS native encryption.
+
+  Encryption is enabled at dataset creation and can not be disabled later.
+
+- Boot pool
+
+  Boot pool can not be encrypted.
+
+- Bootloader
+
+  Bootloader can not be encrypted.
+
+  However, with Secure Boot, bootloader
+  can be verified by motherboard firmware to be untempered,
+  which should be sufficient for most purposes.
+
+  Secure Boot is not supported out-of-the-box due to ZFS module.
+
+Booting with disk failure
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This guide is written with disk failure in mind.
+
+If disks used in Root on ZFS pool failed, but
+sufficient redundancy for both root pool and boot pool
+still exists, the system will still boot normally.
+
+Swap partition on the failed disk will fail to mount,
+after an 1m30s timeout.
+
+This feature is useful for use cases such
+as an unattended remote server.
+
+Example:
+
+ - System has disks ``n>1``
+
+ - Installed with mirrored setup
+
+ - Mirrored setup can tolerate up to ``n-1`` disk failures
+
+ - Disconnect one or more disks, keep at least
+   one disk connected
+
+ - System still boots, but fails to mount swap and
+   EFI partition

docs/Getting Started/NixOS/Root on ZFS/1-preparation.rst

@@ -0,0 +1,109 @@
+.. highlight:: sh
+
+Preparation
+======================
+
+.. contents:: Table of Contents
+   :local:
+
+#. Download `Minimal ISO image
+   <https://channels.nixos.org/nixos-21.05/latest-nixos-minimal-x86_64-linux.iso>`__ and boot from it.
+
+#. Connect to network. See `NixOS manual <https://nixos.org/manual/nixos/stable/index.html#sec-installation-booting>`__.
+
+#. SSH server is enabled by default. To connect, set root password with::
+
+    sudo passwd
+
+#. Connect from another computer::
+
+    ssh root@192.168.1.19
+
+#. Unique pool suffix. ZFS expects pool names to be
+   unique, therefore it's recommended to create
+   pools with a unique suffix::
+
+    INST_UUID=$(dd if=/dev/urandom bs=1 count=100 2>/dev/null | tr -dc 'a-z0-9' | cut -c-6)
+
+#. Identify this installation in ZFS filesystem path::
+
+    INST_ID=nixos
+
+#. Root on ZFS configuration file name::
+
+    INST_CONFIG_FILE='zfs.nix'
+
+#. Target disk
+
+   List available disks with::
+
+    ls /dev/disk/by-id/*
+
+   If using virtio as disk bus, use
+   ``/dev/disk/by-path/*``.
+
+   Declare disk array::
+
+    DISK='/dev/disk/by-id/ata-FOO /dev/disk/by-id/nvme-BAR'
+
+   For single disk installation, use::
+
+    DISK='/dev/disk/by-id/disk1'
+
+#. Choose a primary disk. This disk will be used
+   for primary EFI partition, default to
+   first disk in the array::
+
+    INST_PRIMARY_DISK=$(echo $DISK | cut -f1 -d\ )
+
+#. Set vdev topology, possible values are:
+
+   - (not set, single disk or striped; no redundancy)
+   - mirror
+   - raidz1
+   - raidz2
+   - raidz3
+
+   ::
+
+    INST_VDEV=
+
+   This will create a single vdev with the topology of your choice.
+   It is also possible to manually create a pool with multiple vdevs, such as::
+
+    zpool create --options \
+          poolName \
+          mirror sda sdb \
+          raidz2 sdc ... \
+          raidz3 sde ... \
+          spare  sdf ...
+
+   Notice the cost of parity when using RAID-Z. See
+   `here <https://www.delphix.com/blog/delphix-engineering/zfs-raidz-stripe-width-or-how-i-learned-stop-worrying-and-love-raidz>`__
+   and `here <https://docs.google.com/spreadsheets/d/1tf4qx1aMJp8Lo_R6gpT689wTjHv6CGVElrPqTA0w_ZY/>`__.
+
+   Refer to `zpoolconcepts <https://openzfs.github.io/openzfs-docs/man/7/zpoolconcepts.7.html>`__
+   and `zpool-create <https://openzfs.github.io/openzfs-docs/man/8/zpool-create.8.html>`__
+   man pages for details.
+
+#. Set partition size:
+
+   Set ESP size::
+
+    INST_PARTSIZE_ESP=2 # in GB
+
+   Set boot pool size. To avoid running out of space while using
+   boot environments, the minimum is 4GB. Adjust the size if you
+   intend to use multiple kernel/distros::
+
+    INST_PARTSIZE_BPOOL=4
+
+   Set swap size. It's `recommended <https://chrisdown.name/2018/01/02/in-defence-of-swap.html>`__
+   to setup a swap partition. If you intend to use hibernation,
+   the minimum should be no less than RAM size. Skip if swap is not needed::
+
+    INST_PARTSIZE_SWAP=8
+
+   Root pool size, use all remaining disk space if not set::
+
+    INST_PARTSIZE_RPOOL=

docs/Getting Started/NixOS/Root on ZFS/2-system-configuration.rst

@@ -0,0 +1,356 @@
+.. highlight:: sh
+
+System Configuration
+======================
+
+.. contents:: Table of Contents
+   :local:
+
+#. Optional: wipe solid-state drives with the generic tool
+   `blkdiscard <https://utcc.utoronto.ca/~cks/space/blog/linux/ErasingSSDsWithBlkdiscard>`__,
+   to clean previous partition tables and improve performance.
+
+   All content will be irrevocably destroyed::
+
+    for i in ${DISK}; do
+    blkdiscard -f $i &
+    done
+    wait
+
+   This is a quick operation and should be completed under one
+   minute.
+
+   For other device specific methods, see
+   `Memory cell clearing <https://wiki.archlinux.org/title/Solid_state_drive/Memory_cell_clearing>`__
+
+#. Partition the disks.
+   See `Overview <0-overview.html>`__ for details::
+
+     for i in ${DISK}; do
+     sgdisk --zap-all $i
+     sgdisk -n1:1M:+${INST_PARTSIZE_ESP}G -t1:EF00 $i
+     sgdisk -n2:0:+${INST_PARTSIZE_BPOOL}G -t2:BE00 $i
+     if [ "${INST_PARTSIZE_SWAP}" != "" ]; then
+         sgdisk -n4:0:+${INST_PARTSIZE_SWAP}G -t4:8200 $i
+     fi
+     if [ "${INST_PARTSIZE_RPOOL}" = "" ]; then
+         sgdisk -n3:0:0   -t3:BF00 $i
+     else
+         sgdisk -n3:0:+${INST_PARTSIZE_RPOOL}G -t3:BF00 $i
+     fi
+     sgdisk -a1 -n5:24K:+1000K -t5:EF02 $i
+     done
+
+#. Create boot pool::
+
+    zpool create \
+    -d -o feature@async_destroy=enabled \
+    -o feature@bookmarks=enabled \
+    -o feature@embedded_data=enabled \
+    -o feature@empty_bpobj=enabled \
+    -o feature@enabled_txg=enabled \
+    -o feature@extensible_dataset=enabled \
+    -o feature@filesystem_limits=enabled \
+    -o feature@hole_birth=enabled \
+    -o feature@large_blocks=enabled \
+    -o feature@lz4_compress=enabled \
+    -o feature@spacemap_histogram=enabled \
+        -o ashift=12 \
+        -o autotrim=on \
+        -O acltype=posixacl \
+        -O canmount=off \
+        -O compression=lz4 \
+        -O devices=off \
+        -O normalization=formD \
+        -O relatime=on \
+        -O xattr=sa \
+        -O mountpoint=/boot \
+        -R /mnt \
+        bpool_$INST_UUID \
+        $INST_VDEV \
+        $(for i in ${DISK}; do
+           printf "$i-part2 ";
+          done)
+
+   You should not need to customize any of the options for the boot pool.
+
+   GRUB does not support all of the zpool features. See ``spa_feature_names``
+   in `grub-core/fs/zfs/zfs.c
+   <http://git.savannah.gnu.org/cgit/grub.git/tree/grub-core/fs/zfs/zfs.c#n276>`__.
+   This step creates a separate boot pool for ``/boot`` with the features
+   limited to only those that GRUB supports, allowing the root pool to use
+   any/all features.
+
+   Features enabled with ``-o compatibility=grub2`` can be seen
+   `here <https://github.com/openzfs/zfs/blob/master/cmd/zpool/compatibility.d/grub2>`__.
+
+#. Create root pool::
+
+       zpool create \
+           -o ashift=12 \
+           -o autotrim=on \
+           -R /mnt \
+           -O acltype=posixacl \
+           -O canmount=off \
+           -O compression=zstd \
+           -O dnodesize=auto \
+           -O normalization=formD \
+           -O relatime=on \
+           -O xattr=sa \
+           -O mountpoint=/ \
+           rpool_$INST_UUID \
+           $INST_VDEV \
+          $(for i in ${DISK}; do
+             printf "$i-part3 ";
+            done)
+
+   **Notes:**
+
+   - The use of ``ashift=12`` is recommended here because many drives
+     today have 4 KiB (or larger) physical sectors, even though they
+     present 512 B logical sectors. Also, a future replacement drive may
+     have 4 KiB physical sectors (in which case ``ashift=12`` is desirable)
+     or 4 KiB logical sectors (in which case ``ashift=12`` is required).
+   - Setting ``-O acltype=posixacl`` enables POSIX ACLs globally. If you
+     do not want this, remove that option, but later add
+     ``-o acltype=posixacl`` (note: lowercase “o”) to the ``zfs create``
+     for ``/var/log``, as `journald requires ACLs
+     <https://askubuntu.com/questions/970886/journalctl-says-failed-to-search-journal-acl-operation-not-supported>`__
+   - Setting ``normalization=formD`` eliminates some corner cases relating
+     to UTF-8 filename normalization. It also implies ``utf8only=on``,
+     which means that only UTF-8 filenames are allowed. If you care to
+     support non-UTF-8 filenames, do not use this option. For a discussion
+     of why requiring UTF-8 filenames may be a bad idea, see `The problems
+     with enforced UTF-8 only filenames
+     <http://utcc.utoronto.ca/~cks/space/blog/linux/ForcedUTF8Filenames>`__.
+   - ``recordsize`` is unset (leaving it at the default of 128 KiB). If you
+     want to tune it (e.g. ``-o recordsize=1M``), see `these
+     <https://jrs-s.net/2019/04/03/on-zfs-recordsize/>`__ `various
+     <http://blog.programster.org/zfs-record-size>`__ `blog
+     <https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSFileRecordsizeGrowth>`__
+     `posts
+     <https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSRecordsizeAndCompression>`__.
+   - Setting ``relatime=on`` is a middle ground between classic POSIX
+     ``atime`` behavior (with its significant performance impact) and
+     ``atime=off`` (which provides the best performance by completely
+     disabling atime updates). Since Linux 2.6.30, ``relatime`` has been
+     the default for other filesystems. See `RedHat’s documentation
+     <https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/power_management_guide/relatime>`__
+     for further information.
+   - Setting ``xattr=sa`` `vastly improves the performance of extended
+     attributes
+     <https://github.com/zfsonlinux/zfs/commit/82a37189aac955c81a59a5ecc3400475adb56355>`__.
+     Inside ZFS, extended attributes are used to implement POSIX ACLs.
+     Extended attributes can also be used by user-space applications.
+     `They are used by some desktop GUI applications.
+     <https://en.wikipedia.org/wiki/Extended_file_attributes#Linux>`__
+     `They can be used by Samba to store Windows ACLs and DOS attributes;
+     they are required for a Samba Active Directory domain controller.
+     <https://wiki.samba.org/index.php/Setting_up_a_Share_Using_Windows_ACLs>`__
+     Note that ``xattr=sa`` is `Linux-specific
+     <https://openzfs.org/wiki/Platform_code_differences>`__. If you move your
+     ``xattr=sa`` pool to another OpenZFS implementation besides ZFS-on-Linux,
+     extended attributes will not be readable (though your data will be). If
+     portability of extended attributes is important to you, omit the
+     ``-O xattr=sa`` above. Even if you do not want ``xattr=sa`` for the whole
+     pool, it is probably fine to use it for ``/var/log``.
+   - Make sure to include the ``-part3`` portion of the drive path. If you
+     forget that, you are specifying the whole disk, which ZFS will then
+     re-partition, and you will lose the bootloader partition(s).
+
+#. This section implements dataset layout as described in `overview <0-overview.html>`__.
+
+   Create root system container:
+
+   - Unencrypted::
+
+      zfs create \
+       -o canmount=off \
+       -o mountpoint=none \
+       rpool_$INST_UUID/$INST_ID
+
+   - Encrypted:
+
+     Pick a strong password. Once compromised, changing password will not keep your
+     data safe. See ``zfs-change-key(8)`` for more info::
+
+      zfs create \
+       -o canmount=off \
+       -o mountpoint=none \
+       -o encryption=aes-256-gcm \
+       -o keylocation=prompt \
+       -o keyformat=passphrase \
+       rpool_$INST_UUID/$INST_ID
+
+   Create other system datasets::
+
+    zfs create -o canmount=off -o mountpoint=none bpool_$INST_UUID/$INST_ID
+    zfs create -o canmount=off -o mountpoint=none bpool_$INST_UUID/$INST_ID/BOOT
+    zfs create -o canmount=off -o mountpoint=none rpool_$INST_UUID/$INST_ID/ROOT
+    zfs create -o canmount=off -o mountpoint=none rpool_$INST_UUID/$INST_ID/DATA
+    zfs create -o mountpoint=/boot -o canmount=noauto bpool_$INST_UUID/$INST_ID/BOOT/default
+    zfs create -o mountpoint=/ -o canmount=off    rpool_$INST_UUID/$INST_ID/DATA/default
+    zfs create -o mountpoint=/ -o canmount=off    rpool_$INST_UUID/$INST_ID/DATA/local
+    zfs create -o mountpoint=/ -o canmount=noauto rpool_$INST_UUID/$INST_ID/ROOT/default
+    zfs mount rpool_$INST_UUID/$INST_ID/ROOT/default
+    zfs mount bpool_$INST_UUID/$INST_ID/BOOT/default
+    for i in {usr,var,var/lib};
+    do
+        zfs create -o canmount=off rpool_$INST_UUID/$INST_ID/DATA/default/$i
+    done
+    for i in {home,root,srv,usr/local,var/log,var/spool};
+    do
+        zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/$i
+    done
+    chmod 750 /mnt/root
+    for i in {nix,}; do
+        zfs create -o canmount=on -o mountpoint=/$i rpool_$INST_UUID/$INST_ID/DATA/local/$i
+    done
+
+   Datasets for immutable root filesystem::
+
+    zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/state
+    for i in {/etc/nixos,/etc/cryptkey.d}; do
+      mkdir -p /mnt/state/$i /mnt/$i
+      mount -o bind /mnt/state/$i /mnt/$i
+    done
+    zfs create -o mountpoint=/ -o canmount=noauto rpool_$INST_UUID/$INST_ID/ROOT/empty
+    zfs snapshot rpool_$INST_UUID/$INST_ID/ROOT/empty@start
+
+#. Format and mount ESP::
+
+    for i in ${DISK}; do
+     mkfs.vfat -n EFI ${i}-part1
+     mkdir -p /mnt/boot/efis/${i##*/}-part1
+     mount -t vfat ${i}-part1 /mnt/boot/efis/${i##*/}-part1
+    done
+
+#. Create optional user data datasets to omit data from rollback::
+
+     zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/games
+     zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/www
+     # for GNOME
+     zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/AccountsService
+     # for Docker
+     zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/docker
+     # for NFS
+     zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/nfs
+     # for LXC
+     zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/lxc
+     # for LibVirt
+     zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/libvirt
+     ##other application
+     # zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/$name
+
+   Add other datasets when needed, such as PostgreSQL.
+
+#. Generate initial NixOS system configuration::
+
+    nixos-generate-config --root /mnt
+
+   This command will generate two files, ``configuration.nix``
+   and ``hardware-configuration-zfs.nix``, which will be the starting point
+   of configuring the system.
+
+#. Edit config file to import ZFS options::
+
+    sed -i "s|./hardware-configuration.nix|./hardware-configuration-zfs.nix ./${INST_CONFIG_FILE}|g" /mnt/etc/nixos/configuration.nix
+    # backup, prevent being overwritten by nixos-generate-config
+    mv /mnt/etc/nixos/hardware-configuration.nix /mnt/etc/nixos/hardware-configuration-zfs.nix
+
+#. ZFS options::
+
+    tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} <<EOF
+    { config, pkgs, ... }:
+
+    { boot.supportedFilesystems = [ "zfs" ];
+      networking.hostId = "$(head -c 8 /etc/machine-id)";
+      boot.zfs.devNodes = "${INST_PRIMARY_DISK%/*}";
+    EOF
+
+   ZFS datasets should be mounted with ``-o zfsutil`` option::
+
+    sed -i 's|fsType = "zfs";|fsType = "zfs"; options = [ "zfsutil" ];|g' \
+    /mnt/etc/nixos/hardware-configuration-zfs.nix
+
+   Allow EFI system partition mounting to fail at boot::
+
+    sed -i 's|fsType = "vfat";|fsType = "vfat"; options = [ "x-systemd.idle-timeout=1min" "x-systemd.automount" "noauto" ];|g' \
+    /mnt/etc/nixos/hardware-configuration-zfs.nix
+
+   Disable cache::
+
+    mkdir -p /mnt/state/etc/zfs/
+    rm -f /mnt/state/etc/zfs/zpool.cache
+    touch /mnt/state/etc/zfs/zpool.cache
+    chmod a-w /mnt/state/etc/zfs/zpool.cache
+    chattr +i /mnt/state/etc/zfs/zpool.cache
+
+#. If swap is enabled::
+
+    if [ "${INST_PARTSIZE_SWAP}" != "" ]; then
+    sed -i '/swapDevices/d' /mnt/etc/nixos/hardware-configuration-zfs.nix
+
+    tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} <<EOF
+      swapDevices = [
+    EOF
+    for i in $DISK; do
+    tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} <<EOF
+        { device = "$i-part4"; randomEncryption.enable = true; }
+    EOF
+    done
+    tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} <<EOF
+      ];
+    EOF
+    fi
+
+#. For immutable root file system, save machine-id and other files::
+
+    mkdir -p /mnt/state/etc/{ssh,zfs}
+    systemd-machine-id-setup --print > /mnt/state/etc/machine-id
+    tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} <<EOF
+      systemd.services.zfs-mount.enable = false;
+      environment.etc."machine-id".source = "/state/etc/machine-id";
+      environment.etc."zfs/zpool.cache".source
+        = "/state/etc/zfs/zpool.cache";
+    EOF
+
+#. Configure GRUB boot loader for both legacy boot and UEFI::
+
+    sed -i '/boot.loader/d' /mnt/etc/nixos/configuration.nix
+    tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} <<EOF
+      boot.loader = {
+        generationsDir.copyKernels = true;
+        ##for problematic UEFI firmware
+        grub.efiInstallAsRemovable = true;
+        efi.canTouchEfiVariables = false;
+        ##if UEFI firmware can detect entries
+        #efi.canTouchEfiVariables = true;
+        efi.efiSysMountPoint = "/boot/efis/${INST_PRIMARY_DISK##*/}-part1";
+        grub.enable = true;
+        grub.version = 2;
+        grub.copyKernels = true;
+        grub.efiSupport = true;
+        grub.zfsSupport = true;
+        # for systemd-autofs
+        grub.extraPrepareConfig = ''
+          mkdir -p /boot/efis
+          for i in  /boot/efis/*; do mount \$i ; done
+        '';
+        grub.devices = [
+    EOF
+    for i in $DISK; do
+      printf "      \"$i\"\n" >>/mnt/etc/nixos/${INST_CONFIG_FILE}
+    done
+    tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} <<EOF
+        ];
+        grub.mirroredBoots = [
+    EOF
+    for i in $DISK; do
+      printf "      { devices = [ \"$i\" ] ; efiSysMountPoint = \"/boot/efis/${i##*/}-part1\"; path = \"/boot\"; }\n" \
+      >>/mnt/etc/nixos/${INST_CONFIG_FILE}
+    done
+    tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} <<EOF
+        ];
+      };
+    EOF

docs/Getting Started/NixOS/Root on ZFS/3-optional-configuration.rst

@@ -0,0 +1,254 @@
+.. highlight:: sh
+
+Optional Configuration
+======================
+
+.. contents:: Table of Contents
+   :local:
+
+Skip to `System Installation <./4-system-installation.html>`__ section if
+no optional configuration is needed.
+
+Supply password with SSH
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Note: if you choose to encrypt boot pool, where decryption is handled
+by GRUB, as described in the next section, configuration performed
+in this section will have no effect.
+
+This example uses DHCP::
+
+ mkdir -p /mnt/etc/state/ssh/
+ ssh-keygen -t ed25519 -N "" -f /mnt/state/etc/ssh/ssh_host_initrd_ed25519_key
+ tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} <<EOF
+   #networking.interfaces.enp1s0.useDHCP = true;
+   boot = {
+     initrd.network = {
+       enable = true;
+       ssh = {
+         enable = true;
+         hostKeys = [ /state/etc/ssh/ssh_host_initrd_ed25519_key ];
+         authorizedKeys = [ "$YOUR_PUBLIC_KEY" ];
+       };
+       postCommands = ''
+         echo "zfs load-key -a; killall zfs" >> /root/.profile
+       '';
+     };
+   };
+ EOF
+
+Encrypt boot pool
+~~~~~~~~~~~~~~~~~~~
+Note: This will disable password with SSH. The password previously set for
+root pool will be replaced by keyfile, embedded in initrd.
+
+#. Add package::
+
+    tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} <<EOF
+      environment.systemPackages = [ pkgs.cryptsetup ];
+    EOF
+
+#. LUKS password::
+
+    LUKS_PWD=secure-passwd
+
+   You will need to enter the same password for
+   each disk at boot. As root pool key is
+   protected by this password, the previous warning
+   about password strength still apply.
+
+   Double-check password here. Complete reinstallation is
+   needed if entered wrong.
+
+#. Create encryption keys::
+
+    mkdir -p /mnt/etc/cryptkey.d/
+    chmod 700 /mnt/etc/cryptkey.d/
+    dd bs=32 count=1 if=/dev/urandom of=/mnt/etc/cryptkey.d/rpool_$INST_UUID-${INST_ID}-key-zfs
+    dd bs=32 count=1 if=/dev/urandom of=/mnt/etc/cryptkey.d/bpool_$INST_UUID-key-luks
+    chmod u=r,go= /mnt/etc/cryptkey.d/*
+
+#. Backup boot pool::
+
+    zfs snapshot -r bpool_$INST_UUID/$INST_ID@pre-luks
+    zfs send -Rv bpool_$INST_UUID/$INST_ID@pre-luks > /mnt/root/bpool_$INST_UUID-${INST_ID}-pre-luks
+
+#. Unmount EFI partition::
+
+    for i in ${DISK}; do
+     umount /mnt/boot/efis/${i##*/}-part1
+    done
+
+#. Destroy boot pool::
+
+    zpool destroy bpool_$INST_UUID
+
+#. Create LUKS containers::
+
+    for i in ${DISK}; do
+     cryptsetup luksFormat -q --type luks1 --key-file /mnt/etc/cryptkey.d/bpool_$INST_UUID-key-luks $i-part2
+     echo $LUKS_PWD | cryptsetup luksAddKey --key-file /mnt/etc/cryptkey.d/bpool_$INST_UUID-key-luks $i-part2
+     cryptsetup open ${i}-part2 ${i##*/}-part2-luks-bpool_$INST_UUID --key-file /mnt/etc/cryptkey.d/bpool_$INST_UUID-key-luks
+    tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} <<EOF
+      boot.initrd.luks.devices = {
+        "${i##*/}-part2-luks-bpool_$INST_UUID" = {
+          device = "${i}-part2";
+          allowDiscards = true;
+          keyFile = "/etc/cryptkey.d/bpool_$INST_UUID-key-luks";
+        };
+      };
+    EOF
+    done
+
+   GRUB 2.06 still does not have complete support for LUKS2, LUKS1
+   is used instead.
+
+#. Embed key file in initrd::
+
+    tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} <<EOF
+      boot.initrd.secrets = {
+        "/etc/cryptkey.d/rpool_$INST_UUID-${INST_ID}-key-zfs" = "/etc/cryptkey.d/rpool_$INST_UUID-${INST_ID}-key-zfs";
+        "/etc/cryptkey.d/bpool_$INST_UUID-key-luks" = "/etc/cryptkey.d/bpool_$INST_UUID-key-luks";
+      };
+    EOF
+
+#. Recreate boot pool with mappers as vdev::
+
+    zpool create \
+        -d -o feature@async_destroy=enabled \
+        -o feature@bookmarks=enabled \
+        -o feature@embedded_data=enabled \
+        -o feature@empty_bpobj=enabled \
+        -o feature@enabled_txg=enabled \
+        -o feature@extensible_dataset=enabled \
+        -o feature@filesystem_limits=enabled \
+        -o feature@hole_birth=enabled \
+        -o feature@large_blocks=enabled \
+        -o feature@lz4_compress=enabled \
+        -o feature@spacemap_histogram=enabled \
+        -o ashift=12 \
+        -o autotrim=on \
+        -O acltype=posixacl \
+        -O canmount=off \
+        -O compression=lz4 \
+        -O devices=off \
+        -O normalization=formD \
+        -O relatime=on \
+        -O xattr=sa \
+        -O mountpoint=/boot \
+        -R /mnt \
+        bpool_$INST_UUID \
+        $INST_VDEV \
+        $(for i in ${DISK}; do
+           printf "/dev/mapper/${i##*/}-part2-luks-bpool_$INST_UUID ";
+          done)
+
+#. Restore boot pool backup::
+
+    zfs recv bpool_${INST_UUID}/${INST_ID} < /mnt/root/bpool_$INST_UUID-${INST_ID}-pre-luks
+    rm /mnt/root/bpool_$INST_UUID-${INST_ID}-pre-luks
+
+#. Mount boot dataset and EFI partitions::
+
+    zfs mount bpool_$INST_UUID/$INST_ID/BOOT/default
+
+    for i in ${DISK}; do
+     mount ${i}-part1 /mnt/boot/efis/${i##*/}-part1
+    done
+
+#. As keys are stored in initrd,
+   set secure permissions for ``/boot``::
+
+    chmod 700 /mnt/boot
+
+#. Change root pool password to key file::
+
+    mkdir -p /etc/cryptkey.d/
+    cp /mnt/etc/cryptkey.d/rpool_$INST_UUID-${INST_ID}-key-zfs /etc/cryptkey.d/rpool_$INST_UUID-${INST_ID}-key-zfs
+    zfs change-key -l \
+    -o keylocation=file:///etc/cryptkey.d/rpool_$INST_UUID-${INST_ID}-key-zfs \
+    -o keyformat=raw \
+    rpool_$INST_UUID/$INST_ID
+
+#. Import encrypted boot pool from ``/dev/mapper``::
+
+    tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} <<-'EOF'
+      systemd.services.zfs-import-bpool-mapper = {
+        wantedBy = [ "zfs-import.target" ];
+        description = "Import encrypted boot pool";
+        after = [ "cryptsetup.target" ];
+        before = [ "boot.mount" ];
+        serviceConfig = {
+          Type = "oneshot";
+          ExecStart = ''${pkgs.zfs}/bin/zpool import -aNd /dev/mapper'';
+        };
+      };
+    EOF
+
+#. Enable GRUB cryptodisk::
+
+    tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} <<EOF
+      boot.loader.grub.enableCryptodisk = true;
+    EOF
+
+#. **Important**: Back up root dataset key ``/etc/cryptkey.d/rpool_$INST_UUID-${INST_ID}-key-zfs``
+   to a secure location.
+
+   In the possible event of LUKS container corruption,
+   data on root set will only be available
+   with this key.
+
+Persistent swap and hibernation
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#. Optional: enable persistent swap partition. By default
+   encryption key of swap partition is discarded on reboot::
+
+    INST_SWAPKEY=/mnt/etc/cryptkey.d/${INST_PRIMARY_DISK##*/}-part4-key-luks-swap
+    INST_SWAPMAPPER=${INST_PRIMARY_DISK##*/}-part4-luks-swap
+
+    # fstab
+    # remove existing swap entries
+    for i in $DISK; do echo $i; done | grep -v ${INST_PRIMARY_DISK##*/} \
+    | while read j; do sed -i "\,$j-part4\"; randomEncryption.enable,d" /mnt/etc/nixos/${INST_CONFIG_FILE} ; done
+    sed -i "s|${INST_PRIMARY_DISK}-part4\"; randomEncryption.enable = true|/dev/mapper/${INST_SWAPMAPPER}\"|g" /mnt/etc/nixos/${INST_CONFIG_FILE}
+
+    # create key and format partition as LUKS container
+    dd bs=32 count=1 if=/dev/urandom of=${INST_SWAPKEY};
+    chmod u=r,go= /mnt/etc/cryptkey.d/*
+    cryptsetup luksFormat -q --type luks2 --key-file ${INST_SWAPKEY} ${INST_PRIMARY_DISK}-part4
+    cryptsetup luksOpen ${INST_PRIMARY_DISK}-part4 ${INST_SWAPMAPPER} --key-file ${INST_SWAPKEY}
+
+    # initialize swap space
+    mkswap /dev/mapper/${INST_SWAPMAPPER}
+
+    # add initrd key
+    tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} <<EOF
+      boot.initrd.secrets = {
+        "/etc/cryptkey.d/${INST_PRIMARY_DISK##*/}-part4-key-luks-swap" = "/etc/cryptkey.d/${INST_PRIMARY_DISK##*/}-part4-key-luks-swap";
+      };
+      boot.initrd.luks.devices = {
+        "${INST_SWAPMAPPER}" = {
+          device = "${INST_PRIMARY_DISK}-part4";
+          allowDiscards = true;
+          keyFile = "/etc/cryptkey.d/${INST_PRIMARY_DISK##*/}-part4-key-luks-swap";
+        };
+      };
+    EOF
+
+#. Optional: after enabling persistent swap partition,
+   enable hibernation::
+
+    tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} <<EOF
+      boot.resumeDevice = "/dev/mapper/${INST_SWAPMAPPER}";
+    EOF
+
+
+   Note that hibernation might not work with discrete graphics, virtio graphics or
+   AMD APU integrated graphics. This is not specific to this guide.
+
+   Computer must resume from a continuous swap space, resume
+   from multiple swap partitions is not supported.
+
+   Do not touch anything on disk while the computer is
+   in hibernation, see `kernel documentation
+   <https://www.kernel.org/doc/html/latest/power/swsusp.html>`__.

docs/Getting Started/NixOS/Root on ZFS/4-system-installation.rst

@@ -0,0 +1,176 @@
+.. highlight:: sh
+
+System Installation
+======================
+
+.. contents:: Table of Contents
+   :local:
+
+
+Additional configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+As NixOS configuration is declarative, post-installation tasks,
+such as user accounts and package selection, can all be done by
+specifing them in configuration. See `NixOS manual <https://nixos.org/nixos/manual/>`__
+for details.
+
+For timezone, hostname, networking, keyboard layout, etc,
+see ``/mnt/etc/nixos/configuration.nix``.
+
+Set root password
+-----------------
+
+This optional step is an example
+of declaratively configuring the system.
+
+#. Generate password hash::
+
+    mkpasswd -m SHA-512 -s
+    #output: $6$DeHnzc
+
+#. Declare `initialHashedPassword
+   <https://nixos.org/manual/nixos/stable/options.html#opt-users.users._name_.initialHashedPassword>`__
+   for root user::
+
+    tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} <<-'EOF'
+      users.users.root.initialHashedPassword = "$6$DeHnzc";
+    EOF
+
+System installation
+~~~~~~~~~~~~~~~~~~~
+
+#. Finalize the config file::
+
+    tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} <<EOF
+    }
+    EOF
+
+#. Take a snapshot of the clean installation, without state
+   for future use::
+
+    zfs snapshot -r rpool_$INST_UUID/$INST_ID@install_start
+    zfs snapshot -r bpool_$INST_UUID/$INST_ID@install_start
+
+#. Apply configuration
+
+   If root password hash is not set::
+
+    nixos-install -v --show-trace --root /mnt
+
+   You will be prompted for a new root password.
+
+   If password hash has been set::
+
+    nixos-install -v --show-trace --no-root-passwd --root /mnt
+
+
+Finish installation
+~~~~~~~~~~~~~~~~~~~~
+
+#. Take a snapshot of the clean installation for future use::
+
+    zfs snapshot -r rpool_$INST_UUID/$INST_ID@install
+    zfs snapshot -r bpool_$INST_UUID/$INST_ID@install
+
+#. Unmount EFI system partition::
+
+    umount /mnt/boot/efis/*
+
+#. Export pools::
+
+    zpool export bpool_$INST_UUID
+    zpool export rpool_$INST_UUID
+
+#. Reboot::
+
+    reboot
+
+Immutable root file system
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This section is optional.
+
+Often, programs generate mutable files in paths such as
+``/etc`` and ``/var/lib``. The generated files can be considered a
+part of the system state.
+
+This generated state is not declaratively managed
+by NixOS and can not be reproduced from NixOS configuration.
+
+To ensure that the system state is fully managed by NixOS and reproducible,
+we need to periodically purge the system state and let NixOS 
+regenerate root file system from scratch.
+
+Also see: `Erase your darlings: 
+immutable infrastructure for mutable systems <https://grahamc.com/blog/erase-your-darlings>`__.
+
+Save mutable data to alternative path
+-------------------------------------
+
+Before enabling purging on root dataset, we need to back up
+essential mutable data first, such as host SSH key and network connections.
+Below are some tips.
+
+- Some programs support specifying another
+  location for mutable data, such as
+  Wireguard::
+
+   networking.wireguard.interfaces.wg0.privateKeyFile = "/state/etc/wireguard/wg0";
+
+- For programs without a configurable data path,
+  `environment.etc <https://nixos.org/manual/nixos/stable/options.html#opt-environment.etc>`__
+  may be used::
+
+   environment.etc = { 
+     "ssh/ssh_host_rsa_key".source = "/state/etc/ssh/ssh_host_rsa_key";
+   }
+
+- systemd’s tmpfiles.d rules are also an option::
+
+   systemd.tmpfiles.rules = [
+     "L /var/lib/bluetooth - - - - /state/var/lib/bluetooth"
+   ];
+
+- Bind mount::
+
+   for i in {/etc/nixos,/etc/cryptkey.d}; do
+     mkdir -p /state/$i /$i
+     mount -o bind /state/$i /$i
+   done
+   nixos-generate-config --show-hardware-config
+
+Boot from empty root file system
+--------------------------------
+
+After backing up mutable data, you can try switching to
+an empty dataset as root file system.
+
+#. Check current root file system::
+
+    ROOT_FS=$(df --output=source /|tail -n1)
+    # rpool/ROOT/default
+
+#. Set empty file system as root::
+
+    sed -i "s,${ROOT_FS},${ROOT_FS%/*}/empty,g" /etc/nixos/hardware-configuration-zfs.nix
+
+#. Apply changes and reboot::
+
+    nixos-rebuild boot
+    reboot
+
+#. If everything went fine, add the output of the following command to configuration::
+
+    ROOT_FS=$(df --output=source /|tail -n1)
+    cat <<EOF
+      boot.initrd.postDeviceCommands = ''
+        zpool import -Nf ${ROOT_FS%%/*}
+        zfs rollback -r ${ROOT_FS%/*}/empty@start
+      '';
+    EOF
+
+#. Apply and reboot::
+
+    nixos-rebuild boot
+    reboot

docs/Getting Started/NixOS/Root on ZFS/6-recovery.rst

@@ -0,0 +1,145 @@
+.. highlight:: sh
+
+Recovery
+======================
+
+.. contents:: Table of Contents
+   :local:
+
+GRUB Tips
+-------------
+
+Boot from GRUB rescue
+~~~~~~~~~~~~~~~~~~~~~~~
+
+If bootloader file is damaged, it's still possible
+to boot computer with GRUB rescue image.
+
+This section is also applicable if you are in
+``grub rescue>``.
+
+#. On another computer, generate rescue image with::
+
+     pacman -S --needed mtools libisoburn grub
+     grub-install
+     grub-mkrescue -o grub-rescue.img
+     dd if=grub-rescue.img of=/dev/your-usb-stick
+
+   Boot computer from the rescue media.
+   Both legacy and EFI mode are supported.
+
+   Or `download generated GRUB rescue image <https://gitlab.com/m_zhou/bieaz/uploads/4a1b7cefb42723de6eb04f9dc485be3b/grub-rescue.img.7z>`__.
+
+#. List available disks with ``ls`` command::
+
+    grub> ls (hd # press tab
+    Possible devices are:
+
+     hd0 hd1 hd2 hd3
+
+#. List partitions by pressing tab key:
+
+   .. code-block:: text
+
+     grub> ls (hd0 # press tab
+     Possible partitions are:
+
+     Device hd0: No known filesystem detected - Sector size 512B - Total size  20971520KiB
+     Partition hd0,gpt1: Filesystem type fat - Label `EFI', UUID 0DF5-3A76 - Partition start at 1024KiB - Total size 1048576KiB
+     Partition hd0,gpt2: No known filesystem detected - Partition start at 1049600KiB - Total size 4194304KiB
+
+   - If boot pool is encrypted:
+
+     Unlock it with ``cryptomount``::
+
+         grub> insmod luks
+         grub> cryptomount hd0,gpt2
+         Attempting to decrypt master key...
+         Enter passphrase for hd0,gpt2 (af5a240e13e24483acf02600d61e0f36):
+         Slot 1 opened
+
+     Unlocked LUKS container is ``(crypto0)``:
+
+     .. code-block:: text
+
+         grub> ls (crypto0)
+         Device crypto0: Filesystem type zfs - Label `bpool_ip3tdb' - Last modification
+         time 2021-05-03 12:14:08 Monday, UUID f14d7bdf89fe21fb - Sector size 512B -
+         Total size 4192256KiB
+
+   - If boot pool is not encrypted:
+
+     .. code-block:: text
+
+       grub> ls (hd0,gpt2)
+       Device hd0,gpt2: Filesystem type zfs - Label `bpool_ip3tdb' - Last modification
+       time 2021-05-03 12:14:08 Monday, UUID f14d7bdf89fe21fb - Sector size 512B -
+       Total size 4192256KiB
+
+#. List boot environments nested inside ``bpool/$INST_ID/BOOT``::
+
+     grub> ls (crypto0)/sys/BOOT
+     @/ default/ be0/
+
+#. Instruct GRUB to load configuration from ``be0`` boot environment::
+
+     grub> prefix=(crypto0)/sys/BOOT/be0/@/grub
+     grub> configfile $prefix/grub.cfg
+
+#. GRUB menu should now appear.
+
+#. After entering system, reinstall GRUB.
+
+Switch GRUB prefix when disk fails
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you are using LUKS encrypted boot pool with multiple disks,
+the primary disk failed, GRUB will fail to load configuration.
+
+If there's still enough redundancy for the boot pool, try fix
+GRUB with the following method:
+
+#. Ensure ``Slot 1 opened`` message
+   is shown
+
+   .. code-block:: text
+
+      Welcome to GRUB!
+
+      error: no such cryptodisk found.
+      Attempting to decrypt master key...
+      Enter passphrase for hd0,gpt2 (c0987ea1a51049e9b3056622804de62a):
+      Slot 1 opened
+      error: disk `cryptouuid/47ed1b7eb0014bc9a70aede3d8714faf' not found.
+      Entering rescue mode...
+      grub rescue>
+
+   If ``error: access denied.`` is shown,
+   try re-enter password with::
+
+     grub rescue> cryptomount hd0,gpt2
+
+#. Check prefix::
+
+      grub rescue > set
+      # prefix=(cryptouuid/47ed1b7eb0014bc9a70aede3d8714faf)/sys/BOOT/be0@/grub
+      # root=cryptouuid/47ed1b7eb0014bc9a70aede3d8714faf
+
+#. Set correct ``prefix`` and ``root`` by replacing
+   ``cryptouuid/UUID`` with ``crypto0``::
+
+      grub rescue> prefix=(crypto0)/sys/BOOT/default@/grub
+      grub rescue> root=crypto0
+
+#. Boot GRUB::
+
+      grub rescue> insmod normal
+      grub rescue> normal
+
+   GRUB should then boot normally.
+
+#. After entering system, edit ``/etc/fstab`` to promote
+   one backup to ``/boot/efi``.
+
+#. Make the change to ``prefix`` and ``root``
+   permanent by `reinstalling GRUB <#grub-installation>`__.

docs/Getting Started/NixOS/index.rst

@@ -0,0 +1,64 @@
+.. highlight:: sh
+
+NixOS
+=====
+
+Contents
+--------
+.. toctree::
+  :maxdepth: 1
+  :glob:
+
+  *
+
+Support
+-------
+Reach out to the community using the :ref:`mailing_lists` or IRC at
+`#zfsonlinux <ircs://irc.libera.chat/#zfsonlinux>`__ on `Libera Chat
+<https://libera.chat/>`__.
+
+If you have a bug report or feature request
+related to this HOWTO, please `file a new issue and mention @ne9z
+<https://github.com/openzfs/openzfs-docs/issues/new?body=@ne9z,%20I%20have%20the%20following%20issue%20with%20the%20Nix%20ZFS%20HOWTO:>`__.
+
+Installation
+------------
+
+Note: this is for installing ZFS on an existing
+NixOS installation. To use ZFS as root file system,
+see below.
+
+Live image ships with ZFS support by default.
+
+#. Import separate configration file for ZFS options::
+
+    vim /etc/nixos/configuration.nix
+    ##add './zfs.nix' to 'imports'
+    # imports = [ ./zfs.nix ];
+
+#. Configure ZFS options::
+
+    tee -a /etc/nixos/zfs.nix <<EOF
+    { config, pkgs, ... }:
+
+    { boot.supportedFilesystems = [ "zfs" ];
+      networking.hostId = "$(head -c 8 /etc/machine-id)";
+    }
+    EOF
+
+#. Apply configuation changes::
+
+    nixos-rebuild switch
+
+Root on ZFS
+-----------
+ZFS can be used as root file system for NixOS.
+An installation guide is available.
+
+`Start here <Root%20on%20ZFS/0-overview.html>`__.
+
+.. toctree::
+  :maxdepth: 1
+  :glob:
+
+  Root on ZFS/*

docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS/0-overview.rst

@@ -63,12 +63,12 @@ Dataset layout
 |                           |                      |                      |                                     | datasets", "persistent datasets"; also    |
 |                           |                      |                      |                                     | include /var/lib, /srv, ...               |
 +---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
-|   bpool/sys/BOOT/default  | noauto               | legacy /boot         | no                                  | noauto is used to switch BE. because of   |
+|   bpool/sys/BOOT/default  | noauto               |        /boot         | no                                  | noauto is used to switch BE. because of   |
 |                           |                      |                      |                                     | noauto, must use fstab to mount           |
 +---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
 |   rpool/sys/ROOT/default  | noauto               | /                    | no                                  | mounted by initrd zfs hook                |
 +---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
-|   bpool/sys/BOOT/be1      | noauto               | legacy /boot         | no                                  | see bpool/sys/BOOT/default                |
+|   bpool/sys/BOOT/be1      | noauto               |        /boot         | no                                  | see bpool/sys/BOOT/default                |
 +---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
 |   rpool/sys/ROOT/be1      | noauto               | /                    | no                                  | see rpool/sys/ROOT/default                |
 +---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
@@ -110,3 +110,32 @@ Encryption
   which should be sufficient for most purposes.
 
   Secure Boot is not supported out-of-the-box due to ZFS module.
+
+Booting with disk failure
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This guide is written with disk failure in mind.
+
+If disks used in Root on ZFS pool failed, but
+sufficient redundancy for both root pool and boot pool
+still exists, the system will still boot normally.
+
+Swap partition on the failed disk will fail to mount,
+after an 1m30s timeout.
+
+This feature is useful for use cases such
+as an unattended remote server.
+
+Example:
+
+ - System has disks ``n>1``
+
+ - Installed with mirrored setup
+
+ - Mirrored setup can tolerate up to ``n-1`` disk failures
+
+ - Disconnect one or more disks, keep at least
+   one disk connected
+
+ - System still boots, but fails to mount swap and
+   EFI partition

docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS/1-preparation.rst

@@ -77,7 +77,7 @@ Preparation
     ls /dev/disk/by-id/*
 
    If using virtio as disk bus, use
-   ``/dev/disk/by-path/*`` or ``/dev/vd*``.
+   ``/dev/disk/by-path/*``.
 
    Declare disk array::
 

docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS/2-system-installation.rst

@@ -188,12 +188,11 @@ System Installation
     zfs create -o canmount=off -o mountpoint=none bpool_$INST_UUID/$INST_ID/BOOT
     zfs create -o canmount=off -o mountpoint=none rpool_$INST_UUID/$INST_ID/ROOT
     zfs create -o canmount=off -o mountpoint=none rpool_$INST_UUID/$INST_ID/DATA
-    zfs create -o mountpoint=legacy -o canmount=noauto bpool_$INST_UUID/$INST_ID/BOOT/default
+    zfs create -o mountpoint=/boot -o canmount=noauto bpool_$INST_UUID/$INST_ID/BOOT/default
     zfs create -o mountpoint=/ -o canmount=off    rpool_$INST_UUID/$INST_ID/DATA/default
     zfs create -o mountpoint=/ -o canmount=noauto rpool_$INST_UUID/$INST_ID/ROOT/default
     zfs mount rpool_$INST_UUID/$INST_ID/ROOT/default
-    mkdir /mnt/boot
+    zfs mount bpool_$INST_UUID/$INST_ID/BOOT/default
-    mount -t zfs bpool_$INST_UUID/$INST_ID/BOOT/default /mnt/boot
     for i in {usr,var,var/lib};
     do
         zfs create -o canmount=off rpool_$INST_UUID/$INST_ID/DATA/default/$i

docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS/3-system-configuration.rst

@@ -6,21 +6,9 @@ System Configuration
 .. contents:: Table of Contents
    :local:
 
-#. Generate list of datasets for `zfs-mount-generator
-   <https://manpages.ubuntu.com/manpages/focal/man8/zfs-mount-generator.8.html>`__ to mount them at boot::
-
-    # tab-separated zfs properties
-    # see /etc/zfs/zed.d/history_event-zfs-list-cacher.sh
-    export \
-    PROPS="name,mountpoint,canmount,atime,relatime,devices,exec\
-    ,readonly,setuid,nbmand,encroot,keylocation"
-    mkdir -p /mnt/etc/zfs/zfs-list.cache
-    zfs list -H -t filesystem -o $PROPS -r rpool_$INST_UUID > /mnt/etc/zfs/zfs-list.cache/rpool_$INST_UUID
-    sed -Ei "s|/mnt/?|/|" /mnt/etc/zfs/zfs-list.cache/*
-
 #. Generate fstab::
 
-    echo bpool_$INST_UUID/$INST_ID/BOOT/default /boot zfs rw,xattr,posixacl 0 0 >> /mnt/etc/fstab
+    genfstab -U /mnt | sed 's;zfs[[:space:]]*;zfs zfsutil,;g' | grep "zfs zfsutil" >> /mnt/etc/fstab
     for i in ${DISK}; do
        echo UUID=$(blkid -s UUID -o value ${i}-part1) /boot/efis/${i##*/}-part1 vfat \
        x-systemd.idle-timeout=1min,x-systemd.automount,noauto,umask=0022,fmask=0022,dmask=0022 0 1 >> /mnt/etc/fstab
@@ -73,7 +61,7 @@ System Configuration
     systemctl enable zfs-import-scan.service zfs-import.target zfs-zed zfs.target --root=/mnt
     systemctl disable zfs-mount --root=/mnt
 
-   At boot, datasets on rpool are mounted with ``zfs-mount-generator``,
+   At boot, datasets on rpool are mounted with ``/etc/fstab``,
    which can control the mounting process more precisely than ``zfs-mount.service``.
 
 #. By default SSH server is enabled, allowing root login by password,

docs/Getting Started/index.rst

@@ -16,7 +16,7 @@ documentation <https://pthree.org/2012/04/17/install-zfs-on-debian-gnulinux/>`__
    Fedora/index
    FreeBSD
    Gentoo <https://wiki.gentoo.org/wiki/ZFS>
-   NixOS <https://nixos.wiki/wiki/NixOS_on_ZFS>
+   NixOS/index
    openSUSE/index
    RHEL-based distro/index
    Ubuntu/index