jacobpascual/openzfs-docs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Updates and cleanup for Arch, RHEL, NixOS and Fedora

Browse Source 
Signed-off-by: Maurice Zhou <ja@apvc.uk>
This commit is contained in:
Maurice Zhou
2022-07-22 17:14:14 +02:00
committed by George Melikov
parent 5777295f0a
commit 2766cb7197
43 changed files with 937 additions and 4474 deletions
Download Patch File Download Diff File Expand all files Collapse all files

49
docs/Getting Started/Arch Linux/3-live.rst
View File

@@ -1,49 +0,0 @@
.. highlight:: sh
Live image
============
Latest live image might contain a kernel incompatible with
ZFS. Check the compatibility with the following procedure.
#. Choose a mirror::
https://archlinux.org/mirrorlist/all/
https://gitea.artixlinux.org/packagesA/artix-mirrorlist/src/branch/master/trunk/mirrorlist
#. Check the build date of the
latest Arch Linux live image::
https://mirrors.dotsrc.org/archlinux/iso/latest/
https://mirrors.dotsrc.org/artix-linux/iso/
# archlinux-2021.01.01-x86_64.iso
#. Check the kernel version of the live image::
https://archive.archlinux.org/repos/2021/01/01/core/os/x86_64
https://archive.artixlinux.org/repos/2021/01/01/system/os/x86_64
# linux-5.10.3.arch1-1-x86_64.pkg.tar.zst
#. Check latest zfs-dkms package version::
https://archzfs.com/archzfs/x86_64/
# zfs-dkms-2.0.1-1-x86_64.pkg.tar.zst
# zfs-linux-2.0.1_5.10.10.arch1.1-1-x86_64.pkg.tar.zst
#. Visit OpenZFS release page::
curl -L https://github.com/openzfs/zfs/raw/zfs-2.0.1/META \
| grep Linux
# Linux-Maximum: 5.10
# Linux-Minimum: 3.10
- If compatible, download the latest live image::
https://mirrors.dotsrc.org/archlinux/iso/latest/archlinux-2021.01.01-x86_64.iso
https://mirrors.dotsrc.org/artix-linux/iso/artix-base-openrc-20210101-x86_64.iso
- If not compatible, use an older live image and verify that it contains
a supported kernel using the above method::
https://mirrors.dotsrc.org/archlinux/iso/
https://iso.artixlinux.org/archived-isos.php

2
docs/Getting Started/Arch Linux/Arch Linux Root on ZFS.rst
View File

@@ -1,6 +1,6 @@
Arch Linux Root on ZFS
======================
`Start here <Root%20on%20ZFS/0-overview.html>`__.
`Start here <Root%20on%20ZFS/1-preparation.html>`__.
Contents
--------

151
docs/Getting Started/Arch Linux/Root on ZFS/0-overview.rst
View File

@@ -1,151 +0,0 @@
.. highlight:: sh
Overview
======================
This document describes how to install Arch Linux with ZFS as root
file system.
Caution
~~~~~~~
- This guide wipes entire physical disks. Back up existing data.
- `GRUB does not and
will not work on 4Kn drive with legacy (BIOS) booting.
<http://savannah.gnu.org/bugs/?46700>`__
Partition layout
~~~~~~~~~~~~~~~~
GUID partition table (GPT) is used.
EFI system partition will be referred to as **ESP** in this document.
+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+
| Name | legacy boot | ESP | Boot pool | swap | root pool | remaining space |
+======================+======================+=======================+======================+=====================+=======================+=================+
| File system | | vfat | ZFS | swap | ZFS | |
+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+
| Size | 1M | 4G, or 1G w/o ISO | 4G | depends on RAM size | | |
+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+
| Optional encryption | | *Secure Boot* | luks 1 | plain dm-crypt or | ZFS native encryption | |
| | | | | luks2 | | |
+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+
| Partition no. | 5 | 1 | 2 | 4 | 3 | |
+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+
| Mount point | | /boot/efi | /boot | | / | |
| | | /boot/efis/disk-part1 | | | | |
+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+
Dataset layout
~~~~~~~~~~~~~~
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| Dataset | canmount | mountpoint | container | notes |
+===========================+======================+======================+=====================================+===========================================+
| bpool | off | /boot | contains sys | |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool | off | / | contains sys | |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| bpool/sys | off | none | contains BOOT | |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool/sys | off | none | contains ROOT | sys is encryptionroot |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| bpool/sys/BOOT | off | none | contains boot environments | |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool/sys/ROOT | off | none | contains boot environments | |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool/sys/DATA | off | none | contains placeholder "default" | |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool/sys/DATA/default | off | / | contains user datasets | child datsets inherits mountpoint |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool/sys/DATA/default/ | on | /home (inherited) | no | |
| home | | | | user datasets, also called "shared |
| | | | | datasets", "persistent datasets"; also |
| | | | | include /var/lib, /srv, ... |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| bpool/sys/BOOT/default | noauto | /boot | no | noauto is used to switch BE. because of |
| | | | | noauto, must use fstab to mount |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool/sys/ROOT/default | noauto | / | no | mounted by initrd zfs hook |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| bpool/sys/BOOT/be1 | noauto | /boot | no | see bpool/sys/BOOT/default |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool/sys/ROOT/be1 | noauto | / | no | see rpool/sys/ROOT/default |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
Encryption
~~~~~~~~~~
- Swap
Swap is always encrypted. By default, swap is encrypted
with plain dm-crypt with key generated from ``/dev/urandom``
at every boot. Swap content does not persist between reboots.
LUKS2-encrypted persistent swap can be
enabled after encrypting both boot pool and root pool, see below.
With persistent swap, hibernation (suspend-to-disk) can be enabled.
- Root pool
ZFS native encryption can be optionally enabled for ``rpool/sys``
and child datasets.
User should be aware that, ZFS native encryption does not
encrypt some metadata of the datasets.
ZFS native encryption also does not change master key when ``zfs change-key`` is invoked.
Therefore, you should wipe the disk when password is compromised to protect confidentiality.
See `zfs-load-key.8 <https://openzfs.github.io/openzfs-docs/man/8/zfs-load-key.8.html>`__
and `zfs-change-key.8 <https://openzfs.github.io/openzfs-docs/man/8/zfs-change-key.8.html>`__
for more information regarding ZFS native encryption.
Encryption is enabled at dataset creation and can not be disabled later.
Password can be supplied via SSH.
- Boot pool
After encrypting root pool, boot pool can also be encrypted with LUKS1.
This protects initrd from attacks and also protects key material in initrd.
Password must be interactively entered at boot in GRUB. This disables
password with SSH.
- Bootloader
Bootloader can not be encrypted.
However, with Secure Boot, bootloader
can be verified by motherboard firmware to be untempered,
which should be sufficient for most purposes.
As enabling Secure Boot is device specific, this is not
covered in detail.
Booting with disk failure
~~~~~~~~~~~~~~~~~~~~~~~~~
This guide is written with disk failure in mind.
If disks used in Root on ZFS pool failed, but
sufficient redundancy for both root pool and boot pool
still exists, the system will still boot normally.
Swap partition on the failed disk will fail to mount,
after an 1m30s timeout.
This feature is useful for use cases such
as an unattended remote server.
Example:
- System has disks ``n>1``
- Installed with mirrored setup
- Mirrored setup can tolerate up to ``n-1`` disk failures
- Disconnect one or more disks, keep at least
one disk connected
- System still boots, but fails to mount swap and
EFI partition

179
docs/Getting Started/Arch Linux/Root on ZFS/1-preparation.rst
View File

@@ -6,97 +6,29 @@ Preparation
.. contents:: Table of Contents
:local:
#. Download live ISO compatible with ZFS:
#. Check zfs-dkms version
Visit `archzfs repo <https://archzfs.com/archzfs/x86_64/>`__,
search for ``zfs-dkms`` in page::
zfs-dkms-2.1.2-1-x86_64.pkg.tar.zst
Ignore unstable variants such as zfs-dkms-git or zfs-dkms-rc.
In this case, version number is 2.1.2, between two ``-``.
#. Check supported kernel version. Plug the version number into the
following link::
https://github.com/openzfs/zfs/blob/zfs-2.1.2/META
Visit the page, the supported max kernel version is::
Linux-Maximum: 5.15
#. Visit `Arch Linux releases page
<https://archlinux.org/releng/releases/>`__ and download a
supported version. Note that minor kernel releases does not
affect ZFS compatibility, i. e., both ``5.15.5`` and ``5.15.15``
are supported by ZFS with ``Linux-Maximum: 5.15``.
#. Follow `installation guide on Arch wiki <https://wiki.archlinux.org/title/Installation_guide>`__
up to **Update the system clock**.
#. Disable Secure Boot. ZFS modules can not be loaded if Secure Boot is enabled.
#. Download latest `Arch Linux live image <https://archlinux.org/download/>`__ and boot from it.
#. Connect to the Internet.
#. Set root password or ``/root/.ssh/authorized_keys``.
#. Start SSH server::
systemctl start sshd
systemctl restart sshd
#. Connect from another computer::
ssh root@192.168.1.19
and, most important, enter a bash shell::
#. Use bash shell. Other shell not tested::
bash
This guide is untested with the default shell ``zsh`` in live environment.
#. Expand live root filesystem::
mount -o remount,size=2G /run/archiso/cowspace/
#. `Add archzfs repo <../0-archzfs-repo.html>`__.
#. `Install zfs-dkms in live environment <../2-zfs-dkms.html#installation>`__.
#. Load zfs kernel module::
modprobe zfs
#. Kernel variant
Store the kernel variant in a variable.
Available variants in official repo are:
- linux
- linux-lts
- linux-zen
- linux-hardened
::
INST_LINVAR='linux'
``linux-hardened`` does not support hibernation.
#. Unique pool suffix. ZFS expects pool names to be
unique, therefore it's recommended to create
pools with a unique suffix::
INST_UUID=$(dd if=/dev/urandom bs=1 count=100 2>/dev/null | tr -dc 'a-z0-9' | cut -c-6)
#. Identify this installation in ZFS filesystem path::
INST_ID=arch
#. Target disk
List available disks with::
ls /dev/disk/by-id/*
If using virtio as disk bus, use
``/dev/disk/by-path/*``.
If using virtio as disk bus, use ``/dev/disk/by-path/*``.
Declare disk array::
@@ -106,64 +38,8 @@ Preparation
DISK='/dev/disk/by-id/disk1'
#. Choose a primary disk. This disk will be used
for primary EFI partition and hibernation, default to
first disk in the array::
INST_PRIMARY_DISK=$(echo $DISK | cut -f1 -d\ )
If disk path contains colon ``:``, this disk
can not be used for hibernation. ``encrypt`` mkinitcpio
hook treats ``:`` as argument separator without a means to
escape this character.
#. Set vdev topology, possible values are:
- (not set, single disk or striped; no redundancy)
- mirror
- raidz1
- raidz2
- raidz3
::
INST_VDEV=
This will create a single vdev with the topology of your choice.
It is also possible to manually create a pool with multiple vdevs, such as::
zpool create --options \
poolName \
mirror sda sdb \
raidz2 sdc ... \
raidz3 sde ... \
spare sdf ...
Notice the cost of parity when using RAID-Z. See
`here <https://www.delphix.com/blog/delphix-engineering/zfs-raidz-stripe-width-or-how-i-learned-stop-worrying-and-love-raidz>`__
and `here <https://docs.google.com/spreadsheets/d/1tf4qx1aMJp8Lo_R6gpT689wTjHv6CGVElrPqTA0w_ZY/>`__.
For boot pool, which must be readable by GRUB, mirrored vdev should always be used for maximum redundancy.
This guide will use mirrored bpool for multi-disk setup.
Refer to `zpoolconcepts <https://openzfs.github.io/openzfs-docs/man/7/zpoolconcepts.7.html>`__
and `zpool-create <https://openzfs.github.io/openzfs-docs/man/8/zpool-create.8.html>`__
man pages for details.
#. Set partition size:
Set ESP size. ESP contains Live ISO for recovery,
as described in `optional configurations <4-optional-configuration.html>`__::
INST_PARTSIZE_ESP=4 # in GB
#INST_PARTSIZE_ESP=1 # if local recovery is not needed
Set boot pool size. To avoid running out of space while using
boot environments, the minimum is 4GB. Adjust the size if you
intend to use multiple kernel/distros::
INST_PARTSIZE_BPOOL=4
Set swap size. It's `recommended <https://chrisdown.name/2018/01/02/in-defence-of-swap.html>`__
to setup a swap partition. If you intend to use hibernation,
the minimum should be no less than RAM size. Skip if swap is not needed::
@@ -173,3 +49,46 @@ Preparation
Root pool size, use all remaining disk space if not set::
INST_PARTSIZE_RPOOL=
#. Check kernel version::
uname -r
#5.18.7-arch1-1
#. Add ZFS repo::
curl -L https://archzfs.com/archzfs.gpg | pacman-key -a -
pacman-key --lsign-key $(curl -L https://git.io/JsfVS)
curl -L https://git.io/Jsfw2 > /etc/pacman.d/mirrorlist-archzfs
tee -a /etc/pacman.conf <<- 'EOF'
#[archzfs-testing]
#Include = /etc/pacman.d/mirrorlist-archzfs
[archzfs]
Include = /etc/pacman.d/mirrorlist-archzfs
EOF
#. Find a ZFS package compatible with the kernel:
Search kernel version string (e.g. 5.18.7) in both pages:
* https://archzfs.com/archive_archzfs/
* https://archzfs.com/archzfs/x86_64/
Result: https/.../archive_archzfs/zfs-linux-2.1.5_5.18.7.arch1.1-1-x86_64.pkg.tar.zst
#. Find compatible zfs-utils package:
Search ZFS version string (e.g. 2.1.5) in both pages above.
Result: https/.../archzfs/x86_64/zfs-utils-2.1.5-2-x86_64.pkg.tar.zst
#. Download both then install::
pacman -U link-to-zfs.zst link-to-utils.zst
#. Load kernel modules::
modprobe zfs

205
docs/Getting Started/Arch Linux/Root on ZFS/2-system-installation.rst
View File

@@ -6,46 +6,29 @@ System Installation
.. contents:: Table of Contents
:local:
#. Optional: wipe solid-state drives with the generic tool
`blkdiscard <https://utcc.utoronto.ca/~cks/space/blog/linux/ErasingSSDsWithBlkdiscard>`__,
to clean previous partition tables and improve performance.
All content will be irrevocably destroyed::
#. Partition the disks::
for i in ${DISK}; do
blkdiscard -f $i &
done
wait
This is a quick operation and should be completed under one
minute.
For other device specific methods, see
`Memory cell clearing <https://wiki.archlinux.org/title/Solid_state_drive/Memory_cell_clearing>`__
#. Partition the disks.
See `Overview <0-overview.html>`__ for details::
for i in ${DISK}; do
sgdisk --zap-all $i
sgdisk -n1:1M:+${INST_PARTSIZE_ESP}G -t1:EF00 $i
sgdisk -n2:0:+${INST_PARTSIZE_BPOOL}G -t2:BE00 $i
if [ "${INST_PARTSIZE_SWAP}" != "" ]; then
sgdisk -n4:0:+${INST_PARTSIZE_SWAP}G -t4:8200 $i
fi
if [ "${INST_PARTSIZE_RPOOL}" = "" ]; then
sgdisk -n1:1M:+1G -t1:EF00 $i
sgdisk -n2:0:+4G -t2:BE00 $i
test -z $INST_PARTSIZE_SWAP || sgdisk -n4:0:+${INST_PARTSIZE_SWAP}G -t4:8200 $i
if test -z $INST_PARTSIZE_RPOOL; then
sgdisk -n3:0:0 -t3:BF00 $i
else
sgdisk -n3:0:+${INST_PARTSIZE_RPOOL}G -t3:BF00 $i
fi
sgdisk -a1 -n5:24K:+1000K -t5:EF02 $i
done
#. Create boot pool::
disk_num=0; for i in $DISK; do disk_num=$(( $disk_num + 1 )); done
if [ $disk_num -gt 1 ]; then INST_VDEV_BPOOL=mirror; fi
zpool create \
-o compatibility=grub2 \
-o ashift=12 \
@@ -59,12 +42,14 @@ System Installation
-O xattr=sa \
-O mountpoint=/boot \
-R /mnt \
bpool_$INST_UUID \
$INST_VDEV_BPOOL \
bpool \
mirror \
$(for i in ${DISK}; do
printf "$i-part2 ";
done)
If not using a multi-disk setup, remove ``mirror``.
You should not need to customize any of the options for the boot pool.
GRUB does not support all of the zpool features. See ``spa_feature_names``
@@ -91,101 +76,49 @@ System Installation
-O relatime=on \
-O xattr=sa \
-O mountpoint=/ \
rpool_$INST_UUID \
$INST_VDEV \
rpool \
mirror \
$(for i in ${DISK}; do
printf "$i-part3 ";
done)
**Notes:**
If not using a multi-disk setup, remove ``mirror``.
- The use of ``ashift=12`` is recommended here because many drives
today have 4 KiB (or larger) physical sectors, even though they
present 512 B logical sectors. Also, a future replacement drive may
have 4 KiB physical sectors (in which case ``ashift=12`` is desirable)
or 4 KiB logical sectors (in which case ``ashift=12`` is required).
- Setting ``-O acltype=posixacl`` enables POSIX ACLs globally. If you
do not want this, remove that option, but later add
``-o acltype=posixacl`` (note: lowercase “o”) to the ``zfs create``
for ``/var/log``, as `journald requires ACLs
<https://askubuntu.com/questions/970886/journalctl-says-failed-to-search-journal-acl-operation-not-supported>`__
- Setting ``normalization=formD`` eliminates some corner cases relating
to UTF-8 filename normalization. It also implies ``utf8only=on``,
which means that only UTF-8 filenames are allowed. If you care to
support non-UTF-8 filenames, do not use this option. For a discussion
of why requiring UTF-8 filenames may be a bad idea, see `The problems
with enforced UTF-8 only filenames
<http://utcc.utoronto.ca/~cks/space/blog/linux/ForcedUTF8Filenames>`__.
- ``recordsize`` is unset (leaving it at the default of 128 KiB). If you
want to tune it (e.g. ``-o recordsize=1M``), see `these
<https://jrs-s.net/2019/04/03/on-zfs-recordsize/>`__ `various
<http://blog.programster.org/zfs-record-size>`__ `blog
<https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSFileRecordsizeGrowth>`__
`posts
<https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSRecordsizeAndCompression>`__.
- Setting ``relatime=on`` is a middle ground between classic POSIX
``atime`` behavior (with its significant performance impact) and
``atime=off`` (which provides the best performance by completely
disabling atime updates). Since Linux 2.6.30, ``relatime`` has been
the default for other filesystems. See `RedHat’s documentation
<https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/power_management_guide/relatime>`__
for further information.
- Setting ``xattr=sa`` `vastly improves the performance of extended
attributes
<https://github.com/zfsonlinux/zfs/commit/82a37189aac955c81a59a5ecc3400475adb56355>`__.
Inside ZFS, extended attributes are used to implement POSIX ACLs.
Extended attributes can also be used by user-space applications.
`They are used by some desktop GUI applications.
<https://en.wikipedia.org/wiki/Extended_file_attributes#Linux>`__
`They can be used by Samba to store Windows ACLs and DOS attributes;
they are required for a Samba Active Directory domain controller.
<https://wiki.samba.org/index.php/Setting_up_a_Share_Using_Windows_ACLs>`__
Note that ``xattr=sa`` is `Linux-specific
<https://openzfs.org/wiki/Platform_code_differences>`__. If you move your
``xattr=sa`` pool to another OpenZFS implementation besides ZFS-on-Linux,
extended attributes will not be readable (though your data will be). If
portability of extended attributes is important to you, omit the
``-O xattr=sa`` above. Even if you do not want ``xattr=sa`` for the whole
pool, it is probably fine to use it for ``/var/log``.
- Make sure to include the ``-part3`` portion of the drive path. If you
forget that, you are specifying the whole disk, which ZFS will then
re-partition, and you will lose the bootloader partition(s).
#. This section implements dataset layout as described in `overview <0-overview.html>`__.
#. This section implements dataset layout as described in `overview <1-preparation.html>`__.
Create root system container:
- Unencrypted::
zfs create -o canmount=off -o mountpoint=none rpool_$INST_UUID/$INST_ID
zfs create \
-o canmount=off \
-o mountpoint=none \
rpool/archlinux
- Encrypted:
Pick a strong password. Once compromised, changing password will not keep your
data safe. See ``zfs-change-key(8)`` for more info::
zfs create -o canmount=off -o mountpoint=none -o encryption=on -o keylocation=prompt -o keyformat=passphrase rpool_$INST_UUID/$INST_ID
zfs create \
-o canmount=off \
-o mountpoint=none \
-o encryption=on \
-o keylocation=prompt \
-o keyformat=passphrase \
rpool/archlinux
Create other system datasets::
Create system datasets::
zfs create -o canmount=off -o mountpoint=none bpool_$INST_UUID/$INST_ID
zfs create -o canmount=off -o mountpoint=none bpool_$INST_UUID/$INST_ID/BOOT
zfs create -o canmount=off -o mountpoint=none rpool_$INST_UUID/$INST_ID/ROOT
zfs create -o canmount=off -o mountpoint=none rpool_$INST_UUID/$INST_ID/DATA
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$INST_UUID/$INST_ID/BOOT/default
zfs create -o mountpoint=/ -o canmount=off rpool_$INST_UUID/$INST_ID/DATA/default
zfs create -o mountpoint=/ -o canmount=noauto rpool_$INST_UUID/$INST_ID/ROOT/default
zfs mount rpool_$INST_UUID/$INST_ID/ROOT/default
zfs mount bpool_$INST_UUID/$INST_ID/BOOT/default
for i in {usr,var,var/lib};
do
zfs create -o canmount=off rpool_$INST_UUID/$INST_ID/DATA/default/$i
done
for i in {home,root,srv,usr/local,var/log,var/spool};
do
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/$i
done
chmod 750 /mnt/root
zfs create -o canmount=on -o mountpoint=/ rpool/archlinux/root
zfs create -o canmount=on -o mountpoint=/home rpool/archlinux/home
zfs create -o canmount=off -o mountpoint=/var rpool/archlinux/var
zfs create -o canmount=on rpool/archlinux/var/lib
zfs create -o canmount=on rpool/archlinux/var/log
Create boot dataset::
zfs create -o canmount=on -o mountpoint=/boot bpool/archlinux
#. Format and mount ESP::
@@ -196,65 +129,15 @@ System Installation
done
mkdir -p /mnt/boot/efi
mount -t vfat ${INST_PRIMARY_DISK}-part1 /mnt/boot/efi
mount -t vfat $(echo $DISK | cut -f1 -d\ )-part1 /mnt/boot/efi
#. Create separate user dataset at ``/home/User``, dateset name can be
changed later::
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/home/User
If needed, snapshot, rollback and other related permissions can be
delegated to the user later.
#. Create optional program data datasets to omit data from rollback::
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/games
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/www
# for GNOME
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/AccountsService
# for Docker
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/docker
# for NFS
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/nfs
# for LXC
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/lxc
# for LibVirt
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/libvirt
##other application
# zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/$name
Add other datasets when needed, such as PostgreSQL.
#. Install base packages::
#. Install packages::
pacstrap /mnt base vi mandoc grub efibootmgr mkinitcpio
#. Check compatible kernel version::
CompatibleVer=$(pacman -Si zfs-linux | grep 'Depends On' | sed "s|.*linux=||" | awk '{ print $1 }')
pacstrap -U /mnt https://archive.archlinux.org/packages/l/linux/linux-${CompatibleVer}-x86_64.pkg.tar.zst
INST_LINVER=$(pacman -Si zfs-${INST_LINVAR} \
| grep 'Depends On' \
| sed "s|.*${INST_LINVAR}=||" \
| awk '{ print $1 }')
#. Install kernel. Download from archive if kernel is not available::
if [ ${INST_LINVER} = \
$(pacman -Si ${INST_LINVAR} | grep Version | awk '{ print $3 }') ]; then
pacstrap /mnt ${INST_LINVAR}
else
pacstrap -U /mnt \
https://archive.archlinux.org/packages/l/${INST_LINVAR}/${INST_LINVAR}-${INST_LINVER}-x86_64.pkg.tar.zst
fi
Ignore ``error: command failed to execute correctly``.
#. Install archzfs package::
pacstrap /mnt zfs-$INST_LINVAR zfs-utils
#. Install firmware::
pacstrap /mnt zfs-linux zfs-utils
pacstrap /mnt linux-firmware intel-ucode amd-ucode
#. For other optional packages,
see `ArchWiki <https://wiki.archlinux.org/index.php/Installation_guide#Installation>`__.

108
docs/Getting Started/Arch Linux/Root on ZFS/3-system-configuration.rst
View File

@@ -6,31 +6,15 @@ System Configuration
.. contents:: Table of Contents
:local:
#. Set `mkinitcpio zfs hook scan path
<https://github.com/archzfs/archzfs/blob/master/src/zfs-utils/zfs-utils.initcpio.install>`__::
echo GRUB_CMDLINE_LINUX=\"zfs_import_dir=${INST_PRIMARY_DISK%/*}\" >> /mnt/etc/default/grub
#. Generate fstab::
genfstab -U /mnt | sed 's;zfs[[:space:]]*;zfs zfsutil,;g' | grep "zfs zfsutil" >> /mnt/etc/fstab
mkdir -p /mnt/etc/
for i in ${DISK}; do
echo UUID=$(blkid -s UUID -o value ${i}-part1) /boot/efis/${i##*/}-part1 vfat \
x-systemd.idle-timeout=1min,x-systemd.automount,noauto,umask=0022,fmask=0022,dmask=0022 0 1 >> /mnt/etc/fstab
umask=0022,fmask=0022,dmask=0022 0 1 >> /mnt/etc/fstab
done
echo UUID=$(blkid -s UUID -o value ${INST_PRIMARY_DISK}-part1) /boot/efi vfat \
x-systemd.idle-timeout=1min,x-systemd.automount,noauto,umask=0022,fmask=0022,dmask=0022 0 1 >> /mnt/etc/fstab
if [ "${INST_PARTSIZE_SWAP}" != "" ]; then
for i in ${DISK}; do
echo ${i##*/}-part4-swap ${i}-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256,discard >> /mnt/etc/crypttab
echo /dev/mapper/${i##*/}-part4-swap none swap defaults 0 0 >> /mnt/etc/fstab
done
fi
By default, systemd will halt boot process if any entry in ``/etc/fstab`` fails
to mount. This is unnecessary for mirrored EFI boot partitions.
With the above mount options, systemd will skip mounting them at boot,
only mount them on demand when accessed.
echo $(echo $DISK | cut -f1 -d\ )-part1 /boot/efi vfat \
noauto,umask=0022,fmask=0022,dmask=0022 0 1 >> /mnt/etc/fstab
#. Configure mkinitcpio::
@@ -39,88 +23,46 @@ System Configuration
HOOKS=(base udev autodetect modconf block keyboard zfs filesystems)
EOF
For more information on mkinitcpio configuration,
such as support for other keyboard layouts, see
`wiki article <https://wiki.archlinux.org/title/mkinitcpio>`__.
#. Enable DHCP on all ethernet ports::
tee /mnt/etc/systemd/network/20-default.network <<EOF
[Match]
Name=en*
Name=eth*
[Network]
DHCP=yes
EOF
systemctl enable systemd-networkd systemd-resolved --root=/mnt
Customize this file if the system is not using wired DHCP network.
See `Network Configuration <https://wiki.archlinux.org/index.php/Network_configuration>`__.
Alternatively, install a network manager such as
``NetworkManager``.
#. Enable internet time sync::
#. Enable internet time synchronisation::
hwclock --systohc
systemctl enable systemd-timesyncd --root=/mnt
#. Interactively set locale, keymap, timezone, hostname and root password::
#. Set locale, keymap, timezone, hostname and root password::
rm -f /mnt/etc/localtime
systemd-firstboot --root=/mnt --force --prompt --root-password=PASSWORD
This can be non-interactive, see man page for details::
rm -f /mnt/etc/localtime
systemd-firstboot --root=/mnt --force \
--locale="en_US.UTF-8" --locale-messages="en_US.UTF-8" \
--keymap=us --timezone="Europe/Berlin" --hostname=myHost \
--root-password=PASSWORD --root-shell=/bin/bash
``systemd-firstboot`` has bugs for setting root password, reset it here::
arch-chroot /mnt passwd
systemd-firstboot --root=/mnt --prompt --force
#. Generate host id::
zgenhostid -f -o /mnt/etc/hostid
#. Ignore kernel updates::
sed -i 's/#IgnorePkg/IgnorePkg/' /mnt/etc/pacman.conf
sed -i "/^IgnorePkg/ s/$/ ${INST_LINVAR} ${INST_LINVAR}-headers zfs-${INST_LINVAR} zfs-utils/" /mnt/etc/pacman.conf
Kernel will be updated manually. See `here <../1-zfs-linux.html#update-kernel>`__.
#. Enable ZFS services::
systemctl enable zfs-import-scan.service zfs-import.target zfs-zed zfs.target --root=/mnt
systemctl disable zfs-mount --root=/mnt
systemctl enable zfs-import-scan.service zfs-mount zfs-import.target zfs-zed zfs.target --root=/mnt
#. Add archzfs repo::
curl -L https://archzfs.com/archzfs.gpg | pacman-key -a - --gpgdir /mnt/etc/pacman.d/gnupg
pacman-key --lsign-key --gpgdir /mnt/etc/pacman.d/gnupg $(curl -L https://git.io/JsfVS)
curl -L https://git.io/Jsfw2 > /mnt/etc/pacman.d/mirrorlist-archzfs
tee -a /mnt/etc/pacman.conf <<- 'EOF'
#[archzfs-testing]
#Include = /etc/pacman.d/mirrorlist-archzfs
[archzfs]
Include = /etc/pacman.d/mirrorlist-archzfs
EOF
At boot, datasets on rpool are mounted with ``/etc/fstab``,
which can control the mounting process more precisely than ``zfs-mount.service``.
#. Chroot::
echo "INST_PRIMARY_DISK=$INST_PRIMARY_DISK
INST_LINVAR=$INST_LINVAR
INST_UUID=$INST_UUID
INST_ID=$INST_ID
INST_VDEV=$INST_VDEV
DISK=\"$DISK\"" > /mnt/root/chroot
history -w /mnt/home/sys-install-pre-chroot.txt
arch-chroot /mnt bash --login
arch-chroot /mnt /usr/bin/env DISK=$DISK bash
#. Source variables::
source /root/chroot
#. Apply locales, change if needed::
#. Generate locales::
echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen
locale-gen
#. `Add archzfs repo <../0-archzfs-repo.html>`__.

316
docs/Getting Started/Arch Linux/Root on ZFS/4-optional-configuration.rst
View File

@@ -1,316 +0,0 @@
.. highlight:: sh
Optional Configuration
======================
.. contents:: Table of Contents
:local:
Skip to `bootloader <5-bootloader.html>`__ section if
no optional configuration is needed.
Boot environment manager
~~~~~~~~~~~~~~~~~~~~~~~~
A boot environment is a dataset which contains a bootable
instance of an operating system. Within the context of this installation,
boot environments can be created on-the-fly to preserve root file system
states before pacman transactions.
Install an AUR helper of choice then install ``rozb3-pac`` from AUR
for pacman integration::
pacman -S --needed git base-devel sudo
echo 'nobody ALL=(ALL) NOPASSWD: ALL' > /etc/sudoers.d/00_nobody
su - nobody -s /bin/bash
mkdir /tmp/build
export HOME=/tmp/build
cd
git clone https://aur.archlinux.org/paru-bin.git
cd paru-bin
makepkg -si
paru -S rozb3-pac
exit
rm /etc/sudoers.d/00_nobody
If multi-disk setup is used, enable multi-disk
support inside ``/etc/bieaz.cfg``.
Supply password with SSH
~~~~~~~~~~~~~~~~~~~~~~~~
#. Install mkinitcpio tools::
pacman -S mkinitcpio-netconf mkinitcpio-dropbear openssh
#. Store public keys in ``/etc/dropbear/root_key``::
vi /etc/dropbear/root_key
#. Edit mkinitcpio::
tee /etc/mkinitcpio.conf <<- 'EOF'
HOOKS=(base udev autodetect modconf block keyboard netconf dropbear zfsencryptssh zfs filesystems)
EOF
#. Add ``ip=`` to kernel command line::
# example DHCP
echo 'GRUB_CMDLINE_LINUX="ip=::::::dhcp"' >> /etc/default/grub
Details for ``ip=`` can be found at
`here <https://www.kernel.org/doc/html/latest/admin-guide/nfs/nfsroot.html#kernel-command-line>`__.
#. Generate host keys::
ssh-keygen -Am pem
dropbearconvert openssh dropbear /etc/ssh/ssh_host_ed25519_key /etc/dropbear/dropbear_ed25519_host_key
`mkinitcpio-dropbear
<https://archlinux.org/packages/community/any/mkinitcpio-dropbear/>`__
lacks support for converting ed25519 host key,
`see this pull request
<https://github.com/grazzolini/mkinitcpio-dropbear/pull/13>`__.
Encrypt boot pool
~~~~~~~~~~~~~~~~~~~
Note: This will disable password with SSH. The password previously set for
root pool will be replaced by keyfile, embedded in initrd.
#. LUKS password::
LUKS_PWD=secure-passwd
You will need to enter the same password for
each disk at boot. As root pool key is
protected by this password, the previous warning
about password strength still apply.
Double-check password here. Complete reinstallation is
needed if entered wrong.
#. Create encryption keys::
mkdir /etc/cryptkey.d/
chmod 700 /etc/cryptkey.d/
dd bs=32 count=1 if=/dev/urandom of=/etc/cryptkey.d/rpool_$INST_UUID-${INST_ID}-key-zfs
dd bs=32 count=1 if=/dev/urandom of=/etc/cryptkey.d/bpool_$INST_UUID-key-luks
chmod u=r,go= /etc/cryptkey.d/*
#. Backup boot pool::
zfs snapshot -r bpool_$INST_UUID/$INST_ID@pre-luks
zfs send -Rv bpool_$INST_UUID/$INST_ID@pre-luks > /root/bpool_$INST_UUID-${INST_ID}-pre-luks
#. Unmount EFI partition::
umount /boot/efi
for i in ${DISK}; do
umount /boot/efis/${i##*/}-part1
done
#. Destroy boot pool::
zpool destroy bpool_$INST_UUID
#. Create LUKS containers::
for i in ${DISK}; do
cryptsetup luksFormat -q --type luks2 --key-file /etc/cryptkey.d/bpool_$INST_UUID-key-luks $i-part2
echo $LUKS_PWD | cryptsetup luksAddKey --pbkdf pbkdf2 --key-file /etc/cryptkey.d/bpool_$INST_UUID-key-luks $i-part2
cryptsetup open ${i}-part2 ${i##*/}-part2-luks-bpool_$INST_UUID --key-file /etc/cryptkey.d/bpool_$INST_UUID-key-luks
echo ${i##*/}-part2-luks-bpool_$INST_UUID ${i}-part2 /etc/cryptkey.d/bpool_$INST_UUID-key-luks discard >> /etc/crypttab
done
In GRUB 2.06, only the PBKDF2 key derivation function
is supported, thus PBKDF2 is used
for passphrase key slot. Other slots are not affected.
#. Embed key file in initrd::
echo 'FILES=(/etc/cryptkey.d/* )' >> /etc/mkinitcpio.conf
#. Recreate boot pool with mappers as vdev::
disk_num=0; for i in $DISK; do disk_num=$(( $disk_num + 1 )); done
if [ $disk_num -gt 1 ]; then INST_VDEV_BPOOL=mirror; fi
zpool create \
-o compatibility=grub2 \
-o ashift=12 \
-o autotrim=on \
-O acltype=posixacl \
-O canmount=off \
-O compression=lz4 \
-O devices=off \
-O normalization=formD \
-O relatime=on \
-O xattr=sa \
-O mountpoint=/boot \
bpool_$INST_UUID \
$INST_VDEV_BPOOL \
$(for i in ${DISK}; do
printf "/dev/mapper/${i##*/}-part2-luks-bpool_$INST_UUID ";
done)
#. Restore boot pool backup::
zfs recv bpool_${INST_UUID}/${INST_ID} < /root/bpool_$INST_UUID-${INST_ID}-pre-luks
rm /root/bpool_$INST_UUID-${INST_ID}-pre-luks
#. Mount boot dataset and EFI partitions::
mount /boot
mount /boot/efi
for i in ${DISK}; do
mount /boot/efis/${i##*/}-part1
done
#. As keys are stored in initrd,
set secure permissions for ``/boot``::
chmod 700 /boot
#. Change root pool password to key file::
zfs change-key -l \
-o keylocation=file:///etc/cryptkey.d/rpool_$INST_UUID-${INST_ID}-key-zfs \
-o keyformat=raw \
rpool_$INST_UUID/$INST_ID
#. Import encrypted boot pool from ``/dev/mapper``::
curl -L https://git.io/Jsfwj > /etc/systemd/system/zfs-import-bpool-mapper.service
systemctl enable zfs-import-bpool-mapper.service
#. Remove ``zfsencryptssh`` hook.
Encrypted boot pool is incompatible with
password by SSH::
sed -i 's|zfsencryptssh||g' /etc/mkinitcpio.conf
If ``zfsencryptssh`` is not removed, initrd will
stuck at ``fail to load key material`` and fail to boot.
#. Enable GRUB cryptodisk::
echo "GRUB_ENABLE_CRYPTODISK=y" >> /etc/default/grub
#. Let GRUB decrypt all LUKS containers on boot::
tee -a /etc/grub.d/09_bpool_luks2-decryption <<FOE
#!/bin/sh
cat <<EOF
insmod luks2
insmod pbkdf2
insmod part_gpt
insmod gcry_rijndael
insmod gcry_sha256
insmod cryptodisk
cryptomount hd0,gpt2
EOF
FOE
chmod +x /etc/grub.d/09_bpool_luks2-decryption
#. **Important**: Back up root dataset key ``/etc/cryptkey.d/rpool_$INST_UUID-${INST_ID}-key-zfs``
to a secure location.
In the possible event of LUKS container corruption,
data on root set will only be available
with this key.
Persistent swap and hibernation
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#. Optional: enable persistent swap partition. By default
encryption key of swap partition is discarded on reboot::
INST_SWAPKEY=/etc/cryptkey.d/${INST_PRIMARY_DISK##*/}-part4-key-luks-swap
INST_SWAPMAPPER=${INST_PRIMARY_DISK##*/}-part4-luks-swap
# fstab
# remove all existing swap entries
sed -i '/ none swap defaults 0 0/d' /etc/fstab
# add single swap entry for LUKS encrypted swap partition
echo "/dev/mapper/${INST_SWAPMAPPER} none swap defaults 0 0" >> /etc/fstab
# comment out entry in crypttab
sed -i "s|^${INST_PRIMARY_DISK##*/}-part4-swap|#${INST_PRIMARY_DISK##*/}-part4-swap|" /etc/crypttab
# create key and format partition as LUKS container
dd bs=32 count=1 if=/dev/urandom of=${INST_SWAPKEY};
chmod u=r,go= /etc/cryptkey.d/*
cryptsetup luksFormat -q --type luks2 --key-file ${INST_SWAPKEY} ${INST_PRIMARY_DISK}-part4
cryptsetup luksOpen ${INST_PRIMARY_DISK}-part4 ${INST_SWAPMAPPER} --key-file ${INST_SWAPKEY}
# initialize swap space
mkswap /dev/mapper/${INST_SWAPMAPPER}
#. Optional: after enabling persistent swap partition,
enable hibernation::
# add hook in initrd
sed -i 's| zfs | encrypt resume zfs |' /etc/mkinitcpio.conf
# add kernel cmdline to decrypt swap in initrd
echo "GRUB_CMDLINE_LINUX=\" \
zfs_import_dir=${INST_PRIMARY_DISK%/*} \
cryptdevice=PARTUUID=$(blkid -s PARTUUID -o value ${INST_PRIMARY_DISK}-part4):${INST_SWAPMAPPER}:allow-discards \
cryptkey=rootfs:${INST_SWAPKEY} \
resume=/dev/mapper/${INST_SWAPMAPPER}\"" \
>> /etc/default/grub
Note that hibernation might not work with discrete graphics, virtio graphics or
AMD APU integrated graphics. This is not specific to this guide.
Computer must resume from a continuous swap space, resume
from multiple swap partitions is not supported.
``encrypt`` hook can only decrypt one container at boot.
``sd-encrypt`` can decrypt multiple devices but is
not compatible with ``zfs`` hook.
Do not touch anything on disk while the computer is
in hibernation, see `kernel documentation
<https://www.kernel.org/doc/html/latest/power/swsusp.html>`__.
Boot Live ISO with GRUB
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
GRUB `can be configured <https://wiki.archlinux.org/title/Multiboot_USB_drive>`__ to boot ISO file directly.
In this section, we will download Live ISO to ESP and configure GRUB to
boot from it. This enables system recovery and re-installation.
#. Download Live iso to EFI system partition::
mkdir /boot/efi/iso
cd /boot/efi/iso
# select a mirror and disk version
# curl -O https://mirrors.ocf.berkeley.edu/archlinux/iso/2021.11.01/archlinux-2021.11.01-x86_64.iso
curl -O https://archlinux.org/iso/2021.11.01/archlinux-2021.11.01-x86_64.iso.sig
gpg --auto-key-retrieve --verify archlinux-2021.11.01-x86_64.iso.sig
Additionally you can build your own live image
with `archiso package <https://gitlab.archlinux.org/archlinux/archiso>`__.
GRUB supports verifying checksum.
See `manual page
<https://www.gnu.org/software/grub/manual/grub/html_node/Command_002dline-and-menu-entry-commands.html#Command_002dline-and-menu-entry-commands>`__
for details.
#. Add custom GRUB entry for ``/boot/efi/iso/archlinux-*.iso``::
curl -L https://git.io/Jsfr3 > /etc/grub.d/43_archiso
chmod +x /etc/grub.d/43_archiso
You can also boot Live ISO for other distros, see `glim
<https://github.com/thias/glim/tree/master/grub2>`__
configurations.
ISO is not mirrored to other devices due to its size.
Change ``$ESP_MNT`` to adapt to other ESP.
#. Generate ``grub.cfg`` in the next step. If a new file
has been added later, regenerate ``grub.cfg``.

121
docs/Getting Started/Arch Linux/Root on ZFS/5-bootloader.rst
View File

@@ -25,13 +25,6 @@ Workarounds have to be applied.
echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile.d/zpool_vdev_name_path.sh
source /etc/profile.d/zpool_vdev_name_path.sh
Note that ``sudo`` will not read ``/etc/profile`` and will
not pass variables in parent shell. Consider setting the following
in ``/etc/sudoers``::
pacman -S --noconfirm --needed sudo
echo 'Defaults env_keep += "ZPOOL_VDEV_NAME_PATH"' >> /etc/sudoers
#. Pool name missing
See `this bug report <https://savannah.gnu.org/bugs/?59614>`__.
@@ -43,60 +36,51 @@ Workarounds have to be applied.
sed -i "s|rpool=.*|rpool=\`zdb -l \${GRUB_DEVICE} \| grep -E '[[:blank:]]name' \| cut -d\\\' -f 2\`|" /etc/grub.d/10_linux
Caution: this fix must be applied after every GRUB update and before generating the menu.
Install GRUB
~~~~~~~~~~~~~~~~~~~~
#. Generate initrd::
#. Create empty cache file and generate initrd::
rm -f /etc/zfs/zpool.cache
touch /etc/zfs/zpool.cache
chmod a-w /etc/zfs/zpool.cache
chattr +i /etc/zfs/zpool.cache
mkinitcpio -P
#. Create GRUB boot directory, in ESP and boot pool::
mkdir -p /boot/efi/EFI/arch
mkdir -p /boot/grub
Boot environment-specific configuration (kernel, etc)
is stored in ``/boot/grub/grub.cfg``, enabling rollback.
#. When in doubt, install both legacy boot
and EFI.
#. If using legacy booting, install GRUB to every disk::
for i in ${DISK}; do
grub-install --boot-directory /boot/efi/EFI/arch --target=i386-pc $i
grub-install --target=i386-pc $i
done
#. If using EFI::
grub-install --boot-directory /boot/efi/EFI/arch --efi-directory /boot/efi/
grub-install --boot-directory /boot/efi/EFI/arch --efi-directory /boot/efi/ --removable
grub-install --target x86_64-efi
grub-install --target x86_64-efi --removable
for i in ${DISK}; do
efibootmgr -cgp 1 -l "\EFI\arch\grubx64.efi" \
-L "arch-${i##*/}" -d ${i}
done
#. Generate GRUB Menu::
#. Generate GRUB Menu:
grub-mkconfig -o /boot/efi/EFI/arch/grub/grub.cfg
cp /boot/efi/EFI/arch/grub/grub.cfg /boot/grub/grub.cfg
Generate menu::
echo GRUB_CMDLINE_LINUX=\"zfs_import_dir=/dev/disk/by-id/\" >> /etc/default/grub
grub-mkconfig -o /boot/grub/grub.cfg
cp /boot/grub/grub.cfg /boot/efi/EFI/arch/
#. For both legacy and EFI booting: mirror ESP content::
ESP_MIRROR=$(mktemp -d)
unalias -a
cp -r /boot/efi/EFI $ESP_MIRROR
for i in /boot/efis/*; do
cp -r $ESP_MIRROR/EFI $i
done
Enable Secure Boot
----------------------------
This is optional. `See Arch Wiki article <https://wiki.archlinux.org/title/Secure_Boot>`__.
rm -rf $ESP_MIRROR
Finish Installation
~~~~~~~~~~~~~~~~~~~~
@@ -105,82 +89,11 @@ Finish Installation
exit
#. Take a snapshot of the clean installation for future use::
zfs snapshot -r rpool_$INST_UUID/$INST_ID@install
zfs snapshot -r bpool_$INST_UUID/$INST_ID@install
#. Unmount EFI system partition::
umount /mnt/boot/efi
umount /mnt/boot/efis/*
#. Export pools::
zpool export bpool_$INST_UUID
zpool export rpool_$INST_UUID
umount -Rl /mnt
zpool export -a
#. Reboot::
reboot
Post installaion
~~~~~~~~~~~~~~~~
#. If you have other data pools, generate list of datasets for `zfs-mount-generator
<https://manpages.ubuntu.com/manpages/focal/man8/zfs-mount-generator.8.html>`__ to mount them at boot::
DATA_POOL='tank0 tank1'
# tab-separated zfs properties
# see /etc/zfs/zed.d/history_event-zfs-list-cacher.sh
export \
PROPS="name,mountpoint,canmount,atime,relatime,devices,exec\
,readonly,setuid,nbmand,encroot,keylocation"
for i in $DATA_POOL; do
zfs list -H -t filesystem -o $PROPS -r $i > /etc/zfs/zfs-list.cache/$i
done
#. After reboot, consider adding a normal user::
# with root permissions
sudo -i
# store user name in a variable
myUser=UserName
# rename default `User` to new user name
zfs rename $(df --output=source /home | tail -n +2)/User $(df --output=source /home | tail -n +2)/${myUser}
# update entry in fstab
sed -i "s|/home/User|/home/${myUser}|g" /etc/fstab
# add user
useradd --no-create-home --user-group --home-dir /home/${myUser} --comment 'My Name' ${myUser}
# delegate snapshot and destroy permissions of the home dataset to
# new user
zfs allow -u ${myUser} mount,snapshot,destroy $(df --output=source /home | tail -n +2)/${myUser}
# fix permissions
chown --recursive ${myUser}:${myUser} /home/${myUser}
chmod 700 /home/${myUser}
# set new password for user
passwd ${myUser}
Set up cron job to snapshot user home everyday::
pacman -S cronie
systemctl enable --now cronie
crontab -eu ${myUser}
#@daily zfs snap $(df --output=source /home/${myUser} | tail -n +2)@$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)
zfs list -t snapshot -S creation $(df --output=source /home/${myUser} | tail -n +2)
Install package groups::
pacman -Sg # query package groups
pacman -S 'gnome'
pacman -S 'plasma'

211
docs/Getting Started/Arch Linux/Root on ZFS/6-recovery.rst
View File

@@ -1,211 +0,0 @@
.. highlight:: sh
Recovery
======================
.. contents:: Table of Contents
:local:
GRUB Tips
-------------
Boot from GRUB rescue
~~~~~~~~~~~~~~~~~~~~~~~
If bootloader file is damaged, it's still possible
to boot computer with GRUB rescue image.
This section is also applicable if you are in
``grub rescue>``.
#. On another computer, generate rescue image with::
pacman -S --needed mtools libisoburn grub
grub-install
grub-mkrescue -o grub-rescue.img
dd if=grub-rescue.img of=/dev/your-usb-stick
Boot computer from the rescue media.
Both legacy and EFI mode are supported.
Or `download generated GRUB rescue image <https://gitlab.com/m_zhou/bieaz/uploads/e0847a8675cda4317ea7f48abb1d9f10/grub-rescue-2.06.img.7z>`__.
#. List available disks with ``ls`` command::
grub> ls (hd # press tab
Possible devices are:
hd0 hd1 hd2 hd3
#. List partitions by pressing tab key:
.. code-block:: text
grub> ls (hd0 # press tab
Possible partitions are:
Device hd0: No known filesystem detected - Sector size 512B - Total size 20971520KiB
Partition hd0,gpt1: Filesystem type fat - Label `EFI', UUID 0DF5-3A76 - Partition start at 1024KiB - Total size 1048576KiB
Partition hd0,gpt2: No known filesystem detected - Partition start at 1049600KiB - Total size 4194304KiB
- If boot pool is encrypted:
Unlock it with ``cryptomount``::
grub> insmod luks
grub> cryptomount hd0,gpt2
Attempting to decrypt master key...
Enter passphrase for hd0,gpt2 (af5a240e13e24483acf02600d61e0f36):
Slot 1 opened
Unlocked LUKS container is ``(crypto0)``:
.. code-block:: text
grub> ls (crypto0)
Device crypto0: Filesystem type zfs - Label `bpool_ip3tdb' - Last modification
time 2021-05-03 12:14:08 Monday, UUID f14d7bdf89fe21fb - Sector size 512B -
Total size 4192256KiB
- If boot pool is not encrypted:
.. code-block:: text
grub> ls (hd0,gpt2)
Device hd0,gpt2: Filesystem type zfs - Label `bpool_ip3tdb' - Last modification
time 2021-05-03 12:14:08 Monday, UUID f14d7bdf89fe21fb - Sector size 512B -
Total size 4192256KiB
#. List boot environments nested inside ``bpool/$INST_ID/BOOT``::
grub> ls (crypto0)/sys/BOOT
@/ default/ be0/
#. Instruct GRUB to load configuration from ``be0`` boot environment::
grub> prefix=(crypto0)/sys/BOOT/be0/@/grub
grub> configfile $prefix/grub.cfg
#. GRUB menu should now appear.
#. After entering system, `reinstall GRUB <#grub-installation>`__.
Switch GRUB prefix when disk fails
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you are using LUKS encrypted boot pool with multiple disks,
the primary disk failed, GRUB will fail to load configuration.
If there's still enough redundancy for the boot pool, try fix
GRUB with the following method:
#. Ensure ``Slot 1 opened`` message
is shown
.. code-block:: text
Welcome to GRUB!
error: no such cryptodisk found.
Attempting to decrypt master key...
Enter passphrase for hd0,gpt2 (c0987ea1a51049e9b3056622804de62a):
Slot 1 opened
error: disk `cryptouuid/47ed1b7eb0014bc9a70aede3d8714faf' not found.
Entering rescue mode...
grub rescue>
If ``error: access denied.`` is shown,
try re-enter password with::
grub rescue> cryptomount hd0,gpt2
#. Check prefix::
grub rescue > set
# prefix=(cryptouuid/47ed1b7eb0014bc9a70aede3d8714faf)/sys/BOOT/be0@/grub
# root=cryptouuid/47ed1b7eb0014bc9a70aede3d8714faf
#. Set correct ``prefix`` and ``root`` by replacing
``cryptouuid/UUID`` with ``crypto0``::
grub rescue> prefix=(crypto0)/sys/BOOT/default@/grub
grub rescue> root=crypto0
#. Boot GRUB::
grub rescue> insmod normal
grub rescue> normal
GRUB should then boot normally.
#. After entering system, edit ``/etc/fstab`` to promote
one backup to ``/boot/efi``.
#. Make the change to ``prefix`` and ``root``
permanent by `reinstalling GRUB <#grub-installation>`__.
Access system in chroot
-----------------------
#. Go through `preparation <1-preparation.html>`__.
#. Import and unlock root and boot pool::
zpool import -NR /mnt rpool_$INST_UUID
zpool import -NR /mnt bpool_$INST_UUID
If using password::
zfs load-key rpool_$INST_UUID/$INST_ID
If using keyfile::
zfs load-key -L file:///path/to/keyfile rpool_$INST_UUID/$INST_ID
#. Find the current boot environment::
zfs list
BE=default
#. Mount root filesystem::
zfs mount rpool_$INST_UUID/$INST_ID/ROOT/$BE
#. chroot into the system::
arch-chroot /mnt /bin/bash --login
zfs mount -a
mount -a
#. Finish rescue. See `finish installation <5-bootloader.html#finish-installation>`__.
Backup and migrate existing installation
----------------------------------------
With the help of `zfs send
<https://openzfs.github.io/openzfs-docs/man/8/zfs-send.8.html>`__
it is relatively easy to perform a system backup and migration.
#. Create a snapshot of root file system::
zfs snapshot -r rpool/arch@backup
zfs snapshot -r bpool/arch@backup
#. Save snapshot to a file or pipe to SSH::
zfs send --options rpool/arch@backup > /backup/arch-rpool
zfs send --options bpool/arch@backup > /backup/arch-bpool
#. Re-create partitions and root/boot
pool on target system.
#. Restore backup::
zfs recv rpool_new/arch < /backup/arch-rpool
zfs recv bpool_new/arch < /backup/arch-bpool
#. Chroot and reinstall bootloader.
#. Update pool name in ``/etc/fstab``, ``/boot/grub/grub.cfg``
and ``/etc/zfs/zfs-list.cache/*``.
#. Update device name, etc, in ``/etc/fstab`` and ``/etc/crypttab``.

102
docs/Getting Started/Arch Linux/Root on ZFS/snippets/43_archiso.txt
View File

@@ -1,102 +0,0 @@
#!/bin/sh
# mountpoint of vfat-formatted partition
ESP_MNT=/boot/efi
# path to iso files relative to the partition
ISO_REL=/iso
# absolute path to iso files
ISO_PATH=${ESP_MNT}/${ISO_REL}
# df command needs warm up due to systemd mount-on-demand
ls $ISO_PATH 1> /dev/null
# vfat partition UUID
ESP_UUID=$(blkid -s UUID -o value $(df --output=source ${ISO_PATH} | tail -n +2))
cat <<EOF
submenu 'Boot from Live ISO' {
#if tpm module is loaded, grub might fail to setup loop
#error message: out of memory
#rmmod tpm
EOF
# limit detected number of ISOs, too many
# lines might crush grub
ISO_NUM=0
# for archlinux
for isofile in $ISO_PATH/archlinux-*.iso; do
if [ "$ISO_NUM" -gt 300 ]; then break; fi
isoname=${isofile##*/}
cat <<EOF
submenu "$isoname" {
insmod search_fs_uuid
set isorootuuid=$ESP_UUID
search --fs-uuid --no-floppy --set=isopart \$isorootuuid
set isopath=$ISO_REL
loopback loop0 (\$isopart)\$isopath/$isoname
set root=(loop0)
menuentry "Arch Linux install medium" {
linux /arch/boot/x86_64/vmlinuz-linux \\
earlymodules=loop img_dev=/dev/disk/by-uuid/\$isorootuuid img_loop=\$isopath/$isoname
initrd /arch/boot/intel-ucode.img
initrd /arch/boot/amd-ucode.img
initrd /arch/boot/x86_64/initramfs-linux.img
}
menuentry "Arch Linux install medium, Copy to RAM" {
linux /arch/boot/x86_64/vmlinuz-linux \\
earlymodules=loop img_dev=/dev/disk/by-uuid/\$isorootuuid img_loop=\$isopath/$isoname \\
copytoram
initrd /arch/boot/intel-ucode.img
initrd /arch/boot/amd-ucode.img
initrd /arch/boot/x86_64/initramfs-linux.img
}
menuentry "Arch Linux install medium with speech" {
linux /arch/boot/x86_64/vmlinuz-linux \\
earlymodules=loop img_dev=/dev/disk/by-uuid/\$isorootuuid img_loop=\$isopath/$isoname \\
accessibility=on
initrd /arch/boot/intel-ucode.img
initrd /arch/boot/amd-ucode.img
initrd /arch/boot/x86_64/initramfs-linux.img
}
}
EOF
ISO_NUM=$(( $ISO_NUM + 1 ))
done
# for ubuntu
for isofile in $ISO_PATH/ubuntu-*.iso; do
if [ "$ISO_NUM" -gt 300 ]; then break; fi
isoname=${isofile##*/}
cat <<EOF
submenu "$isoname" {
insmod search_fs_uuid
set isorootuuid=$ESP_UUID
search --fs-uuid --no-floppy --set=isopart \$isorootuuid
set isopath=$ISO_REL
loopback loop0 (\$isopart)\$isopath/$isoname
set root=(loop0)
menuentry "Ubuntu" {
linux /casper/vmlinuz \\
boot=casper iso-scan/filename=\$isopath/$isoname
initrd /casper/initrd
}
menuentry "Ubuntu, Copy to RAM" {
linux /casper/vmlinuz \\
boot=casper iso-scan/filename=\$isopath/$isoname \\
toram
initrd /casper/initrd
}
}
EOF
ISO_NUM=$(( $ISO_NUM + 1 ))
done
cat <<EOF
}
EOF
# archiso kernel cmdline from
# https://gitlab.archlinux.org/archlinux/archiso/-/tree/master/configs/releng/efiboot/loader/entries

16
docs/Getting Started/Arch Linux/Root on ZFS/snippets/zfs-import-bpool-mapper.service.txt
View File

@@ -1,16 +0,0 @@
[Unit]
Description=Import encrypted boot pool
Documentation=man:zpool(8)
DefaultDependencies=no
Requires=systemd-udev-settle.service
After=cryptsetup.target
Before=boot.mount
ConditionPathIsDirectory=/sys/module/zfs
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/zpool import -aNd /dev/mapper
[Install]
WantedBy=zfs-import.target

2
docs/Getting Started/Arch Linux/index.rst
View File

@@ -52,7 +52,7 @@ Root on ZFS
ZFS can be used as root file system for Arch Linux.
An installation guide is available.
`Start here <Root%20on%20ZFS/0-overview.html>`__.
`Start here <Root%20on%20ZFS/1-preparation.html>`__.
.. toctree::
:maxdepth: 1

2
docs/Getting Started/Fedora/Fedora Root on ZFS.rst
View File

@@ -1,6 +1,6 @@
Fedora Root on ZFS
======================
`Start here <Root%20on%20ZFS/0-overview.html>`__.
`Start here <Root%20on%20ZFS/1-preparation.html>`__.
Contents
--------

141
docs/Getting Started/Fedora/Root on ZFS/0-overview.rst
View File

@@ -1,141 +0,0 @@
.. highlight:: sh
Overview
======================
This document describes how to install Fedora with ZFS as root
file system.
Caution
~~~~~~~
- With less than 4GB RAM, DKMS might fail to build
in live environment.
- This guide wipes entire physical disks. Back up existing data.
- `GRUB does not and
will not work on 4Kn drive with legacy (BIOS) booting.
<http://savannah.gnu.org/bugs/?46700>`__
Partition layout
~~~~~~~~~~~~~~~~
GUID partition table (GPT) is used.
EFI system partition will be referred to as **ESP** in this document.
+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+
| Name | legacy boot | ESP | Boot pool | swap | root pool | remaining space |
+======================+======================+=======================+======================+=====================+=======================+=================+
| File system | | vfat | ZFS | swap | ZFS | |
+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+
| Size | 1M | 2G | 4G | depends on RAM size | | |
+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+
| Optional encryption | | *Secure Boot* | | plain dm-crypt | ZFS native encryption | |
| | | | | | | |
+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+
| Partition no. | 5 | 1 | 2 | 4 | 3 | |
+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+
| Mount point | | /boot/efi | /boot | | / | |
| | | /boot/efis/disk-part1 | | | | |
+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+
Dataset layout
~~~~~~~~~~~~~~
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| Dataset | canmount | mountpoint | container | notes |
+===========================+======================+======================+=====================================+===========================================+
| bpool | off | /boot | contains sys | |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool | off | / | contains sys | |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| bpool/sys | off | none | contains BOOT | |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool/sys | off | none | contains ROOT | sys is encryptionroot |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| bpool/sys/BOOT | off | none | contains boot environments | |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool/sys/ROOT | off | none | contains boot environments | |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool/sys/DATA | off | none | contains placeholder "default" | |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool/sys/DATA/default | off | / | contains user datasets | child datsets inherits mountpoint |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool/sys/DATA/default/ | on | /home (inherited) | no | |
| home | | | | user datasets, also called "shared |
| | | | | datasets", "persistent datasets"; also |
| | | | | include /var/lib, /srv, ... |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| bpool/sys/BOOT/default | noauto | /boot | no | noauto is used to switch BE. because of |
| | | | | noauto, must use fstab to mount |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool/sys/ROOT/default | noauto | / | no | mounted by initrd zfs hook |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| bpool/sys/BOOT/be1 | noauto | /boot | no | see bpool/sys/BOOT/default |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool/sys/ROOT/be1 | noauto | / | no | see rpool/sys/ROOT/default |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
Encryption
~~~~~~~~~~
- Swap
Swap is always encrypted. By default, swap is encrypted
with plain dm-crypt with key generated from ``/dev/urandom``
at every boot. Swap content does not persist between reboots.
- Root pool
ZFS native encryption can be optionally enabled for ``rpool/sys``
and child datasets.
User should be aware that, ZFS native encryption does not
encrypt some metadata of the datasets.
ZFS native encryption also does not change master key when ``zfs change-key`` is invoked.
Therefore, you should wipe the disk when password is compromised to protect confidentiality.
See `zfs-load-key.8 <https://openzfs.github.io/openzfs-docs/man/8/zfs-load-key.8.html>`__
and `zfs-change-key.8 <https://openzfs.github.io/openzfs-docs/man/8/zfs-change-key.8.html>`__
for more information regarding ZFS native encryption.
Encryption is enabled at dataset creation and can not be disabled later.
- Boot pool
Boot pool can not be encrypted.
- Bootloader
Bootloader can not be encrypted.
However, with Secure Boot, bootloader
can be verified by motherboard firmware to be untempered,
which should be sufficient for most purposes.
Secure Boot is not supported out-of-the-box due to ZFS module.
Booting with disk failure
~~~~~~~~~~~~~~~~~~~~~~~~~
This guide is written with disk failure in mind.
If disks used in Root on ZFS pool failed, but
sufficient redundancy for both root pool and boot pool
still exists, the system will still boot normally.
Swap partition on the failed disk will fail to mount,
after an 1m30s timeout.
This feature is useful for use cases such
as an unattended remote server.
Example:
- System has disks ``n>1``
- Installed with mirrored setup
- Mirrored setup can tolerate up to ``n-1`` disk failures
- Disconnect one or more disks, keep at least
one disk connected
- System still boots, but fails to mount swap and
EFI partition

136
docs/Getting Started/Fedora/Root on ZFS/1-preparation.rst
View File

@@ -7,75 +7,26 @@ Preparation
:local:
#. Disable Secure Boot. ZFS modules can not be loaded if Secure Boot is enabled.
#. Download a variant of Fedora 36 live image
and boot from it.
- `Fedora Workstation (GNOME) <https://download.fedoraproject.org/pub/fedora/linux/releases/36/Workstation/x86_64/iso/>`__
- `Fedora Spins (Xfce, i3, ...) <https://download.fedoraproject.org/pub/fedora/linux/releases/36/Spins/x86_64/iso/>`__
#. Download a variant of `Fedora live image
<https://download.fedoraproject.org/pub/fedora/linux/releases/>`__ and boot from it.
#. Connect to the Internet.
#. Set root password or ``/root/.ssh/authorized_keys``.
#. Start SSH server::
echo PermitRootLogin yes >> /etc/ssh/sshd_config
systemctl start sshd
systemctl restart sshd
#. Connect from another computer::
ssh root@192.168.1.19
#. Temporarily set SELinux to permissive in live environment::
setenforce 0
SELinux will be enabled on the installed system.
#. Install ``kernel-devel``::
source /etc/os-release
dnf install -y https://dl.fedoraproject.org/pub/fedora/linux/releases/${VERSION_ID}/Everything/x86_64/os/Packages/k/kernel-devel-$(uname -r).rpm
#. Add ZFS repo::
dnf install -y https://zfsonlinux.org/fedora/zfs-release.fc${VERSION_ID}.noarch.rpm
#. If zfs-fuse from official Fedora repo is installed, remove it first. It is not maintained and should not be used under any circumstance::
rpm -e --nodeps zfs-fuse
#. Install ZFS packages::
dnf install -y zfs
#. Load kernel modules::
modprobe zfs
#. Install helper script and partition tool::
dnf install -y arch-install-scripts gdisk dosfstools
#. Target Fedora version::
INST_FEDORA_VER=${VERSION_ID}
#. Unique pool suffix. ZFS expects pool names to be
unique, therefore it's recommended to create
pools with a unique suffix::
INST_UUID=$(dd if=/dev/urandom bs=1 count=100 2>/dev/null | tr -dc 'a-z0-9' | cut -c-6)
#. Identify this installation in ZFS filesystem path::
INST_ID=fedora
#. Target disk
List available disks with::
ls /dev/disk/by-id/*
If using virtio as disk bus, use
``/dev/disk/by-path/*``.
If using virtio as disk bus, use ``/dev/disk/by-path/*``.
Declare disk array::
@@ -85,57 +36,8 @@ Preparation
DISK='/dev/disk/by-id/disk1'
#. Choose a primary disk. This disk will be used
for primary EFI partition and hibernation, default to
first disk in the array::
INST_PRIMARY_DISK=$(echo $DISK | cut -f1 -d\ )
#. Set vdev topology, possible values are:
- (not set, single disk or striped; no redundancy)
- mirror
- raidz1
- raidz2
- raidz3
::
INST_VDEV=
This will create a single vdev with the topology of your choice.
It is also possible to manually create a pool with multiple vdevs, such as::
zpool create --options \
poolName \
mirror sda sdb \
raidz2 sdc ... \
raidz3 sde ... \
spare sdf ...
Notice the cost of parity when using RAID-Z. See
`here <https://www.delphix.com/blog/delphix-engineering/zfs-raidz-stripe-width-or-how-i-learned-stop-worrying-and-love-raidz>`__
and `here <https://docs.google.com/spreadsheets/d/1tf4qx1aMJp8Lo_R6gpT689wTjHv6CGVElrPqTA0w_ZY/>`__.
For boot pool, which must be readable by GRUB, mirrored vdev should always be used for maximum redundancy.
This guide will use mirrored bpool for multi-disk setup.
Refer to `zpoolconcepts <https://openzfs.github.io/openzfs-docs/man/7/zpoolconcepts.7.html>`__
and `zpool-create <https://openzfs.github.io/openzfs-docs/man/8/zpool-create.8.html>`__
man pages for details.
#. Set partition size:
Set ESP size::
INST_PARTSIZE_ESP=2 # in GB
Set boot pool size. To avoid running out of space while using
boot environments, the minimum is 4GB. Adjust the size if you
intend to use multiple kernel/distros::
INST_PARTSIZE_BPOOL=4
Set swap size. It's `recommended <https://chrisdown.name/2018/01/02/in-defence-of-swap.html>`__
to setup a swap partition. If you intend to use hibernation,
the minimum should be no less than RAM size. Skip if swap is not needed::
@@ -145,3 +47,31 @@ Preparation
Root pool size, use all remaining disk space if not set::
INST_PARTSIZE_RPOOL=
#. Temporarily set SELinux to permissive in live environment::
setenforce 0
SELinux will be enabled on the installed system.
#. Add ZFS repo::
dnf install -y https://zfsonlinux.org/fedora/zfs-release-fedora-2-1.noarch.rpm
#. Check available repos::
dnf repolist --all
#. Install ZFS packages::
rpm -e --nodeps zfs-fuse
dnf install -y https://dl.fedoraproject.org/pub/fedora/linux/releases/$(source /etc/os-release; echo $VERSION_ID)/Everything/x86_64/os/Packages/k/kernel-devel-$(uname -r).rpm
dnf install -y zfs
#. Load kernel modules::
modprobe zfs
#. Install partition tool::
dnf install -y gdisk dosfstools

189
docs/Getting Started/Fedora/Root on ZFS/2-system-installation.rst
View File

@@ -6,48 +6,29 @@ System Installation
.. contents:: Table of Contents
:local:
#. Optional: wipe solid-state drives with the generic tool
`blkdiscard <https://utcc.utoronto.ca/~cks/space/blog/linux/ErasingSSDsWithBlkdiscard>`__,
to clean previous partition tables and improve performance.
All content will be irrevocably destroyed::
#. Partition the disks::
for i in ${DISK}; do
blkdiscard -f $i &
done
wait
This is a quick operation and should be completed under one
minute.
For other device specific methods, see
`Memory cell clearing <https://wiki.archlinux.org/title/Solid_state_drive/Memory_cell_clearing>`__
#. Partition the disks.
See `Overview <0-overview.html>`__ for details::
for i in ${DISK}; do
sgdisk --zap-all $i
sgdisk -n1:1M:+${INST_PARTSIZE_ESP}G -t1:EF00 $i
sgdisk -n2:0:+${INST_PARTSIZE_BPOOL}G -t2:BE00 $i
if [ "${INST_PARTSIZE_SWAP}" != "" ]; then
sgdisk -n4:0:+${INST_PARTSIZE_SWAP}G -t4:8200 $i
fi
if [ "${INST_PARTSIZE_RPOOL}" = "" ]; then
sgdisk -n1:1M:+1G -t1:EF00 $i
sgdisk -n2:0:+4G -t2:BE00 $i
test -z $INST_PARTSIZE_SWAP || sgdisk -n4:0:+${INST_PARTSIZE_SWAP}G -t4:8200 $i
if test -z $INST_PARTSIZE_RPOOL; then
sgdisk -n3:0:0 -t3:BF00 $i
else
sgdisk -n3:0:+${INST_PARTSIZE_RPOOL}G -t3:BF00 $i
fi
sgdisk -a1 -n5:24K:+1000K -t5:EF02 $i
done
#. Create boot pool::
disk_num=0; for i in $DISK; do disk_num=$(( $disk_num + 1 )); done
if [ $disk_num -gt 1 ]; then INST_VDEV_BPOOL=mirror; fi
zpool create \
-o compatibility=grub2 \
-o ashift=12 \
@@ -61,12 +42,14 @@ System Installation
-O xattr=sa \
-O mountpoint=/boot \
-R /mnt \
bpool_$INST_UUID \
$INST_VDEV_BPOOL \
bpool \
mirror \
$(for i in ${DISK}; do
printf "$i-part2 ";
done)
If not using a multi-disk setup, remove ``mirror``.
You should not need to customize any of the options for the boot pool.
GRUB does not support all of the zpool features. See ``spa_feature_names``
@@ -93,67 +76,15 @@ System Installation
-O relatime=on \
-O xattr=sa \
-O mountpoint=/ \
rpool_$INST_UUID \
$INST_VDEV \
rpool \
mirror \
$(for i in ${DISK}; do
printf "$i-part3 ";
done)
**Notes:**
If not using a multi-disk setup, remove ``mirror``.
- The use of ``ashift=12`` is recommended here because many drives
today have 4 KiB (or larger) physical sectors, even though they
present 512 B logical sectors. Also, a future replacement drive may
have 4 KiB physical sectors (in which case ``ashift=12`` is desirable)
or 4 KiB logical sectors (in which case ``ashift=12`` is required).
- Setting ``-O acltype=posixacl`` enables POSIX ACLs globally. If you
do not want this, remove that option, but later add
``-o acltype=posixacl`` (note: lowercase “o”) to the ``zfs create``
for ``/var/log``, as `journald requires ACLs
<https://askubuntu.com/questions/970886/journalctl-says-failed-to-search-journal-acl-operation-not-supported>`__
- Setting ``normalization=formD`` eliminates some corner cases relating
to UTF-8 filename normalization. It also implies ``utf8only=on``,
which means that only UTF-8 filenames are allowed. If you care to
support non-UTF-8 filenames, do not use this option. For a discussion
of why requiring UTF-8 filenames may be a bad idea, see `The problems
with enforced UTF-8 only filenames
<http://utcc.utoronto.ca/~cks/space/blog/linux/ForcedUTF8Filenames>`__.
- ``recordsize`` is unset (leaving it at the default of 128 KiB). If you
want to tune it (e.g. ``-o recordsize=1M``), see `these
<https://jrs-s.net/2019/04/03/on-zfs-recordsize/>`__ `various
<http://blog.programster.org/zfs-record-size>`__ `blog
<https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSFileRecordsizeGrowth>`__
`posts
<https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSRecordsizeAndCompression>`__.
- Setting ``relatime=on`` is a middle ground between classic POSIX
``atime`` behavior (with its significant performance impact) and
``atime=off`` (which provides the best performance by completely
disabling atime updates). Since Linux 2.6.30, ``relatime`` has been
the default for other filesystems. See `RedHat’s documentation
<https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/power_management_guide/relatime>`__
for further information.
- Setting ``xattr=sa`` `vastly improves the performance of extended
attributes
<https://github.com/zfsonlinux/zfs/commit/82a37189aac955c81a59a5ecc3400475adb56355>`__.
Inside ZFS, extended attributes are used to implement POSIX ACLs.
Extended attributes can also be used by user-space applications.
`They are used by some desktop GUI applications.
<https://en.wikipedia.org/wiki/Extended_file_attributes#Linux>`__
`They can be used by Samba to store Windows ACLs and DOS attributes;
they are required for a Samba Active Directory domain controller.
<https://wiki.samba.org/index.php/Setting_up_a_Share_Using_Windows_ACLs>`__
Note that ``xattr=sa`` is `Linux-specific
<https://openzfs.org/wiki/Platform_code_differences>`__. If you move your
``xattr=sa`` pool to another OpenZFS implementation besides ZFS-on-Linux,
extended attributes will not be readable (though your data will be). If
portability of extended attributes is important to you, omit the
``-O xattr=sa`` above. Even if you do not want ``xattr=sa`` for the whole
pool, it is probably fine to use it for ``/var/log``.
- Make sure to include the ``-part3`` portion of the drive path. If you
forget that, you are specifying the whole disk, which ZFS will then
re-partition, and you will lose the bootloader partition(s).
#. This section implements dataset layout as described in `overview <0-overview.html>`__.
#. This section implements dataset layout as described in `overview <1-preparation.html>`__.
Create root system container:
@@ -162,7 +93,7 @@ System Installation
zfs create \
-o canmount=off \
-o mountpoint=none \
rpool_$INST_UUID/$INST_ID
rpool/redhat
- Encrypted:
@@ -175,28 +106,19 @@ System Installation
-o encryption=on \
-o keylocation=prompt \
-o keyformat=passphrase \
rpool_$INST_UUID/$INST_ID
rpool/redhat
Create other system datasets::
Create system datasets::
zfs create -o canmount=off -o mountpoint=none bpool_$INST_UUID/$INST_ID
zfs create -o canmount=off -o mountpoint=none bpool_$INST_UUID/$INST_ID/BOOT
zfs create -o canmount=off -o mountpoint=none rpool_$INST_UUID/$INST_ID/ROOT
zfs create -o canmount=off -o mountpoint=none rpool_$INST_UUID/$INST_ID/DATA
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$INST_UUID/$INST_ID/BOOT/default
zfs create -o mountpoint=/ -o canmount=off rpool_$INST_UUID/$INST_ID/DATA/default
zfs create -o mountpoint=/ -o canmount=noauto rpool_$INST_UUID/$INST_ID/ROOT/default
zfs mount rpool_$INST_UUID/$INST_ID/ROOT/default
zfs mount bpool_$INST_UUID/$INST_ID/BOOT/default
for i in {usr,var,var/lib};
do
zfs create -o canmount=off rpool_$INST_UUID/$INST_ID/DATA/default/$i
done
for i in {home,root,srv,usr/local,var/log,var/spool};
do
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/$i
done
chmod 750 /mnt/root
zfs create -o canmount=on -o mountpoint=/ rpool/redhat/root
zfs create -o canmount=on -o mountpoint=/home rpool/redhat/home
zfs create -o canmount=off -o mountpoint=/var rpool/redhat/var
zfs create -o canmount=on rpool/redhat/var/lib
zfs create -o canmount=on rpool/redhat/var/log
Create boot dataset::
zfs create -o canmount=on -o mountpoint=/boot bpool/redhat
#. Format and mount ESP::
@@ -207,50 +129,15 @@ System Installation
done
mkdir -p /mnt/boot/efi
mount -t vfat ${INST_PRIMARY_DISK}-part1 /mnt/boot/efi
mount -t vfat $(echo $DISK | cut -f1 -d\ )-part1 /mnt/boot/efi
#. Create separate user dataset at ``/home/User``, dateset name can be
changed later::
#. Install packages::
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/home/User
dnf --installroot=/mnt --releasever=$(source /etc/os-release ; echo $VERSION_ID) -y install \
@core grub2-efi-x64 grub2-pc-modules grub2-efi-x64-modules shim-x64 efibootmgr kernel \
kernel-devel
If needed, snapshot, rollback and other related permissions can be
delegated to the user later.
dnf --installroot=/mnt --releasever=$(source /etc/os-release ; echo $VERSION_ID) -y install \
https://zfsonlinux.org/fedora/zfs-release-fedora-2-1.noarch.rpm
#. Create optional program data datasets to omit data from rollback::
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/games
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/www
# for GNOME
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/AccountsService
# for Docker
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/docker
# for NFS
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/nfs
# for LXC
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/lxc
# for LibVirt
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/libvirt
##other application
# zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/$name
Add other datasets when needed, such as PostgreSQL.
#. Install base packages::
dnf --installroot=/mnt --releasever=${INST_FEDORA_VER} -y install \
https://zfsonlinux.org/fedora/zfs-release.fc${INST_FEDORA_VER}.noarch.rpm \
@core grub2-efi-x64 grub2-pc-modules grub2-efi-x64-modules shim-x64 efibootmgr cryptsetup \
kernel kernel-devel python3-dnf-plugin-post-transaction-actions
#. Install ZFS::
dnf --installroot=/mnt -y install zfs zfs-dracut
#. Optional: enable boot environment support and dnf integration::
dnf --installroot=/mnt copr enable -y m0p/bieaz
dnf --installroot=/mnt install -y bieaz python3-dnf-plugin-rozb3
If multi-disk setup is used, enable multi-disk
support inside ``/mnt/etc/bieaz.cfg``.
dnf --installroot=/mnt --releasever=$(source /etc/os-release ; echo $VERSION_ID) -y install zfs zfs-dracut

66
docs/Getting Started/Fedora/Root on ZFS/3-system-configuration.rst
View File

@@ -8,24 +8,13 @@ System Configuration
#. Generate fstab::
genfstab -U /mnt | sed 's;zfs[[:space:]]*;zfs zfsutil,;g' | grep "zfs zfsutil" >> /mnt/etc/fstab
mkdir -p /mnt/etc/
for i in ${DISK}; do
echo UUID=$(blkid -s UUID -o value ${i}-part1) /boot/efis/${i##*/}-part1 vfat \
x-systemd.idle-timeout=1min,x-systemd.automount,noauto,umask=0022,fmask=0022,dmask=0022 0 1 >> /mnt/etc/fstab
umask=0022,fmask=0022,dmask=0022 0 1 >> /mnt/etc/fstab
done
echo UUID=$(blkid -s UUID -o value ${INST_PRIMARY_DISK}-part1) /boot/efi vfat \
x-systemd.idle-timeout=1min,x-systemd.automount,noauto,umask=0022,fmask=0022,dmask=0022 0 1 >> /mnt/etc/fstab
if [ "${INST_PARTSIZE_SWAP}" != "" ]; then
for i in ${DISK}; do
echo ${i##*/}-part4-swap ${i}-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256,discard >> /mnt/etc/crypttab
echo /dev/mapper/${i##*/}-part4-swap none swap x-systemd.requires=cryptsetup.target,defaults 0 0 >> /mnt/etc/fstab
done
fi
By default, systemd will halt boot process if any entry in ``/etc/fstab`` fails
to mount. This is unnecessary for mirrored EFI boot partitions.
With the above mount options, systemd will skip mounting them at boot,
only mount them on demand when accessed.
echo $(echo $DISK | cut -f1 -d\ )-part1 /boot/efi vfat \
noauto,umask=0022,fmask=0022,dmask=0022 0 1 >> /mnt/etc/fstab
#. Configure dracut::
@@ -37,25 +26,10 @@ System Configuration
echo 'forced_drivers+=" mpt3sas "' >> /mnt/etc/dracut.conf.d/zfs.conf
fi
#. Enable timezone sync::
hwclock --systohc
systemctl enable systemd-timesyncd --root=/mnt
#. Interactively set locale, keymap, timezone, hostname and root password::
#. Set locale, keymap, timezone, hostname and root password::
rm -f /mnt/etc/localtime
systemd-firstboot --root=/mnt --force --prompt --root-password=PASSWORD
This can be non-interactive, see man page for details::
rm -f /mnt/etc/localtime
systemd-firstboot --root=/mnt --force \
--locale="en_US.UTF-8" --locale-messages="en_US.UTF-8" \
--keymap=us --timezone="Europe/Berlin" --hostname=myHost \
--root-password=PASSWORD --root-shell=/bin/bash
``systemd-firstboot`` have bugs, root password is set below.
systemd-firstboot --root=/mnt --prompt --root-password=PASSWORD --force
#. Generate host id::
@@ -65,16 +39,9 @@ System Configuration
dnf --installroot=/mnt install -y glibc-minimal-langpack glibc-langpack-en
Program will show errors if not installed.
#. Enable ZFS services::
systemctl enable zfs-import-scan.service zfs-import.target zfs-zed zfs.target --root=/mnt
systemctl disable zfs-mount --root=/mnt
At boot, datasets on rpool are mounted with ``/etc/fstab``,
which can control the mounting process more precisely than ``zfs-mount.service``.
systemctl enable zfs-import-scan.service zfs-mount zfs-import.target zfs-zed zfs.target --root=/mnt
#. By default SSH server is enabled, allowing root login by password,
disable SSH server::
@@ -84,22 +51,13 @@ System Configuration
#. Chroot::
echo "INST_PRIMARY_DISK=$INST_PRIMARY_DISK
INST_LINVAR=$INST_LINVAR
INST_UUID=$INST_UUID
INST_ID=$INST_ID
unalias -a
INST_VDEV=$INST_VDEV
DISK=\"$DISK\"" > /mnt/root/chroot
#backup bash inputs up to this point
m='/dev /proc /sys'
for i in $m; do mount --rbind $i /mnt/$i; done
history -w /mnt/home/sys-install-pre-chroot.txt
arch-chroot /mnt bash --login
chroot /mnt /usr/bin/env DISK=$DISK bash --login
#. Source variables::
source /root/chroot
#. For SELinux, relabel filesystem on next boot::
#. For SELinux, relabel filesystem on reboot::
fixfiles -F onboot

159
docs/Getting Started/Fedora/Root on ZFS/5-bootloader.rst
View File

@@ -36,6 +36,8 @@ Workarounds have to be applied.
sed -i "s|rpool=.*|rpool=\`zdb -l \${GRUB_DEVICE} \| grep -E '[[:blank:]]name' \| cut -d\\\' -f 2\`|" /etc/grub.d/10_linux
Caution: this fix must be applied after every GRUB update and before generating the menu.
Install GRUB
~~~~~~~~~~~~~~~~~~~~
@@ -43,37 +45,26 @@ Install GRUB
echo 'filesystems+=" virtio_blk "' >> /etc/dracut.conf.d/fs.conf
#. Generate initrd::
#. Create empty cache file and generate initrd::
rm -f /etc/zfs/zpool.cache
touch /etc/zfs/zpool.cache
chmod a-w /etc/zfs/zpool.cache
chattr +i /etc/zfs/zpool.cache
for directory in /lib/modules/*; do
kernel_version=$(basename $directory)
dracut --force --kver $kernel_version
done
#. Disable BLS::
#. Load ZFS modules and disable BLS::
echo "GRUB_ENABLE_BLSCFG=false" >> /etc/default/grub
#. Create GRUB boot directory, in ESP and boot pool::
mkdir -p /boot/efi/EFI/fedora # EFI GRUB dir
mkdir -p /boot/efi/EFI/fedora/grub2 # legacy GRUB dir
mkdir -p /boot/grub2
Boot environment-specific configuration (kernel, etc)
is stored in ``/boot/grub2/grub.cfg``, enabling rollback.
#. When in doubt, install both legacy boot
and EFI.
echo 'GRUB_ENABLE_BLSCFG=false' >> /etc/default/grub
#. If using legacy booting, install GRUB to every disk::
for i in ${DISK}; do
grub2-install --boot-directory /boot/efi/EFI/fedora --target=i386-pc $i
grub2-install --target=i386-pc $i
done
#. If using EFI::
@@ -82,13 +73,14 @@ Install GRUB
efibootmgr -cgp 1 -l "\EFI\fedora\shimx64.efi" \
-L "fedora-${i##*/}" -d ${i}
done
cp -r /usr/lib/grub/x86_64-efi/ /boot/efi/EFI/fedora
cp -r /usr/lib/grub/x86_64-efi/ /boot/efi/EFI/fedora/
#. Generate GRUB Menu::
#. Generate GRUB Menu:
grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
cp /boot/efi/EFI/fedora/grub.cfg /boot/efi/EFI/fedora/grub2/grub.cfg
cp /boot/efi/EFI/fedora/grub.cfg /boot/grub2/grub.cfg
Generate menu::
grub2-mkconfig -o /boot/grub2/grub.cfg
cp /boot/grub2/grub.cfg /boot/efi/EFI/fedora/
#. For both legacy and EFI booting: mirror ESP content::
@@ -98,55 +90,14 @@ Install GRUB
for i in /boot/efis/*; do
cp -r $ESP_MIRROR/EFI $i
done
#. Automatically regenerate GRUB menu on kernel update::
tee /etc/dnf/plugins/post-transaction-actions.d/00-update-grub-menu-for-kernel.action <<EOF >/dev/null
# kernel-core package contains vmlinuz and initramfs
# change package name if non-standard kernel is used
kernel-core:in:/usr/local/sbin/update-grub-menu.sh
kernel-core:out:/usr/local/sbin/update-grub-menu.sh
EOF
tee /usr/local/sbin/update-grub-menu.sh <<-'EOF' >/dev/null
#!/bin/sh
export PATH=$PATH:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
export ZPOOL_VDEV_NAME_PATH=YES
source /etc/os-release
grub2-mkconfig -o /boot/efi/EFI/${ID}/grub.cfg
cp /boot/efi/EFI/${ID}/grub.cfg /boot/efi/EFI/${ID}/grub2/grub.cfg
cp /boot/efi/EFI/${ID}/grub.cfg /boot/grub2/grub.cfg
ESP_MIRROR=$(mktemp -d)
cp -r /boot/efi/EFI $ESP_MIRROR
for i in /boot/efis/*; do
cp -r $ESP_MIRROR/EFI $i
done
rm -rf $ESP_MIRROR
EOF
chmod +x /usr/local/sbin/update-grub-menu.sh
#. Notes for GRUB on Fedora
To support Secure Boot, GRUB has been heavily modified by Fedora,
namely:
- ``grub2-install`` is `disabled for UEFI <https://bugzilla.redhat.com/show_bug.cgi?id=1917213>`__
- Only a static, signed version of bootloader is copied to EFI system partition
- This signed bootloader does not have built-in support for either ZFS or LUKS containers
- This signed bootloader only loads configuration from ``/boot/efi/EFI/fedora/grub.cfg``
Unrelated to Secure Boot, GRUB has also been modified to provide optional
support for `systemd bootloader specification (bls) <https://systemd.io/BOOT_LOADER_SPECIFICATION/>`__.
Currently ``blscfg.mod`` is incompatible with root on ZFS.
#. Notes for GRUB on RHEL
As bls is disabled, you will need to regenerate GRUB menu after each kernel upgrade.
Or else the new kernel will not be recognized and system will boot the old kernel
on reboot.
Also see `Fedora docs for GRUB
<https://docs.fedoraproject.org/en-US/fedora/rawhide/system-administrators-guide/kernel-module-driver-configuration/Working_with_the_GRUB_2_Boot_Loader/>`__.
Finish Installation
~~~~~~~~~~~~~~~~~~~~
@@ -154,85 +105,25 @@ Finish Installation
exit
#. Take a snapshot of the clean installation for future use::
zfs snapshot -r rpool_$INST_UUID/$INST_ID@install
zfs snapshot -r bpool_$INST_UUID/$INST_ID@install
#. Unmount EFI system partition::
umount /mnt/boot/efi
umount /mnt/boot/efis/*
#. Export pools::
zpool export bpool_$INST_UUID
zpool export rpool_$INST_UUID
umount -Rl /mnt
zpool export -a
#. Reboot::
reboot
#. On first reboot, the boot process will fail, with failure messages such
as "You are in Emergency Mode...Press Ctrl-D to continue".
Wait for the computer to automatically reboot and the problem will be resolved.
Post installaion
~~~~~~~~~~~~~~~~
#. If you have other data pools, generate list of datasets for `zfs-mount-generator
<https://manpages.ubuntu.com/manpages/focal/man8/zfs-mount-generator.8.html>`__ to mount them at boot::
DATA_POOL='tank0 tank1'
# tab-separated zfs properties
# see /etc/zfs/zed.d/history_event-zfs-list-cacher.sh
export \
PROPS="name,mountpoint,canmount,atime,relatime,devices,exec\
,readonly,setuid,nbmand,encroot,keylocation"
for i in $DATA_POOL; do
zfs list -H -t filesystem -o $PROPS -r $i > /etc/zfs/zfs-list.cache/$i
done
#. After reboot, consider adding a normal user::
# with root permissions
sudo -i
# store user name in a variable
myUser=UserName
# rename default `User` to new user name
zfs rename $(df --output=source /home | tail -n +2)/User $(df --output=source /home | tail -n +2)/${myUser}
# update entry in fstab
sed -i "s|/home/User|/home/${myUser}|g" /etc/fstab
# add user
useradd --no-create-home --user-group --home-dir /home/${myUser} --comment 'My Name' ${myUser}
# delegate snapshot and destroy permissions of the home dataset to
# new user
zfs allow -u ${myUser} mount,snapshot,destroy $(df --output=source /home | tail -n +2)/${myUser}
# fix permissions
chown --recursive ${myUser}:${myUser} /home/${myUser}
chmod 700 /home/${myUser}
# fix selinux context
restorecon /home/${myUser}
# set new password for user
passwd ${myUser}
Set up cron job to snapshot user home everyday::
dnf install cronie
systemctl enable --now crond
crontab -eu ${myUser}
#@daily /usr/sbin/zfs snap $(df --output=source /home/${myUser} | tail -n +2)@$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)
zfs list -t snapshot -S creation $(df --output=source /home/${myUser} | tail -n +2)
Install package groups::
#. Install package groups::
dnf group list --hidden -v # query package groups
dnf group install 'i3 Desktop'
dnf group install 'Fedora Workstation' # GNOME
dnf group install 'Web Server'
dnf group install @gnome-desktop
#. Add new user, configure swap.

211
docs/Getting Started/Fedora/Root on ZFS/6-recovery.rst
View File

@@ -1,211 +0,0 @@
.. highlight:: sh
Recovery
======================
.. contents:: Table of Contents
:local:
GRUB Tips
-------------
Boot from GRUB rescue
~~~~~~~~~~~~~~~~~~~~~~~
If bootloader file is damaged, it's still possible
to boot computer with GRUB rescue image.
This section is also applicable if you are in
``grub rescue>``.
#. On another computer, generate rescue image with::
pacman -S --needed mtools libisoburn grub
grub-install
grub-mkrescue -o grub-rescue.img
dd if=grub-rescue.img of=/dev/your-usb-stick
Boot computer from the rescue media.
Both legacy and EFI mode are supported.
Or `download generated GRUB rescue image <https://gitlab.com/m_zhou/bieaz/uploads/e0847a8675cda4317ea7f48abb1d9f10/grub-rescue-2.06.img.7z>`__.
#. List available disks with ``ls`` command::
grub> ls (hd # press tab
Possible devices are:
hd0 hd1 hd2 hd3
#. List partitions by pressing tab key:
.. code-block:: text
grub> ls (hd0 # press tab
Possible partitions are:
Device hd0: No known filesystem detected - Sector size 512B - Total size 20971520KiB
Partition hd0,gpt1: Filesystem type fat - Label `EFI', UUID 0DF5-3A76 - Partition start at 1024KiB - Total size 1048576KiB
Partition hd0,gpt2: No known filesystem detected - Partition start at 1049600KiB - Total size 4194304KiB
- If boot pool is encrypted:
Unlock it with ``cryptomount``::
grub> insmod luks
grub> cryptomount hd0,gpt2
Attempting to decrypt master key...
Enter passphrase for hd0,gpt2 (af5a240e13e24483acf02600d61e0f36):
Slot 1 opened
Unlocked LUKS container is ``(crypto0)``:
.. code-block:: text
grub> ls (crypto0)
Device crypto0: Filesystem type zfs - Label `bpool_ip3tdb' - Last modification
time 2021-05-03 12:14:08 Monday, UUID f14d7bdf89fe21fb - Sector size 512B -
Total size 4192256KiB
- If boot pool is not encrypted:
.. code-block:: text
grub> ls (hd0,gpt2)
Device hd0,gpt2: Filesystem type zfs - Label `bpool_ip3tdb' - Last modification
time 2021-05-03 12:14:08 Monday, UUID f14d7bdf89fe21fb - Sector size 512B -
Total size 4192256KiB
#. List boot environments nested inside ``bpool/$INST_ID/BOOT``::
grub> ls (crypto0)/sys/BOOT
@/ default/ be0/
#. Instruct GRUB to load configuration from ``be0`` boot environment::
grub> prefix=(crypto0)/sys/BOOT/be0/@/grub
grub> configfile $prefix/grub.cfg
#. GRUB menu should now appear.
#. After entering system, `reinstall GRUB <#grub-installation>`__.
Switch GRUB prefix when disk fails
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you are using LUKS encrypted boot pool with multiple disks,
the primary disk failed, GRUB will fail to load configuration.
If there's still enough redundancy for the boot pool, try fix
GRUB with the following method:
#. Ensure ``Slot 1 opened`` message
is shown
.. code-block:: text
Welcome to GRUB!
error: no such cryptodisk found.
Attempting to decrypt master key...
Enter passphrase for hd0,gpt2 (c0987ea1a51049e9b3056622804de62a):
Slot 1 opened
error: disk `cryptouuid/47ed1b7eb0014bc9a70aede3d8714faf' not found.
Entering rescue mode...
grub rescue>
If ``error: access denied.`` is shown,
try re-enter password with::
grub rescue> cryptomount hd0,gpt2
#. Check prefix::
grub rescue > set
# prefix=(cryptouuid/47ed1b7eb0014bc9a70aede3d8714faf)/sys/BOOT/be0@/grub
# root=cryptouuid/47ed1b7eb0014bc9a70aede3d8714faf
#. Set correct ``prefix`` and ``root`` by replacing
``cryptouuid/UUID`` with ``crypto0``::
grub rescue> prefix=(crypto0)/sys/BOOT/default@/grub
grub rescue> root=crypto0
#. Boot GRUB::
grub rescue> insmod normal
grub rescue> normal
GRUB should then boot normally.
#. After entering system, edit ``/etc/fstab`` to promote
one backup to ``/boot/efi``.
#. Make the change to ``prefix`` and ``root``
permanent by `reinstalling GRUB <#grub-installation>`__.
Access system in chroot
-----------------------
#. Go through `preparation <1-preparation.html>`__.
#. Import and unlock root and boot pool::
zpool import -NR /mnt rpool_$INST_UUID
zpool import -NR /mnt bpool_$INST_UUID
If using password::
zfs load-key rpool_$INST_UUID/$INST_ID
If using keyfile::
zfs load-key -L file:///path/to/keyfile rpool_$INST_UUID/$INST_ID
#. Find the current boot environment::
zfs list
BE=default
#. Mount root filesystem::
zfs mount rpool_$INST_UUID/$INST_ID/ROOT/$BE
#. chroot into the system::
arch-chroot /mnt /bin/bash --login
zfs mount -a
mount -a
#. Finish rescue. See `finish installation <#finish-installation>`__.
Backup and migrate existing installation
----------------------------------------
With the help of `zfs send
<https://openzfs.github.io/openzfs-docs/man/8/zfs-send.8.html>`__
it is relatively easy to perform a system backup and migration.
#. Create a snapshot of root file system::
zfs snapshot -r rpool/$INST_ID@backup
zfs snapshot -r bpool/$INST_ID@backup
#. Save snapshot to a file or pipe to SSH::
zfs send --options rpool/$INST_ID@backup > /backup/$INST_ID-rpool
zfs send --options bpool/$INST_ID@backup > /backup/$INST_ID-bpool
#. Re-create partitions and root/boot
pool on target system.
#. Restore backup::
zfs recv rpool_new/$INST_ID < /backup/$INST_ID-rpool
zfs recv bpool_new/$INST_ID < /backup/$INST_ID-bpool
#. Chroot and reinstall bootloader.
#. Update pool name in ``/etc/fstab``, ``/boot/grub/grub.cfg``
and ``/etc/zfs/zfs-list.cache/*``.
#. Update device name, etc, in ``/etc/fstab`` and ``/etc/crypttab``.

2
docs/Getting Started/Fedora/index.rst
View File

@@ -90,7 +90,7 @@ Root on ZFS
ZFS can be used as root file system for Fedora.
An installation guide is available.
`Start here <Root%20on%20ZFS/0-overview.html>`__.
`Start here <Root%20on%20ZFS/1-preparation.html>`__.
.. toctree::
:maxdepth: 1

2
docs/Getting Started/NixOS/Root on ZFS.rst
View File

@@ -1,6 +1,6 @@
NixOS Root on ZFS
=======================================
`Start here <Root%20on%20ZFS/0-overview.html>`__.
`Start here <Root%20on%20ZFS/1-preparation.html>`__.
Contents
--------

142
docs/Getting Started/NixOS/Root on ZFS/0-overview.rst
View File

@@ -1,142 +0,0 @@
.. highlight:: sh
Overview
======================
This document describes how to install NixOS with ZFS as root
file system.
Caution
~~~~~~~
- This guide wipes entire physical disks. Back up existing data.
- `GRUB does not and
will not work on 4Kn drive with legacy (BIOS) booting.
<http://savannah.gnu.org/bugs/?46700>`__
Partition layout
~~~~~~~~~~~~~~~~
GUID partition table (GPT) is used.
EFI system partition will be referred to as **ESP** in this document.
+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+
| Name | legacy boot | ESP | Boot pool | swap | root pool | remaining space |
+======================+======================+=======================+======================+=====================+=======================+=================+
| File system | | vfat | ZFS | swap | ZFS | |
+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+
| Size | 1M | 2G | 4G | depends on RAM size | | |
+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+
| Optional encryption | | *Secure Boot* | | plain dm-crypt | ZFS native encryption | |
| | | | | | | |
+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+
| Partition no. | 5 | 1 | 2 | 4 | 3 | |
+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+
| Mount point | | | /boot | | / | |
| | | /boot/efis/disk-part1 | | | | |
+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+
Dataset layout
~~~~~~~~~~~~~~
The dataset layout used in this guide follows stardard
mutable file positions (``/var``, ``/etc``, ...), but can
still be modified to `a immutable root <https://grahamc.com/blog/erase-your-darlings>`__
after installation.
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| Dataset | canmount | mountpoint | container | notes |
+===========================+======================+======================+=====================================+===========================================+
| bpool | off | /boot | contains sys | |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool | off | / | contains sys | |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| bpool/sys | off | none | contains BOOT | |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool/sys | off | none | contains ROOT | sys is encryptionroot |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| bpool/sys/BOOT | off | none | contains boot environments | |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool/sys/ROOT | off | none | contains boot environments | |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool/sys/DATA | off | none | contains placeholder "default" | |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool/sys/DATA/default | off | / | contains user datasets | child datsets inherits mountpoint |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool/sys/DATA/local | off | / | contains /nix datasets | child datsets inherits mountpoint |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool/sys/DATA/default/ | on | /home (inherited) | no | |
| home | | | | user datasets, also called "shared |
| | | | | datasets", "persistent datasets"; also |
| | | | | include /var/lib, /srv, ... |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| bpool/sys/BOOT/default | noauto | /boot | no | noauto is used to switch BE. because of |
| | | | | noauto, must use fstab to mount |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool/sys/ROOT/default | noauto | / | no | mounted by initrd zfs hook |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
Encryption
~~~~~~~~~~
- Swap
Swap is always encrypted. By default, swap is encrypted
with plain dm-crypt with key generated from ``/dev/urandom``
at every boot. Swap content does not persist between reboots.
- Root pool
ZFS native encryption can be optionally enabled for ``rpool/sys``
and child datasets.
User should be aware that, ZFS native encryption does not
encrypt some metadata of the datasets.
ZFS native encryption also does not change master key when ``zfs change-key`` is invoked.
Therefore, you should wipe the disk when password is compromised to protect confidentiality.
See `zfs-load-key.8 <https://openzfs.github.io/openzfs-docs/man/8/zfs-load-key.8.html>`__
and `zfs-change-key.8 <https://openzfs.github.io/openzfs-docs/man/8/zfs-change-key.8.html>`__
for more information regarding ZFS native encryption.
Encryption is enabled at dataset creation and can not be disabled later.
- Boot pool
Boot pool can not be encrypted.
- Bootloader
Bootloader can not be encrypted.
However, with Secure Boot, bootloader
can be verified by motherboard firmware to be untempered,
which should be sufficient for most purposes.
Secure Boot is not supported out-of-the-box due to ZFS module.
Booting with disk failure
~~~~~~~~~~~~~~~~~~~~~~~~~
This guide is written with disk failure in mind.
If disks used in Root on ZFS pool failed, but
sufficient redundancy for both root pool and boot pool
still exists, the system will still boot normally.
Swap partition on the failed disk will fail to mount,
after an 1m30s timeout.
This feature is useful for use cases such
as an unattended remote server.
Example:
- System has disks ``n>1``
- Installed with mirrored setup
- Mirrored setup can tolerate up to ``n-1`` disk failures
- Disconnect one or more disks, keep at least
one disk connected
- System still boots, but fails to mount swap and
EFI partition

80
docs/Getting Started/NixOS/Root on ZFS/1-preparation.rst
View File

@@ -6,41 +6,26 @@ Preparation
.. contents:: Table of Contents
:local:
#. Download `Minimal ISO image
<https://channels.nixos.org/nixos-22.05/latest-nixos-minimal-x86_64-linux.iso>`__ and boot from it.
#. Disable Secure Boot. ZFS modules can not be loaded if Secure Boot is enabled.
#. Download `NixOS Live Image
<https://channels.nixos.org/nixos-22.05/latest-nixos-gnome-x86_64-linux.iso>`__ and boot from it.
#. Connect to the Internet.
#. Set root password or ``/root/.ssh/authorized_keys``.
#. Start SSH server::
#. Connect to network. See `NixOS manual <https://nixos.org/manual/nixos/stable/index.html#sec-installation-booting>`__.
#. SSH server is enabled by default. To connect, set root password with::
sudo passwd
systemctl restart sshd
#. Connect from another computer::
ssh root@192.168.1.19
#. Unique pool suffix. ZFS expects pool names to be
unique, therefore it's recommended to create
pools with a unique suffix::
INST_UUID=$(dd if=/dev/urandom bs=1 count=100 2>/dev/null | tr -dc 'a-z0-9' | cut -c-6)
#. Identify this installation in ZFS filesystem path::
INST_ID=nixos
#. Root on ZFS configuration file name::
INST_CONFIG_FILE='zfs.nix'
#. Target disk
List available disks with::
ls /dev/disk/by-id/*
If using virtio as disk bus, use
``/dev/disk/by-path/*``.
If using virtio as disk bus, use ``/dev/disk/by-path/*``.
Declare disk array::
@@ -50,57 +35,8 @@ Preparation
DISK='/dev/disk/by-id/disk1'
#. Choose a primary disk. This disk will be used
for primary EFI partition, default to
first disk in the array::
INST_PRIMARY_DISK=$(echo $DISK | cut -f1 -d\ )
#. Set vdev topology, possible values are:
- (not set, single disk or striped; no redundancy)
- mirror
- raidz1
- raidz2
- raidz3
::
INST_VDEV=
This will create a single vdev with the topology of your choice.
It is also possible to manually create a pool with multiple vdevs, such as::
zpool create --options \
poolName \
mirror sda sdb \
raidz2 sdc ... \
raidz3 sde ... \
spare sdf ...
Notice the cost of parity when using RAID-Z. See
`here <https://www.delphix.com/blog/delphix-engineering/zfs-raidz-stripe-width-or-how-i-learned-stop-worrying-and-love-raidz>`__
and `here <https://docs.google.com/spreadsheets/d/1tf4qx1aMJp8Lo_R6gpT689wTjHv6CGVElrPqTA0w_ZY/>`__.
For boot pool, which must be readable by GRUB, mirrored vdev should always be used for maximum redundancy.
This guide will use mirrored bpool for multi-disk setup.
Refer to `zpoolconcepts <https://openzfs.github.io/openzfs-docs/man/7/zpoolconcepts.7.html>`__
and `zpool-create <https://openzfs.github.io/openzfs-docs/man/8/zpool-create.8.html>`__
man pages for details.
#. Set partition size:
Set ESP size::
INST_PARTSIZE_ESP=2 # in GB
Set boot pool size. To avoid running out of space while using
boot environments, the minimum is 4GB. Adjust the size if you
intend to use multiple kernel/distros::
INST_PARTSIZE_BPOOL=4
Set swap size. It's `recommended <https://chrisdown.name/2018/01/02/in-defence-of-swap.html>`__
to setup a swap partition. If you intend to use hibernation,
the minimum should be no less than RAM size. Skip if swap is not needed::

362
docs/Getting Started/NixOS/Root on ZFS/2-system-configuration.rst
View File

@@ -1,362 +0,0 @@
.. highlight:: sh
System Configuration
======================
.. contents:: Table of Contents
:local:
#. Optional: wipe solid-state drives with the generic tool
`blkdiscard <https://utcc.utoronto.ca/~cks/space/blog/linux/ErasingSSDsWithBlkdiscard>`__,
to clean previous partition tables and improve performance.
All content will be irrevocably destroyed::
for i in ${DISK}; do
blkdiscard -f $i &
done
wait
This is a quick operation and should be completed under one
minute.
For other device specific methods, see
`Memory cell clearing <https://wiki.archlinux.org/title/Solid_state_drive/Memory_cell_clearing>`__
#. Partition the disks.
See `Overview <0-overview.html>`__ for details::
for i in ${DISK}; do
sgdisk --zap-all $i
sgdisk -n1:1M:+${INST_PARTSIZE_ESP}G -t1:EF00 $i
sgdisk -n2:0:+${INST_PARTSIZE_BPOOL}G -t2:BE00 $i
if [ "${INST_PARTSIZE_SWAP}" != "" ]; then
sgdisk -n4:0:+${INST_PARTSIZE_SWAP}G -t4:8200 $i
fi
if [ "${INST_PARTSIZE_RPOOL}" = "" ]; then
sgdisk -n3:0:0 -t3:BF00 $i
else
sgdisk -n3:0:+${INST_PARTSIZE_RPOOL}G -t3:BF00 $i
fi
sgdisk -a1 -n5:24K:+1000K -t5:EF02 $i
done
#. Create boot pool::
disk_num=0; for i in $DISK; do disk_num=$(( $disk_num + 1 )); done
if [ $disk_num -gt 1 ]; then INST_VDEV_BPOOL=mirror; fi
zpool create \
-o compatibility=grub2 \
-o ashift=12 \
-o autotrim=on \
-O acltype=posixacl \
-O canmount=off \
-O compression=lz4 \
-O devices=off \
-O normalization=formD \
-O relatime=on \
-O xattr=sa \
-O mountpoint=/boot \
-R /mnt \
bpool_$INST_UUID \
$INST_VDEV_BPOOL \
$(for i in ${DISK}; do
printf "$i-part2 ";
done)
You should not need to customize any of the options for the boot pool.
GRUB does not support all of the zpool features. See ``spa_feature_names``
in `grub-core/fs/zfs/zfs.c
<http://git.savannah.gnu.org/cgit/grub.git/tree/grub-core/fs/zfs/zfs.c#n276>`__.
This step creates a separate boot pool for ``/boot`` with the features
limited to only those that GRUB supports, allowing the root pool to use
any/all features.
Features enabled with ``-o compatibility=grub2`` can be seen
`here <https://github.com/openzfs/zfs/blob/master/cmd/zpool/compatibility.d/grub2>`__.
#. Create root pool::
zpool create \
-o ashift=12 \
-o autotrim=on \
-R /mnt \
-O acltype=posixacl \
-O canmount=off \
-O compression=zstd \
-O dnodesize=auto \
-O normalization=formD \
-O relatime=on \
-O xattr=sa \
-O mountpoint=/ \
rpool_$INST_UUID \
$INST_VDEV \
$(for i in ${DISK}; do
printf "$i-part3 ";
done)
**Notes:**
- The use of ``ashift=12`` is recommended here because many drives
today have 4 KiB (or larger) physical sectors, even though they
present 512 B logical sectors. Also, a future replacement drive may
have 4 KiB physical sectors (in which case ``ashift=12`` is desirable)
or 4 KiB logical sectors (in which case ``ashift=12`` is required).
- Setting ``-O acltype=posixacl`` enables POSIX ACLs globally. If you
do not want this, remove that option, but later add
``-o acltype=posixacl`` (note: lowercase “o”) to the ``zfs create``
for ``/var/log``, as `journald requires ACLs
<https://askubuntu.com/questions/970886/journalctl-says-failed-to-search-journal-acl-operation-not-supported>`__
- Setting ``normalization=formD`` eliminates some corner cases relating
to UTF-8 filename normalization. It also implies ``utf8only=on``,
which means that only UTF-8 filenames are allowed. If you care to
support non-UTF-8 filenames, do not use this option. For a discussion
of why requiring UTF-8 filenames may be a bad idea, see `The problems
with enforced UTF-8 only filenames
<http://utcc.utoronto.ca/~cks/space/blog/linux/ForcedUTF8Filenames>`__.
- ``recordsize`` is unset (leaving it at the default of 128 KiB). If you
want to tune it (e.g. ``-o recordsize=1M``), see `these
<https://jrs-s.net/2019/04/03/on-zfs-recordsize/>`__ `various
<http://blog.programster.org/zfs-record-size>`__ `blog
<https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSFileRecordsizeGrowth>`__
`posts
<https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSRecordsizeAndCompression>`__.
- Setting ``relatime=on`` is a middle ground between classic POSIX
``atime`` behavior (with its significant performance impact) and
``atime=off`` (which provides the best performance by completely
disabling atime updates). Since Linux 2.6.30, ``relatime`` has been
the default for other filesystems. See `RedHat’s documentation
<https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/power_management_guide/relatime>`__
for further information.
- Setting ``xattr=sa`` `vastly improves the performance of extended
attributes
<https://github.com/zfsonlinux/zfs/commit/82a37189aac955c81a59a5ecc3400475adb56355>`__.
Inside ZFS, extended attributes are used to implement POSIX ACLs.
Extended attributes can also be used by user-space applications.
`They are used by some desktop GUI applications.
<https://en.wikipedia.org/wiki/Extended_file_attributes#Linux>`__
`They can be used by Samba to store Windows ACLs and DOS attributes;
they are required for a Samba Active Directory domain controller.
<https://wiki.samba.org/index.php/Setting_up_a_Share_Using_Windows_ACLs>`__
Note that ``xattr=sa`` is `Linux-specific
<https://openzfs.org/wiki/Platform_code_differences>`__. If you move your
``xattr=sa`` pool to another OpenZFS implementation besides ZFS-on-Linux,
extended attributes will not be readable (though your data will be). If
portability of extended attributes is important to you, omit the
``-O xattr=sa`` above. Even if you do not want ``xattr=sa`` for the whole
pool, it is probably fine to use it for ``/var/log``.
- Make sure to include the ``-part3`` portion of the drive path. If you
forget that, you are specifying the whole disk, which ZFS will then
re-partition, and you will lose the bootloader partition(s).
#. This section implements dataset layout as described in `overview <0-overview.html>`__.
Create root system container:
- Unencrypted::
zfs create \
-o canmount=off \
-o mountpoint=none \
rpool_$INST_UUID/$INST_ID
- Encrypted:
Pick a strong password. Once compromised, changing password will not keep your
data safe. See ``zfs-change-key(8)`` for more info::
zfs create \
-o canmount=off \
-o mountpoint=none \
-o encryption=on \
-o keylocation=prompt \
-o keyformat=passphrase \
rpool_$INST_UUID/$INST_ID
Create other system datasets::
zfs create -o canmount=off -o mountpoint=none bpool_$INST_UUID/$INST_ID
zfs create -o canmount=off -o mountpoint=none bpool_$INST_UUID/$INST_ID/BOOT
zfs create -o canmount=off -o mountpoint=none rpool_$INST_UUID/$INST_ID/ROOT
zfs create -o canmount=off -o mountpoint=none rpool_$INST_UUID/$INST_ID/DATA
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$INST_UUID/$INST_ID/BOOT/default
zfs create -o mountpoint=/ -o canmount=off rpool_$INST_UUID/$INST_ID/DATA/default
zfs create -o mountpoint=/ -o canmount=off rpool_$INST_UUID/$INST_ID/DATA/local
zfs create -o mountpoint=/ -o canmount=noauto rpool_$INST_UUID/$INST_ID/ROOT/default
zfs mount rpool_$INST_UUID/$INST_ID/ROOT/default
zfs mount bpool_$INST_UUID/$INST_ID/BOOT/default
for i in {usr,var,var/lib};
do
zfs create -o canmount=off rpool_$INST_UUID/$INST_ID/DATA/default/$i
done
for i in {home,root,srv,usr/local,var/log,var/spool};
do
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/$i
done
chmod 750 /mnt/root
for i in {nix,}; do
zfs create -o canmount=on -o mountpoint=/$i rpool_$INST_UUID/$INST_ID/DATA/local/$i
done
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/state
for i in {/etc/nixos,/etc/cryptkey.d}; do
mkdir -p /mnt/state/$i /mnt/$i
mount -o bind /mnt/state/$i /mnt/$i
done
zfs create -o mountpoint=/ -o canmount=noauto rpool_$INST_UUID/$INST_ID/ROOT/empty
zfs snapshot rpool_$INST_UUID/$INST_ID/ROOT/empty@start
#. Format and mount ESP::
for i in ${DISK}; do
mkfs.vfat -n EFI ${i}-part1
mkdir -p /mnt/boot/efis/${i##*/}-part1
mount -t vfat ${i}-part1 /mnt/boot/efis/${i##*/}-part1
done
#. Create optional user data datasets to omit data from rollback::
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/games
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/www
# for GNOME
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/AccountsService
# for Docker
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/docker
# for NFS
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/nfs
# for LXC
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/lxc
# for LibVirt
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/libvirt
##other application
# zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/$name
Add other datasets when needed, such as PostgreSQL.
#. Generate initial NixOS system configuration::
nixos-generate-config --root /mnt
This command will generate two files, ``configuration.nix``
and ``hardware-configuration-zfs.nix``, which will be the starting point
of configuring the system.
#. Edit config file to import ZFS options::
sed -i "s|./hardware-configuration.nix|./hardware-configuration-zfs.nix ./${INST_CONFIG_FILE}|g" /mnt/etc/nixos/configuration.nix
# backup, prevent being overwritten by nixos-generate-config
mv /mnt/etc/nixos/hardware-configuration.nix /mnt/etc/nixos/hardware-configuration-zfs.nix
#. ZFS options::
tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} <<EOF
{ config, pkgs, ... }:
{ boot.supportedFilesystems = [ "zfs" ];
networking.hostId = "$(head -c 8 /etc/machine-id)";
boot.zfs.devNodes = "${INST_PRIMARY_DISK%/*}";
EOF
ZFS datasets should be mounted with ``-o zfsutil`` option::
sed -i 's|fsType = "zfs";|fsType = "zfs"; options = [ "zfsutil" "X-mount.mkdir" ];|g' \
/mnt/etc/nixos/hardware-configuration-zfs.nix
Allow EFI system partition mounting to fail at boot::
sed -i 's|fsType = "vfat";|fsType = "vfat"; options = [ "x-systemd.idle-timeout=1min" "x-systemd.automount" "noauto" ];|g' \
/mnt/etc/nixos/hardware-configuration-zfs.nix
Restrict kernel to versions supported by ZFS::
tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} <<EOF
boot.kernelPackages = config.boot.zfs.package.latestCompatibleLinuxPackages;
EOF
Disable cache::
mkdir -p /mnt/state/etc/zfs/
rm -f /mnt/state/etc/zfs/zpool.cache
touch /mnt/state/etc/zfs/zpool.cache
chmod a-w /mnt/state/etc/zfs/zpool.cache
chattr +i /mnt/state/etc/zfs/zpool.cache
#. If swap is enabled::
if [ "${INST_PARTSIZE_SWAP}" != "" ]; then
sed -i '/swapDevices/d' /mnt/etc/nixos/hardware-configuration-zfs.nix
tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} <<EOF
swapDevices = [
EOF
for i in $DISK; do
tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} <<EOF
{ device = "$i-part4"; randomEncryption.enable = true; }
EOF
done
tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} <<EOF
];
EOF
fi
#. For immutable root file system, save machine-id and other files::
mkdir -p /mnt/state/etc/{ssh,zfs}
systemd-machine-id-setup --print > /mnt/state/etc/machine-id
tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} <<EOF
systemd.services.zfs-mount.enable = false;
environment.etc."machine-id".source = "/state/etc/machine-id";
environment.etc."aliases".source = "/state/etc/aliases";
environment.etc."zfs/zpool.cache".source
= "/state/etc/zfs/zpool.cache";
boot.loader.efi.efiSysMountPoint = "/boot/efis/${INST_PRIMARY_DISK##*/}-part1";
EOF
touch /mnt/state/etc/aliases
#. Configure GRUB boot loader for both legacy boot and UEFI::
sed -i '/boot.loader/d' /mnt/etc/nixos/configuration.nix
tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} <<-'EOF'
boot.loader.efi.canTouchEfiVariables = false;
##if UEFI firmware can detect entries
#boot.loader.efi.canTouchEfiVariables = true;
boot.loader = {
generationsDir.copyKernels = true;
##for problematic UEFI firmware
grub.efiInstallAsRemovable = true;
grub.enable = true;
grub.version = 2;
grub.copyKernels = true;
grub.efiSupport = true;
grub.zfsSupport = true;
# for systemd-autofs
grub.extraPrepareConfig = ''
mkdir -p /boot/efis
for i in /boot/efis/*; do mount $i ; done
'';
grub.extraInstallCommands = ''
export ESP_MIRROR=$(mktemp -d -p /tmp)
EOF
tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} <<EOF
cp -r /boot/efis/${INST_PRIMARY_DISK##*/}-part1/EFI \$ESP_MIRROR
EOF
tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} <<-'EOF'
for i in /boot/efis/*; do
cp -r $ESP_MIRROR/EFI $i
done
rm -rf $ESP_MIRROR
'';
grub.devices = [
EOF
for i in $DISK; do
printf " \"$i\"\n" >>/mnt/etc/nixos/${INST_CONFIG_FILE}
done
tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} <<EOF
];
};
EOF

132
docs/Getting Started/NixOS/Root on ZFS/2-system-installation.rst Normal file
View File

@@ -0,0 +1,132 @@
.. highlight:: sh
System Installation
======================
.. contents:: Table of Contents
:local:
#. Partition the disks::
for i in ${DISK}; do
sgdisk --zap-all $i
sgdisk -n1:1M:+1G -t1:EF00 $i
sgdisk -n2:0:+4G -t2:BE00 $i
test -z $INST_PARTSIZE_SWAP || sgdisk -n4:0:+${INST_PARTSIZE_SWAP}G -t4:8200 $i
if test -z $INST_PARTSIZE_RPOOL; then
sgdisk -n3:0:0 -t3:BF00 $i
else
sgdisk -n3:0:+${INST_PARTSIZE_RPOOL}G -t3:BF00 $i
fi
sgdisk -a1 -n5:24K:+1000K -t5:EF02 $i
done
#. Create boot pool::
zpool create \
-o compatibility=grub2 \
-o ashift=12 \
-o autotrim=on \
-O acltype=posixacl \
-O canmount=off \
-O compression=lz4 \
-O devices=off \
-O normalization=formD \
-O relatime=on \
-O xattr=sa \
-O mountpoint=/boot \
-R /mnt \
bpool \
mirror \
$(for i in ${DISK}; do
printf "$i-part2 ";
done)
If not using a multi-disk setup, remove ``mirror``.
You should not need to customize any of the options for the boot pool.
GRUB does not support all of the zpool features. See ``spa_feature_names``
in `grub-core/fs/zfs/zfs.c
<http://git.savannah.gnu.org/cgit/grub.git/tree/grub-core/fs/zfs/zfs.c#n276>`__.
This step creates a separate boot pool for ``/boot`` with the features
limited to only those that GRUB supports, allowing the root pool to use
any/all features.
Features enabled with ``-o compatibility=grub2`` can be seen
`here <https://github.com/openzfs/zfs/blob/master/cmd/zpool/compatibility.d/grub2>`__.
#. Create root pool::
zpool create \
-o ashift=12 \
-o autotrim=on \
-R /mnt \
-O acltype=posixacl \
-O canmount=off \
-O compression=zstd \
-O dnodesize=auto \
-O normalization=formD \
-O relatime=on \
-O xattr=sa \
-O mountpoint=/ \
rpool \
mirror \
$(for i in ${DISK}; do
printf "$i-part3 ";
done)
If not using a multi-disk setup, remove ``mirror``.
#. This section implements dataset layout as described in `overview <1-preparation.html>`__.
Create root system container:
- Unencrypted::
zfs create \
-o canmount=off \
-o mountpoint=none \
rpool/nixos
- Encrypted:
Pick a strong password. Once compromised, changing password will not keep your
data safe. See ``zfs-change-key(8)`` for more info::
zfs create \
-o canmount=off \
-o mountpoint=none \
-o encryption=on \
-o keylocation=prompt \
-o keyformat=passphrase \
rpool/nixos
Create system datasets::
zfs create -o canmount=on -o mountpoint=/ rpool/nixos/root
zfs create -o canmount=on -o mountpoint=/home rpool/nixos/home
zfs create -o canmount=off -o mountpoint=/var rpool/nixos/var
zfs create -o canmount=on rpool/nixos/var/lib
zfs create -o canmount=on rpool/nixos/var/log
Create boot dataset::
zfs create -o canmount=on -o mountpoint=/boot bpool/nixos
#. Format and mount ESP::
for i in ${DISK}; do
mkfs.vfat -n EFI ${i}-part1
mkdir -p /mnt/boot/efis/${i##*/}-part1
mount -t vfat ${i}-part1 /mnt/boot/efis/${i##*/}-part1
done
mkdir -p /mnt/boot/efi
mount -t vfat $(echo $DISK | cut -f1 -d\ )-part1 /mnt/boot/efi

235
docs/Getting Started/NixOS/Root on ZFS/3-optional-configuration.rst
View File

@@ -1,235 +0,0 @@
.. highlight:: sh
Optional Configuration
======================
.. contents:: Table of Contents
:local:
Skip to `System Installation <./4-system-installation.html>`__ section if
no optional configuration is needed.
Mail notification for ZFS status
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
For headless applications such as NAS, it is useful to set up mail notification
for hardware changes and monitor for scrub results.
#. Set up an alias for root account::
tee -a /state/etc/aliases <<EOF
root: user@example.com
EOF
#. Set up mail transfer agent, the program that sends email::
programs.msmtp = {
enable = true;
setSendmail = true;
defaults = {
aliases = "/state/etc/aliases";
port = 465;
tls_trust_file = "/etc/ssl/certs/ca-certificates.crt";
tls = "on";
auth = "plain";
tls_starttls = "off";
};
accounts = {
default = {
host = "mail.example.com";
# set secure permissions for password file
passwordeval = "cat /state/etc/emailpass.txt";
user = "user@example.com";
from = "user@example.com";
};
};
};
#. Enable mail notification for ZFS Event Daemon::
services.zfs.zed.settings = {
ZED_EMAIL_ADDR = [ "root" ];
ZED_EMAIL_PROG = "${pkgs.msmtp}/bin/msmtp";
ZED_EMAIL_OPTS = "@ADDRESS@";
ZED_NOTIFY_VERBOSE = true;
};
# this option does not work
services.zfs.zed.enableMail = false;
Supply password with SSH
~~~~~~~~~~~~~~~~~~~~~~~~
Note: if you choose to encrypt boot pool, where decryption is handled
by GRUB, as described in the next section, configuration performed
in this section will have no effect.
This example uses DHCP::
mkdir -p /mnt/etc/state/ssh/
ssh-keygen -t ed25519 -N "" -f /mnt/state/etc/ssh/ssh_host_initrd_ed25519_key
tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} <<EOF
#networking.interfaces.enp1s0.useDHCP = true;
boot = {
initrd.network = {
enable = true;
ssh = {
enable = true;
hostKeys = [ /state/etc/ssh/ssh_host_initrd_ed25519_key ];
authorizedKeys = [ "$YOUR_PUBLIC_KEY" ];
};
postCommands = ''
echo "zfs load-key -a; killall zfs" >> /root/.profile
'';
};
};
EOF
Encrypt boot pool
~~~~~~~~~~~~~~~~~~~
Note: This will disable password with SSH. The password previously set for
root pool will be replaced by keyfile, embedded in initrd.
#. Add package::
tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} <<EOF
environment.systemPackages = [ pkgs.cryptsetup ];
EOF
#. LUKS password::
LUKS_PWD=secure-passwd
You will need to enter the same password for
each disk at boot. As root pool key is
protected by this password, the previous warning
about password strength still apply.
Double-check password here. Complete reinstallation is
needed if entered wrong.
#. Create encryption keys::
mkdir -p /mnt/etc/cryptkey.d/
chmod 700 /mnt/etc/cryptkey.d/
dd bs=32 count=1 if=/dev/urandom of=/mnt/etc/cryptkey.d/rpool_$INST_UUID-${INST_ID}-key-zfs
dd bs=32 count=1 if=/dev/urandom of=/mnt/etc/cryptkey.d/bpool_$INST_UUID-key-luks
chmod u=r,go= /mnt/etc/cryptkey.d/*
#. Backup boot pool::
zfs snapshot -r bpool_$INST_UUID/$INST_ID@pre-luks
zfs send -Rv bpool_$INST_UUID/$INST_ID@pre-luks > /mnt/root/bpool_$INST_UUID-${INST_ID}-pre-luks
#. Unmount EFI partition::
for i in ${DISK}; do
umount /mnt/boot/efis/${i##*/}-part1
done
#. Destroy boot pool::
zpool destroy bpool_$INST_UUID
#. Create LUKS containers::
for i in ${DISK}; do
cryptsetup luksFormat -q --type luks1 --key-file /mnt/etc/cryptkey.d/bpool_$INST_UUID-key-luks $i-part2
echo $LUKS_PWD | cryptsetup luksAddKey --key-file /mnt/etc/cryptkey.d/bpool_$INST_UUID-key-luks $i-part2
cryptsetup open ${i}-part2 ${i##*/}-part2-luks-bpool_$INST_UUID --key-file /mnt/etc/cryptkey.d/bpool_$INST_UUID-key-luks
tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} <<EOF
boot.initrd.luks.devices = {
"${i##*/}-part2-luks-bpool_$INST_UUID" = {
device = "${i}-part2";
allowDiscards = true;
keyFile = "/etc/cryptkey.d/bpool_$INST_UUID-key-luks";
};
};
EOF
done
GRUB 2.06 still does not have complete support for LUKS2, LUKS1
is used instead.
#. Embed key file in initrd::
tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} <<EOF
boot.initrd.secrets = {
"/etc/cryptkey.d/rpool_$INST_UUID-${INST_ID}-key-zfs" = "/etc/cryptkey.d/rpool_$INST_UUID-${INST_ID}-key-zfs";
"/etc/cryptkey.d/bpool_$INST_UUID-key-luks" = "/etc/cryptkey.d/bpool_$INST_UUID-key-luks";
};
EOF
#. Recreate boot pool with mappers as vdev::
disk_num=0; for i in $DISK; do disk_num=$(( $disk_num + 1 )); done
if [ $disk_num -gt 1 ]; then INST_VDEV_BPOOL=mirror; fi
zpool create \
-d -o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-o ashift=12 \
-o autotrim=on \
-O acltype=posixacl \
-O canmount=off \
-O compression=lz4 \
-O devices=off \
-O normalization=formD \
-O relatime=on \
-O xattr=sa \
-O mountpoint=/boot \
-R /mnt \
bpool_$INST_UUID \
$INST_VDEV_BPOOL \
$(for i in ${DISK}; do
printf "/dev/mapper/${i##*/}-part2-luks-bpool_$INST_UUID ";
done)
#. Restore boot pool backup::
zfs recv bpool_${INST_UUID}/${INST_ID} < /mnt/root/bpool_$INST_UUID-${INST_ID}-pre-luks
rm /mnt/root/bpool_$INST_UUID-${INST_ID}-pre-luks
#. Mount boot dataset and EFI partitions::
zfs mount bpool_$INST_UUID/$INST_ID/BOOT/default
for i in ${DISK}; do
mount ${i}-part1 /mnt/boot/efis/${i##*/}-part1
done
#. As keys are stored in initrd,
set secure permissions for ``/boot``::
chmod 700 /mnt/boot
#. Change root pool password to key file::
mkdir -p /etc/cryptkey.d/
cp /mnt/etc/cryptkey.d/rpool_$INST_UUID-${INST_ID}-key-zfs /etc/cryptkey.d/rpool_$INST_UUID-${INST_ID}-key-zfs
zfs change-key -l \
-o keylocation=file:///etc/cryptkey.d/rpool_$INST_UUID-${INST_ID}-key-zfs \
-o keyformat=raw \
rpool_$INST_UUID/$INST_ID
#. Enable GRUB cryptodisk::
tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} <<EOF
boot.loader.grub.enableCryptodisk = true;
EOF
#. **Important**: Back up root dataset key ``/etc/cryptkey.d/rpool_$INST_UUID-${INST_ID}-key-zfs``
to a secure location.
In the possible event of LUKS container corruption,
data on root set will only be available
with this key.

102
docs/Getting Started/NixOS/Root on ZFS/3-system-configuration.rst Normal file
View File

@@ -0,0 +1,102 @@
.. highlight:: sh
System Configuration
======================
.. contents:: Table of Contents
:local:
#. Disable cache, stale cache will prevent system from booting::
mkdir -p /mnt/state/etc/zfs/
rm -f /mnt/state/etc/zfs/zpool.cache
touch /mnt/state/etc/zfs/zpool.cache
chmod a-w /mnt/state/etc/zfs/zpool.cache
chattr +i /mnt/state/etc/zfs/zpool.cache
#. Generate initial system configuration::
nixos-generate-config --root /mnt
#. Import ZFS-specific configuration::
sed -i "s|./hardware-configuration.nix|./hardware-configuration.nix ./zfs.nix|g" /mnt/etc/nixos/configuration.nix
#. Configure hostid::
tee -a /mnt/etc/nixos/zfs.nix <<EOF
{ config, pkgs, ... }:
{ boot.supportedFilesystems = [ "zfs" ];
networking.hostId = "$(head -c 8 /etc/machine-id)";
boot.kernelPackages = config.boot.zfs.package.latestCompatibleLinuxPackages;
EOF
#. Configure bootloader for both legacy boot and UEFI boot and mirror bootloader::
sed -i '/boot.loader/d' /mnt/etc/nixos/configuration.nix
sed -i '/services.xserver/d' /mnt/etc/nixos/configuration.nix
tee -a /mnt/etc/nixos/zfs.nix <<-'EOF'
boot.loader.efi.efiSysMountPoint = "/boot/efi";
boot.loader.efi.canTouchEfiVariables = false;
boot.loader.generationsDir.copyKernels = true;
boot.loader.grub.efiInstallAsRemovable = true;
boot.loader.grub.enable = true;
boot.loader.grub.version = 2;
boot.loader.grub.copyKernels = true;
boot.loader.grub.efiSupport = true;
boot.loader.grub.zfsSupport = true;
boot.loader.grub.extraPrepareConfig = ''
mkdir -p /boot/efis
for i in /boot/efis/*; do mount $i ; done
mkdir -p /boot/efi
mount /boot/efi
'';
boot.loader.grub.extraInstallCommands = ''
ESP_MIRROR=$(mktemp -d)
cp -r /boot/efi/EFI $ESP_MIRROR
for i in /boot/efis/*; do
cp -r $ESP_MIRROR/EFI $i
done
rm -rf $ESP_MIRROR
'';
boot.loader.grub.devices = [
EOF
for i in $DISK; do
printf " \"$i\"\n" >>/mnt/etc/nixos/zfs.nix
done
tee -a /mnt/etc/nixos/zfs.nix <<EOF
];
EOF
#. Mount datasets with zfsutil option::
sed -i 's|fsType = "zfs";|fsType = "zfs"; options = [ "zfsutil" "X-mount.mkdir" ];|g' \
/mnt/etc/nixos/hardware-configuration.nix
#. Set root password::
rootPwd=$(mkpasswd -m SHA-512 -s)
Declare password in configuration::
tee -a /mnt/etc/nixos/zfs.nix <<EOF
users.users.root.initialHashedPassword = "${rootPwd}";
}
EOF
#. Install system and apply configuration::
nixos-install -v --show-trace --no-root-passwd --root /mnt
#. Unmount filesystems::
umount -Rl /mnt
zpool export -a
#. Reboot::
reboot

235
docs/Getting Started/NixOS/Root on ZFS/4-system-installation.rst
View File

@@ -1,235 +0,0 @@
.. highlight:: sh
System Installation
======================
.. contents:: Table of Contents
:local:
Additional configuration
~~~~~~~~~~~~~~~~~~~~~~~~~
As NixOS configuration is declarative, post-installation tasks,
such as user accounts and package selection, can all be done by
specifing them in configuration. See `NixOS manual <https://nixos.org/nixos/manual/>`__
for details.
For timezone, hostname, networking, keyboard layout, etc,
see ``/mnt/etc/nixos/configuration.nix``.
Set root password
-----------------
This optional step is an example
of declaratively configuring the system.
#. Generate password hash::
INST_ROOT_PASSWD=$(mkpasswd -m SHA-512 -s)
#. Declare `initialHashedPassword
<https://nixos.org/manual/nixos/stable/options.html#opt-users.users._name_.initialHashedPassword>`__
for root user::
tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} <<EOF
users.users.root.initialHashedPassword = "${INST_ROOT_PASSWD}";
EOF
System installation
~~~~~~~~~~~~~~~~~~~
#. Finalize the config file::
tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} <<EOF
}
EOF
#. Take a snapshot of the clean installation, without state
for future use::
zfs snapshot -r rpool_$INST_UUID/$INST_ID@install_start
zfs snapshot -r bpool_$INST_UUID/$INST_ID@install_start
#. Back up bash inputs::
history -w /mnt/home/sys-install-pre-chroot.txt
#. Apply configuration
If root password hash is not set::
nixos-install -v --show-trace --root /mnt
You will be prompted for a new root password.
If password hash has been set::
nixos-install -v --show-trace --no-root-passwd --root /mnt
#. If boot pool encryption is used and installation fails with::
#mktemp: failed to create directory via template
#‘/mnt/tmp.coRUoqzl1P/initrd-secrets.XXXXXXXXXX’: No such file or directory
#failed to create initrd secrets: No such file or directory
This is `a bug <https://github.com/NixOS/nixpkgs/issues/157989>`__.
Complete the installation by executing::
nixos-enter --root /mnt -- nixos-rebuild boot
Finish installation
~~~~~~~~~~~~~~~~~~~~
#. Take a snapshot of the clean installation for future use::
zfs snapshot -r rpool_$INST_UUID/$INST_ID@install
zfs snapshot -r bpool_$INST_UUID/$INST_ID@install
#. Unmount EFI system partition::
umount /mnt/boot/efis/*
#. Export pools::
zpool export bpool_$INST_UUID
zpool export rpool_$INST_UUID
#. Reboot::
reboot
Upgrade NixOS
~~~~~~~~~~~~~
Routine updates within the same major version
=============================================
Updates within the same major version, such as from [21.11].001 to
[21.11].100, can be done with one of the following commands::
# take immediate effect
nixos-rebuild --upgrade switch
# update upon reboot
nixos-rebuild --upgrade boot
Upgrade to a newer major version
================================
Upgrading to a newer major version involves switching software
distribution channel.
#. View existing channels, run as root::
nix-channel --list
#nixos https://nixos.org/channels/nixos-21.11
#this is the major version released around November 2021
#. View available channels::
w3m https://hydra.nixos.org/project/nixos
#. Switch to a newer channel (22.05)::
nix-channel --add https://nixos.org/channels/nixos-22.05 nixos
#. In ``/etc/nixos/configuration.nix``::
system.stateVersion = "22.05";
If using Home Manager, in ``~/.config/nixpkgs/home.nix``::
home.stateVersion = "22.05";
#. Then follow the procedures for updating witin minor versions.
Immutable root file system
~~~~~~~~~~~~~~~~~~~~~~~~~~
This section is optional.
Often, programs generate mutable files in paths such as
``/etc`` and ``/var/lib``. The generated files can be considered a
part of the system state.
This generated state is not declaratively managed
by NixOS and can not be reproduced from NixOS configuration.
To ensure that the system state is fully managed by NixOS and reproducible,
we need to periodically purge the system state and let NixOS
regenerate root file system from scratch.
Also see: `Erase your darlings:
immutable infrastructure for mutable systems <https://grahamc.com/blog/erase-your-darlings>`__.
Save mutable data to alternative path
-------------------------------------
Before enabling purging on root dataset, we need to back up
essential mutable data first, such as host SSH key and network connections.
Below are some tips.
- Some programs support specifying another
location for mutable data, such as
Wireguard::
networking.wireguard.interfaces.wg0.privateKeyFile = "/state/etc/wireguard/wg0";
- For programs without a configurable data path,
`environment.etc <https://nixos.org/manual/nixos/stable/options.html#opt-environment.etc>`__
may be used::
environment.etc = {
"ssh/ssh_host_rsa_key".source = "/state/etc/ssh/ssh_host_rsa_key";
}
- systemd’s tmpfiles.d rules are also an option::
systemd.tmpfiles.rules = [
"L /var/lib/bluetooth - - - - /state/var/lib/bluetooth"
];
- Bind mount::
for i in {/etc/nixos,/etc/cryptkey.d}; do
mkdir -p /state/$i /$i
mount -o bind /state/$i /$i
done
nixos-generate-config --show-hardware-config
Boot from empty root file system
--------------------------------
After backing up mutable data, you can try switching to
an empty dataset as root file system.
#. Check current root file system::
ROOT_FS=$(df --output=source /|tail -n1)
# rpool/ROOT/default
#. Set empty file system as root::
sed -i "s,${ROOT_FS},${ROOT_FS%/*}/empty,g" /etc/nixos/hardware-configuration-zfs.nix
#. Apply changes and reboot::
nixos-rebuild boot
reboot
#. If everything went fine, add the output of the following command to configuration::
ROOT_FS=$(df --output=source /|tail -n1)
cat <<EOF
boot.initrd.postDeviceCommands = ''
zpool import -Nf ${ROOT_FS%%/*}
zfs rollback -r ${ROOT_FS%/*}/empty@start
'';
EOF
#. Apply and reboot::
nixos-rebuild boot
reboot

145
docs/Getting Started/NixOS/Root on ZFS/6-recovery.rst
View File

@@ -1,145 +0,0 @@
.. highlight:: sh
Recovery
======================
.. contents:: Table of Contents
:local:
GRUB Tips
-------------
Boot from GRUB rescue
~~~~~~~~~~~~~~~~~~~~~~~
If bootloader file is damaged, it's still possible
to boot computer with GRUB rescue image.
This section is also applicable if you are in
``grub rescue>``.
#. On another computer, generate rescue image with::
pacman -S --needed mtools libisoburn grub
grub-install
grub-mkrescue -o grub-rescue.img
dd if=grub-rescue.img of=/dev/your-usb-stick
Boot computer from the rescue media.
Both legacy and EFI mode are supported.
Or `download generated GRUB rescue image <https://gitlab.com/m_zhou/bieaz/uploads/e0847a8675cda4317ea7f48abb1d9f10/grub-rescue-2.06.img.7z>`__.
#. List available disks with ``ls`` command::
grub> ls (hd # press tab
Possible devices are:
hd0 hd1 hd2 hd3
#. List partitions by pressing tab key:
.. code-block:: text
grub> ls (hd0 # press tab
Possible partitions are:
Device hd0: No known filesystem detected - Sector size 512B - Total size 20971520KiB
Partition hd0,gpt1: Filesystem type fat - Label `EFI', UUID 0DF5-3A76 - Partition start at 1024KiB - Total size 1048576KiB
Partition hd0,gpt2: No known filesystem detected - Partition start at 1049600KiB - Total size 4194304KiB
- If boot pool is encrypted:
Unlock it with ``cryptomount``::
grub> insmod luks
grub> cryptomount hd0,gpt2
Attempting to decrypt master key...
Enter passphrase for hd0,gpt2 (af5a240e13e24483acf02600d61e0f36):
Slot 1 opened
Unlocked LUKS container is ``(crypto0)``:
.. code-block:: text
grub> ls (crypto0)
Device crypto0: Filesystem type zfs - Label `bpool_ip3tdb' - Last modification
time 2021-05-03 12:14:08 Monday, UUID f14d7bdf89fe21fb - Sector size 512B -
Total size 4192256KiB
- If boot pool is not encrypted:
.. code-block:: text
grub> ls (hd0,gpt2)
Device hd0,gpt2: Filesystem type zfs - Label `bpool_ip3tdb' - Last modification
time 2021-05-03 12:14:08 Monday, UUID f14d7bdf89fe21fb - Sector size 512B -
Total size 4192256KiB
#. List boot environments nested inside ``bpool/$INST_ID/BOOT``::
grub> ls (crypto0)/sys/BOOT
@/ default/ be0/
#. Instruct GRUB to load configuration from ``be0`` boot environment::
grub> prefix=(crypto0)/sys/BOOT/be0/@/grub
grub> configfile $prefix/grub.cfg
#. GRUB menu should now appear.
#. After entering system, reinstall GRUB.
Switch GRUB prefix when disk fails
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you are using LUKS encrypted boot pool with multiple disks,
the primary disk failed, GRUB will fail to load configuration.
If there's still enough redundancy for the boot pool, try fix
GRUB with the following method:
#. Ensure ``Slot 1 opened`` message
is shown
.. code-block:: text
Welcome to GRUB!
error: no such cryptodisk found.
Attempting to decrypt master key...
Enter passphrase for hd0,gpt2 (c0987ea1a51049e9b3056622804de62a):
Slot 1 opened
error: disk `cryptouuid/47ed1b7eb0014bc9a70aede3d8714faf' not found.
Entering rescue mode...
grub rescue>
If ``error: access denied.`` is shown,
try re-enter password with::
grub rescue> cryptomount hd0,gpt2
#. Check prefix::
grub rescue > set
# prefix=(cryptouuid/47ed1b7eb0014bc9a70aede3d8714faf)/sys/BOOT/be0@/grub
# root=cryptouuid/47ed1b7eb0014bc9a70aede3d8714faf
#. Set correct ``prefix`` and ``root`` by replacing
``cryptouuid/UUID`` with ``crypto0``::
grub rescue> prefix=(crypto0)/sys/BOOT/default@/grub
grub rescue> root=crypto0
#. Boot GRUB::
grub rescue> insmod normal
grub rescue> normal
GRUB should then boot normally.
#. After entering system, edit ``/etc/fstab`` to promote
one backup to ``/boot/efi``.
#. Make the change to ``prefix`` and ``root``
permanent by `reinstalling GRUB <#grub-installation>`__.

2
docs/Getting Started/NixOS/index.rst
View File

@@ -55,7 +55,7 @@ Root on ZFS
ZFS can be used as root file system for NixOS.
An installation guide is available.
`Start here <Root%20on%20ZFS/0-overview.html>`__.
`Start here <Root%20on%20ZFS/1-preparation.html>`__.
.. toctree::
:maxdepth: 1

11
docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS.rst
View File

@@ -1,11 +0,0 @@
Rocky Linux 8 Root on ZFS
=======================================
`Start here <RHEL%208-based%20distro%20Root%20on%20ZFS/0-overview.html>`__.
Contents
--------
.. toctree::
:maxdepth: 2
:glob:
RHEL 8-based distro Root on ZFS/*

141
docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS/0-overview.rst
View File

@@ -1,141 +0,0 @@
.. highlight:: sh
Overview
======================
This document describes how to install RHEL 8-based distro with ZFS as root
file system.
Caution
~~~~~~~
- With less than 4GB RAM, DKMS might fail to build
in live environment.
- This guide wipes entire physical disks. Back up existing data.
- `GRUB does not and
will not work on 4Kn drive with legacy (BIOS) booting.
<http://savannah.gnu.org/bugs/?46700>`__
Partition layout
~~~~~~~~~~~~~~~~
GUID partition table (GPT) is used.
EFI system partition will be referred to as **ESP** in this document.
+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+
| Name | legacy boot | ESP | Boot pool | swap | root pool | remaining space |
+======================+======================+=======================+======================+=====================+=======================+=================+
| File system | | vfat | ZFS | swap | ZFS | |
+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+
| Size | 1M | 2G | 4G | depends on RAM size | | |
+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+
| Optional encryption | | *Secure Boot* | | plain dm-crypt | ZFS native encryption | |
| | | | | | | |
+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+
| Partition no. | 5 | 1 | 2 | 4 | 3 | |
+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+
| Mount point | | /boot/efi | /boot | | / | |
| | | /boot/efis/disk-part1 | | | | |
+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+
Dataset layout
~~~~~~~~~~~~~~
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| Dataset | canmount | mountpoint | container | notes |
+===========================+======================+======================+=====================================+===========================================+
| bpool | off | /boot | contains sys | |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool | off | / | contains sys | |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| bpool/sys | off | none | contains BOOT | |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool/sys | off | none | contains ROOT | sys is encryptionroot |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| bpool/sys/BOOT | off | none | contains boot environments | |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool/sys/ROOT | off | none | contains boot environments | |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool/sys/DATA | off | none | contains placeholder "default" | |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool/sys/DATA/default | off | / | contains user datasets | child datsets inherits mountpoint |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool/sys/DATA/default/ | on | /home (inherited) | no | |
| home | | | | user datasets, also called "shared |
| | | | | datasets", "persistent datasets"; also |
| | | | | include /var/lib, /srv, ... |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| bpool/sys/BOOT/default | noauto | /boot | no | noauto is used to switch BE. because of |
| | | | | noauto, must use fstab to mount |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool/sys/ROOT/default | noauto | / | no | mounted by initrd zfs hook |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| bpool/sys/BOOT/be1 | noauto | /boot | no | see bpool/sys/BOOT/default |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool/sys/ROOT/be1 | noauto | / | no | see rpool/sys/ROOT/default |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
Encryption
~~~~~~~~~~
- Swap
Swap is always encrypted. By default, swap is encrypted
with plain dm-crypt with key generated from ``/dev/urandom``
at every boot. Swap content does not persist between reboots.
- Root pool
ZFS native encryption can be optionally enabled for ``rpool/sys``
and child datasets.
User should be aware that, ZFS native encryption does not
encrypt some metadata of the datasets.
ZFS native encryption also does not change master key when ``zfs change-key`` is invoked.
Therefore, you should wipe the disk when password is compromised to protect confidentiality.
See `zfs-load-key.8 <https://openzfs.github.io/openzfs-docs/man/8/zfs-load-key.8.html>`__
and `zfs-change-key.8 <https://openzfs.github.io/openzfs-docs/man/8/zfs-change-key.8.html>`__
for more information regarding ZFS native encryption.
Encryption is enabled at dataset creation and can not be disabled later.
- Boot pool
Boot pool can not be encrypted.
- Bootloader
Bootloader can not be encrypted.
However, with Secure Boot, bootloader
can be verified by motherboard firmware to be untempered,
which should be sufficient for most purposes.
Secure Boot is not supported out-of-the-box due to ZFS module.
Booting with disk failure
~~~~~~~~~~~~~~~~~~~~~~~~~
This guide is written with disk failure in mind.
If disks used in Root on ZFS pool failed, but
sufficient redundancy for both root pool and boot pool
still exists, the system will still boot normally.
Swap partition on the failed disk will fail to mount,
after an 1m30s timeout.
This feature is useful for use cases such
as an unattended remote server.
Example:
- System has disks ``n>1``
- Installed with mirrored setup
- Mirrored setup can tolerate up to ``n-1`` disk failures
- Disconnect one or more disks, keep at least
one disk connected
- System still boots, but fails to mount swap and
EFI partition

149
docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS/1-preparation.rst
View File

@@ -1,149 +0,0 @@
.. highlight:: sh
Preparation
======================
.. contents:: Table of Contents
:local:
#. Disable Secure Boot. ZFS modules can not be loaded if Secure Boot is enabled.
#. Download a variant of `Rocky Linux 8.5 Live
ISO <https://dl.rockylinux.org/pub/rocky/8.5/Live/x86_64/>`__ and boot from it.
#. Set root password or ``/root/.ssh/authorized_keys``.
#. Start SSH server::
echo PermitRootLogin yes >> /etc/ssh/sshd_config
systemctl restart sshd
#. Connect from another computer::
ssh root@192.168.1.19
#. Temporarily set SELinux to permissive in live environment::
setenforce 0
SELinux will be enabled on the installed system.
#. Optional: If mirror speed is slow, you can manually pick a fixed mirror
from `mirrorlist <https://mirrors.rockylinux.org/mirrormanager/mirrors>`__
and apply it::
sed -i 's|^mirrorlist=|#mirrorlist=|g' /etc/yum.repos.d/*
sed -i 's|^#baseurl=|baseurl=|g' /etc/yum.repos.d/*
sed -i 's|dl.rockylinux.org/$contentdir|mirrors.sjtug.sjtu.edu.cn/rocky|g' /etc/yum.repos.d/*
#. Add ZFS repo::
source /etc/os-release
RHEL_ZFS_REPO=https://zfsonlinux.org/epel/zfs-release.el${VERSION_ID/./_}.noarch.rpm
dnf install -y $RHEL_ZFS_REPO
#. Install ZFS packages::
dnf install -y epel-release
dnf config-manager --disable zfs
dnf config-manager --enable zfs-kmod
dnf install -y zfs
#. Load kernel modules::
modprobe zfs
#. Install helper script and partition tool::
rpm -ivh --nodeps https://dl.fedoraproject.org/pub/fedora/linux/releases/36/Everything/x86_64/os/Packages/a/arch-install-scripts-24-3.fc36.noarch.rpm
dnf install -y gdisk dosfstools
#. Set RHEL major version::
INST_RHEL_VER=${VERSION_ID%%.*}
#. Unique pool suffix. ZFS expects pool names to be
unique, therefore it's recommended to create
pools with a unique suffix::
INST_UUID=$(dd if=/dev/urandom bs=1 count=100 2>/dev/null | tr -dc 'a-z0-9' | cut -c-6)
#. Identify this installation in ZFS filesystem path::
INST_ID=rhel
#. Target disk
List available disks with::
ls /dev/disk/by-id/*
If using virtio as disk bus, use
``/dev/disk/by-path/*``.
Declare disk array::
DISK='/dev/disk/by-id/ata-FOO /dev/disk/by-id/nvme-BAR'
For single disk installation, use::
DISK='/dev/disk/by-id/disk1'
#. Choose a primary disk. This disk will be used
for primary EFI partition, default to
first disk in the array::
INST_PRIMARY_DISK=$(echo $DISK | cut -f1 -d\ )
#. Set vdev topology, possible values are:
- (not set, single disk or striped; no redundancy)
- mirror
- raidz1
- raidz2
- raidz3
::
INST_VDEV=
This will create a single vdev with the topology of your choice.
It is also possible to manually create a pool with multiple vdevs, such as::
zpool create --options \
poolName \
mirror sda sdb \
raidz2 sdc ... \
raidz3 sde ... \
spare sdf ...
Notice the cost of parity when using RAID-Z. See
`here <https://www.delphix.com/blog/delphix-engineering/zfs-raidz-stripe-width-or-how-i-learned-stop-worrying-and-love-raidz>`__
and `here <https://docs.google.com/spreadsheets/d/1tf4qx1aMJp8Lo_R6gpT689wTjHv6CGVElrPqTA0w_ZY/>`__.
For boot pool, which must be readable by GRUB, mirrored vdev should always be used for maximum redundancy.
This guide will use mirrored bpool for multi-disk setup.
Refer to `zpoolconcepts <https://openzfs.github.io/openzfs-docs/man/7/zpoolconcepts.7.html>`__
and `zpool-create <https://openzfs.github.io/openzfs-docs/man/8/zpool-create.8.html>`__
man pages for details.
#. Set partition size:
Set ESP size::
INST_PARTSIZE_ESP=2 # in GB
Set boot pool size. To avoid running out of space while using
boot environments, the minimum is 4GB. Adjust the size if you
intend to use multiple kernel/distros::
INST_PARTSIZE_BPOOL=4
Set swap size. It's `recommended <https://chrisdown.name/2018/01/02/in-defence-of-swap.html>`__
to setup a swap partition. If you intend to use hibernation,
the minimum should be no less than RAM size. Skip if swap is not needed::
INST_PARTSIZE_SWAP=8
Root pool size, use all remaining disk space if not set::
INST_PARTSIZE_RPOOL=

271
docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS/2-system-installation.rst
View File

@@ -1,271 +0,0 @@
.. highlight:: sh
System Installation
======================
.. contents:: Table of Contents
:local:
#. Optional: wipe solid-state drives with the generic tool
`blkdiscard <https://utcc.utoronto.ca/~cks/space/blog/linux/ErasingSSDsWithBlkdiscard>`__,
to clean previous partition tables and improve performance.
All content will be irrevocably destroyed::
for i in ${DISK}; do
blkdiscard $i &
done
wait
This is a quick operation and should be completed under one
minute.
For other device specific methods, see
`Memory cell clearing <https://wiki.archlinux.org/title/Solid_state_drive/Memory_cell_clearing>`__
#. Partition the disks.
See `Overview <0-overview.html>`__ for details::
for i in ${DISK}; do
sgdisk --zap-all $i
sgdisk -n1:1M:+${INST_PARTSIZE_ESP}G -t1:EF00 $i
sgdisk -n2:0:+${INST_PARTSIZE_BPOOL}G -t2:BE00 $i
if [ "${INST_PARTSIZE_SWAP}" != "" ]; then
sgdisk -n4:0:+${INST_PARTSIZE_SWAP}G -t4:8200 $i
fi
if [ "${INST_PARTSIZE_RPOOL}" = "" ]; then
sgdisk -n3:0:0 -t3:BF00 $i
else
sgdisk -n3:0:+${INST_PARTSIZE_RPOOL}G -t3:BF00 $i
fi
sgdisk -a1 -n5:24K:+1000K -t5:EF02 $i
done
#. Create boot pool::
disk_num=0; for i in $DISK; do disk_num=$(( $disk_num + 1 )); done
if [ $disk_num -gt 1 ]; then INST_VDEV_BPOOL=mirror; fi
zpool create \
-d -o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-o ashift=12 \
-o autotrim=on \
-O acltype=posixacl \
-O canmount=off \
-O compression=lz4 \
-O devices=off \
-O normalization=formD \
-O relatime=on \
-O xattr=sa \
-O mountpoint=/boot \
-R /mnt \
bpool_$INST_UUID \
$INST_VDEV_BPOOL \
$(for i in ${DISK}; do
printf "$i-part2 ";
done)
You should not need to customize any of the options for the boot pool.
GRUB does not support all of the zpool features. See ``spa_feature_names``
in `grub-core/fs/zfs/zfs.c
<http://git.savannah.gnu.org/cgit/grub.git/tree/grub-core/fs/zfs/zfs.c#n276>`__.
This step creates a separate boot pool for ``/boot`` with the features
limited to only those that GRUB supports, allowing the root pool to use
any/all features.
Features enabled with ``-o compatibility=grub2`` can be seen
`here <https://github.com/openzfs/zfs/blob/master/cmd/zpool/compatibility.d/grub2>`__.
#. Create root pool::
zpool create \
-o ashift=12 \
-o autotrim=on \
-R /mnt \
-O acltype=posixacl \
-O canmount=off \
-O compression=zstd \
-O dnodesize=auto \
-O normalization=formD \
-O relatime=on \
-O xattr=sa \
-O mountpoint=/ \
rpool_$INST_UUID \
$INST_VDEV \
$(for i in ${DISK}; do
printf "$i-part3 ";
done)
**Notes:**
- The use of ``ashift=12`` is recommended here because many drives
today have 4 KiB (or larger) physical sectors, even though they
present 512 B logical sectors. Also, a future replacement drive may
have 4 KiB physical sectors (in which case ``ashift=12`` is desirable)
or 4 KiB logical sectors (in which case ``ashift=12`` is required).
- Setting ``-O acltype=posixacl`` enables POSIX ACLs globally. If you
do not want this, remove that option, but later add
``-o acltype=posixacl`` (note: lowercase “o”) to the ``zfs create``
for ``/var/log``, as `journald requires ACLs
<https://askubuntu.com/questions/970886/journalctl-says-failed-to-search-journal-acl-operation-not-supported>`__
- Setting ``normalization=formD`` eliminates some corner cases relating
to UTF-8 filename normalization. It also implies ``utf8only=on``,
which means that only UTF-8 filenames are allowed. If you care to
support non-UTF-8 filenames, do not use this option. For a discussion
of why requiring UTF-8 filenames may be a bad idea, see `The problems
with enforced UTF-8 only filenames
<http://utcc.utoronto.ca/~cks/space/blog/linux/ForcedUTF8Filenames>`__.
- ``recordsize`` is unset (leaving it at the default of 128 KiB). If you
want to tune it (e.g. ``-o recordsize=1M``), see `these
<https://jrs-s.net/2019/04/03/on-zfs-recordsize/>`__ `various
<http://blog.programster.org/zfs-record-size>`__ `blog
<https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSFileRecordsizeGrowth>`__
`posts
<https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSRecordsizeAndCompression>`__.
- Setting ``relatime=on`` is a middle ground between classic POSIX
``atime`` behavior (with its significant performance impact) and
``atime=off`` (which provides the best performance by completely
disabling atime updates). Since Linux 2.6.30, ``relatime`` has been
the default for other filesystems. See `RedHat’s documentation
<https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/power_management_guide/relatime>`__
for further information.
- Setting ``xattr=sa`` `vastly improves the performance of extended
attributes
<https://github.com/zfsonlinux/zfs/commit/82a37189aac955c81a59a5ecc3400475adb56355>`__.
Inside ZFS, extended attributes are used to implement POSIX ACLs.
Extended attributes can also be used by user-space applications.
`They are used by some desktop GUI applications.
<https://en.wikipedia.org/wiki/Extended_file_attributes#Linux>`__
`They can be used by Samba to store Windows ACLs and DOS attributes;
they are required for a Samba Active Directory domain controller.
<https://wiki.samba.org/index.php/Setting_up_a_Share_Using_Windows_ACLs>`__
Note that ``xattr=sa`` is `Linux-specific
<https://openzfs.org/wiki/Platform_code_differences>`__. If you move your
``xattr=sa`` pool to another OpenZFS implementation besides ZFS-on-Linux,
extended attributes will not be readable (though your data will be). If
portability of extended attributes is important to you, omit the
``-O xattr=sa`` above. Even if you do not want ``xattr=sa`` for the whole
pool, it is probably fine to use it for ``/var/log``.
- Make sure to include the ``-part3`` portion of the drive path. If you
forget that, you are specifying the whole disk, which ZFS will then
re-partition, and you will lose the bootloader partition(s).
#. This section implements dataset layout as described in `overview <0-overview.html>`__.
Create root system container:
- Unencrypted::
zfs create \
-o canmount=off \
-o mountpoint=none \
rpool_$INST_UUID/$INST_ID
- Encrypted:
Pick a strong password. Once compromised, changing password will not keep your
data safe. See ``zfs-change-key(8)`` for more info::
zfs create \
-o canmount=off \
-o mountpoint=none \
-o encryption=on \
-o keylocation=prompt \
-o keyformat=passphrase \
rpool_$INST_UUID/$INST_ID
Create other system datasets::
zfs create -o canmount=off -o mountpoint=none bpool_$INST_UUID/$INST_ID
zfs create -o canmount=off -o mountpoint=none bpool_$INST_UUID/$INST_ID/BOOT
zfs create -o canmount=off -o mountpoint=none rpool_$INST_UUID/$INST_ID/ROOT
zfs create -o canmount=off -o mountpoint=none rpool_$INST_UUID/$INST_ID/DATA
zfs create -o mountpoint=/boot -o canmount=noauto bpool_$INST_UUID/$INST_ID/BOOT/default
zfs create -o mountpoint=/ -o canmount=off rpool_$INST_UUID/$INST_ID/DATA/default
zfs create -o mountpoint=/ -o canmount=noauto rpool_$INST_UUID/$INST_ID/ROOT/default
zfs mount rpool_$INST_UUID/$INST_ID/ROOT/default
zfs mount bpool_$INST_UUID/$INST_ID/BOOT/default
for i in {usr,var,var/lib};
do
zfs create -o canmount=off rpool_$INST_UUID/$INST_ID/DATA/default/$i
done
for i in {home,root,srv,usr/local,var/log,var/spool};
do
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/$i
done
chmod 750 /mnt/root
#. Format and mount ESP::
for i in ${DISK}; do
mkfs.vfat -n EFI ${i}-part1
mkdir -p /mnt/boot/efis/${i##*/}-part1
mount -t vfat ${i}-part1 /mnt/boot/efis/${i##*/}-part1
done
mkdir -p /mnt/boot/efi
mount -t vfat ${INST_PRIMARY_DISK}-part1 /mnt/boot/efi
#. Create separate user dataset at ``/home/User``, dateset name can be
changed later::
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/home/User
If needed, snapshot, rollback and other related permissions can be
delegated to the user later.
#. Create optional program data datasets to omit data from rollback::
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/games
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/www
# for GNOME
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/AccountsService
# for Docker
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/docker
# for NFS
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/nfs
# for LXC
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/lxc
# for LibVirt
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/libvirt
##other application
# zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/$name
Add other datasets when needed, such as PostgreSQL.
#. Install base packages::
dnf --installroot=/mnt --releasever=${INST_RHEL_VER} -y install \
${RHEL_ZFS_REPO} @core epel-release grub2-efi-x64 grub2-pc-modules \
grub2-efi-x64-modules shim-x64 efibootmgr \
kernel kernel-devel python3-dnf-plugin-post-transaction-actions
dnf config-manager --installroot=/mnt --disable zfs
dnf config-manager --installroot=/mnt --enable zfs-kmod
dnf install --installroot=/mnt -y zfs zfs-dracut
#. Update zfs repo if a newer release is available::
source /mnt/etc/os-release
RHEL_ZFS_REPO_NEW=https://zfsonlinux.org/epel/zfs-release.el${VERSION_ID/./_}.noarch.rpm
dnf install --installroot=/mnt -y $RHEL_ZFS_REPO_NEW || true
#. Optional: enable boot environment support and dnf integration::
dnf --installroot=/mnt copr enable -y m0p/bieaz
dnf --installroot=/mnt install -y bieaz python3-dnf-plugin-rozb3
If multi-disk setup is used, enable multi-disk
support inside ``/mnt/etc/bieaz.cfg``.

103
docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS/3-system-configuration.rst
View File

@@ -1,103 +0,0 @@
.. highlight:: sh
System Configuration
======================
.. contents:: Table of Contents
:local:
#. Generate fstab::
genfstab -U /mnt | sed 's;zfs[[:space:]]*;zfs zfsutil,;g' | grep "zfs zfsutil" >> /mnt/etc/fstab
for i in ${DISK}; do
echo UUID=$(blkid -s UUID -o value ${i}-part1) /boot/efis/${i##*/}-part1 vfat \
x-systemd.idle-timeout=1min,x-systemd.automount,noauto,umask=0022,fmask=0022,dmask=0022 0 1 >> /mnt/etc/fstab
done
echo UUID=$(blkid -s UUID -o value ${INST_PRIMARY_DISK}-part1) /boot/efi vfat \
x-systemd.idle-timeout=1min,x-systemd.automount,noauto,umask=0022,fmask=0022,dmask=0022 0 1 >> /mnt/etc/fstab
if [ "${INST_PARTSIZE_SWAP}" != "" ]; then
for i in ${DISK}; do
echo ${i##*/}-part4-swap ${i}-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256,discard >> /mnt/etc/crypttab
echo /dev/mapper/${i##*/}-part4-swap none swap x-systemd.requires=cryptsetup.target,defaults 0 0 >> /mnt/etc/fstab
done
fi
By default, systemd will halt boot process if any entry in ``/etc/fstab`` fails
to mount. This is unnecessary for mirrored EFI boot partitions.
With the above mount options, systemd will skip mounting them at boot,
only mount them on demand when accessed.
#. Configure dracut::
echo 'add_dracutmodules+=" zfs "' > /mnt/etc/dracut.conf.d/zfs.conf
#. Force load mpt3sas module if used::
if grep mpt3sas /proc/modules; then
echo 'forced_drivers+=" mpt3sas "' >> /mnt/etc/dracut.conf.d/zfs.conf
fi
#. Interactively set locale, keymap, timezone, hostname and root password::
rm -f /mnt/etc/localtime
systemd-firstboot --root=/mnt --prompt --root-password=PASSWORD
This can be non-interactive, see man page for details::
rm -f /mnt/etc/localtime
systemd-firstboot --root=/mnt \
--locale="en_US.UTF-8" --locale-messages="en_US.UTF-8" \
--keymap=us --timezone="Europe/Berlin" --hostname=myHost \
--root-password=PASSWORD
``systemd-firstboot`` have bugs, root password is set below.
#. Generate host id::
zgenhostid -f -o /mnt/etc/hostid
#. Install locale package, example for English locale::
dnf --installroot=/mnt install -y glibc-minimal-langpack glibc-langpack-en
Program will show errors if not installed.
#. Enable ZFS services::
systemctl enable zfs-import-scan.service zfs-import.target zfs-zed zfs.target --root=/mnt
systemctl disable zfs-mount --root=/mnt
At boot, datasets on rpool are mounted with ``/etc/fstab``,
which can control the mounting process more precisely than ``zfs-mount.service``.
#. By default SSH server is enabled, allowing root login by password,
disable SSH server::
systemctl disable sshd --root=/mnt
systemctl enable firewalld --root=/mnt
#. Chroot::
echo "INST_PRIMARY_DISK=$INST_PRIMARY_DISK
INST_LINVAR=$INST_LINVAR
INST_UUID=$INST_UUID
INST_ID=$INST_ID
unalias -a
TERM=xterm
INST_VDEV=$INST_VDEV
INST_VDEV=$INST_VDEV
DISK=\"$DISK\"" > /mnt/root/chroot
history -w /mnt/home/sys-install-pre-chroot.txt
arch-chroot /mnt bash --login
#. Source variables::
source /root/chroot
#. For SELinux, relabel filesystem on reboot::
fixfiles -F onboot
#. Set root password::
passwd

258
docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS/5-bootloader.rst
View File

@@ -1,258 +0,0 @@
.. highlight:: sh
Bootloader
======================
.. contents:: Table of Contents
:local:
Apply workarounds
~~~~~~~~~~~~~~~~~~~~
Currently GRUB has multiple compatibility problems with ZFS,
especially with regards to newer ZFS features.
Workarounds have to be applied.
#. grub2-probe fails to get canonical path
When persistent device names ``/dev/disk/by-id/*`` are used
with ZFS, GRUB will fail to resolve the path of the boot pool
device. Error::
# /usr/bin/grub2-probe: error: failed to get canonical path of `/dev/virtio-pci-0000:06:00.0-part3'.
Solution::
echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile.d/zpool_vdev_name_path.sh
source /etc/profile.d/zpool_vdev_name_path.sh
#. Pool name missing
See `this bug report <https://savannah.gnu.org/bugs/?59614>`__.
Root pool name is missing from ``root=ZFS=rpool_$INST_UUID/ROOT/default``
kernel cmdline in generated ``grub.cfg`` file.
A workaround is to replace the pool name detection with ``zdb``
command::
sed -i "s|rpool=.*|rpool=\`zdb -l \${GRUB_DEVICE} \| grep -E '[[:blank:]]name' \| cut -d\\\' -f 2\`|" /etc/grub.d/10_linux
Install GRUB
~~~~~~~~~~~~~~~~~~~~
#. If using virtio disk, add driver to initrd::
echo 'filesystems+=" virtio_blk "' >> /etc/dracut.conf.d/fs.conf
#. Generate initrd::
rm -f /etc/zfs/zpool.cache
touch /etc/zfs/zpool.cache
chmod a-w /etc/zfs/zpool.cache
chattr +i /etc/zfs/zpool.cache
for directory in /lib/modules/*; do
kernel_version=$(basename $directory)
dracut --force --kver $kernel_version
done
#. Load ZFS modules and disable BLS::
echo 'GRUB_ENABLE_BLSCFG=false' >> /etc/default/grub
#. Create GRUB boot directory, in ESP and boot pool::
mkdir -p /boot/efi/EFI/rocky # EFI GRUB dir
mkdir -p /boot/efi/EFI/rocky/grub2 # legacy GRUB dir
mkdir -p /boot/grub2
Boot environment-specific configuration (kernel, etc)
is stored in ``/boot/grub2/grub.cfg``, enabling rollback.
#. When in doubt, install both legacy boot
and EFI.
#. If using legacy booting, install GRUB to every disk::
for i in ${DISK}; do
grub2-install --boot-directory /boot/efi/EFI/rocky --target=i386-pc $i
done
#. If using EFI::
for i in ${DISK}; do
efibootmgr -cgp 1 -l "\EFI\rocky\shimx64.efi" \
-L "rocky-${i##*/}" -d ${i}
done
cp -r /usr/lib/grub/x86_64-efi/ /boot/efi/EFI/rocky
#. Generate GRUB Menu:
Apply workaround::
tee /etc/grub.d/09_fix_root_on_zfs <<EOF
#!/bin/sh
echo 'insmod zfs'
echo 'set root=(hd0,gpt2)'
EOF
chmod +x /etc/grub.d/09_fix_root_on_zfs
Generate menu::
grub2-mkconfig -o /boot/efi/EFI/rocky/grub.cfg
cp /boot/efi/EFI/rocky/grub.cfg /boot/efi/EFI/rocky/grub2/grub.cfg
cp /boot/efi/EFI/rocky/grub.cfg /boot/grub2/grub.cfg
The following errors may be safely ignored:
- ``device-mapper: reload ioctl on osprober-linux-sda2 (253:0) failed: Device or resource busy``
This is caused by os-prober probing OS on the partitions used by ZFS,
harmless but os-prober can be disabled by::
echo GRUB_DISABLE_OS_PROBER=true >> /etc/default/grub
- ``/usr/sbin/grub2-probe: error: ../grub-core/kern/fs.c:120:unknown filesystem.``
This is fixed by /etc/grub.d/09_fix_root_on_zfs
#. For both legacy and EFI booting: mirror ESP content::
ESP_MIRROR=$(mktemp -d)
unalias -a
cp -r /boot/efi/EFI $ESP_MIRROR
for i in /boot/efis/*; do
cp -r $ESP_MIRROR/EFI $i
done
#. Automatically regenerate GRUB menu on kernel update::
tee /etc/dnf/plugins/post-transaction-actions.d/00-update-grub-menu-for-kernel.action <<EOF >/dev/null
# kernel-core package contains vmlinuz and initramfs
# change package name if non-standard kernel is used
kernel-core:in:/usr/local/sbin/update-grub-menu.sh
kernel-core:out:/usr/local/sbin/update-grub-menu.sh
EOF
tee /usr/local/sbin/update-grub-menu.sh <<-'EOF' >/dev/null
#!/bin/sh
export PATH=$PATH:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
export ZPOOL_VDEV_NAME_PATH=YES
source /etc/os-release
grub2-mkconfig -o /boot/efi/EFI/${ID}/grub.cfg
cp /boot/efi/EFI/${ID}/grub.cfg /boot/efi/EFI/${ID}/grub2/grub.cfg
cp /boot/efi/EFI/${ID}/grub.cfg /boot/grub2/grub.cfg
ESP_MIRROR=$(mktemp -d)
cp -r /boot/efi/EFI $ESP_MIRROR
for i in /boot/efis/*; do
cp -r $ESP_MIRROR/EFI $i
done
rm -rf $ESP_MIRROR
EOF
chmod +x /usr/local/sbin/update-grub-menu.sh
#. Notes for GRUB on RHEL
To support Secure Boot, GRUB has been heavily modified by Fedora,
namely:
- ``grub2-install`` is `disabled for UEFI <https://bugzilla.redhat.com/show_bug.cgi?id=1917213>`__
- Only a static, signed version of bootloader is copied to EFI system partition
- This signed bootloader does not have built-in support for either ZFS or LUKS containers
- This signed bootloader only loads configuration from ``/boot/efi/EFI/fedora/grub.cfg``
Unrelated to Secure Boot, GRUB has also been modified to provide optional
support for `systemd bootloader specification (bls) <https://systemd.io/BOOT_LOADER_SPECIFICATION/>`__.
Currently ``blscfg.mod`` is incompatible with root on ZFS.
As bls is disabled, you will need to regenerate GRUB menu after each kernel upgrade.
Or else the new kernel will not be recognized and system will boot the old kernel
on reboot.
Also see `Fedora docs for GRUB
<https://docs.fedoraproject.org/en-US/fedora/rawhide/system-administrators-guide/kernel-module-driver-configuration/Working_with_the_GRUB_2_Boot_Loader/>`__.
Finish Installation
~~~~~~~~~~~~~~~~~~~~
#. Exit chroot::
exit
#. Take a snapshot of the clean installation for future use::
zfs snapshot -r rpool_$INST_UUID/$INST_ID@install
zfs snapshot -r bpool_$INST_UUID/$INST_ID@install
#. Unmount EFI system partition::
umount /mnt/boot/efi
umount /mnt/boot/efis/*
#. Export pools::
zpool export bpool_$INST_UUID
zpool export rpool_$INST_UUID
#. Reboot::
reboot
Post installaion
~~~~~~~~~~~~~~~~
#. If you have other data pools, generate list of datasets for `zfs-mount-generator
<https://manpages.ubuntu.com/manpages/focal/man8/zfs-mount-generator.8.html>`__ to mount them at boot::
DATA_POOL='tank0 tank1'
# tab-separated zfs properties
# see /etc/zfs/zed.d/history_event-zfs-list-cacher.sh
export \
PROPS="name,mountpoint,canmount,atime,relatime,devices,exec\
,readonly,setuid,nbmand,encroot,keylocation"
for i in $DATA_POOL; do
zfs list -H -t filesystem -o $PROPS -r $i > /etc/zfs/zfs-list.cache/$i
done
#. After reboot, consider adding a normal user::
# with root permissions
sudo -i
# store user name in a variable
myUser=UserName
# rename default `User` to new user name
zfs rename $(df --output=source /home | tail -n +2)/User $(df --output=source /home | tail -n +2)/${myUser}
# update entry in fstab
sed -i "s|/home/User|/home/${myUser}|g" /etc/fstab
# add user
useradd --no-create-home --user-group --home-dir /home/${myUser} --comment 'My Name' ${myUser}
# delegate snapshot and destroy permissions of the home dataset to
# new user
zfs allow -u ${myUser} mount,snapshot,destroy $(df --output=source /home | tail -n +2)/${myUser}
# fix permissions
chown --recursive ${myUser}:${myUser} /home/${myUser}
chmod 700 /home/${myUser}
# fix selinux context
restorecon /home/${myUser}
# set new password for user
passwd ${myUser}
Set up cron job to snapshot user home everyday::
dnf install cronie
systemctl enable --now crond
crontab -eu ${myUser}
#@daily zfs snap $(df --output=source /home/${myUser} | tail -n +2)@$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)
zfs list -t snapshot -S creation $(df --output=source /home/${myUser} | tail -n +2)
Install package groups::
dnf group list --hidden -v # query package groups
dnf group install 'Virtualization Host'

211
docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS/6-recovery.rst
View File

@@ -1,211 +0,0 @@
.. highlight:: sh
Recovery
======================
.. contents:: Table of Contents
:local:
GRUB Tips
-------------
Boot from GRUB rescue
~~~~~~~~~~~~~~~~~~~~~~~
If bootloader file is damaged, it's still possible
to boot computer with GRUB rescue image.
This section is also applicable if you are in
``grub rescue>``.
#. On another computer, generate rescue image with::
pacman -S --needed mtools libisoburn grub
grub-install
grub-mkrescue -o grub-rescue.img
dd if=grub-rescue.img of=/dev/your-usb-stick
Boot computer from the rescue media.
Both legacy and EFI mode are supported.
Or `download generated GRUB rescue image <https://gitlab.com/m_zhou/bieaz/uploads/e0847a8675cda4317ea7f48abb1d9f10/grub-rescue-2.06.img.7z>`__.
#. List available disks with ``ls`` command::
grub> ls (hd # press tab
Possible devices are:
hd0 hd1 hd2 hd3
#. List partitions by pressing tab key:
.. code-block:: text
grub> ls (hd0 # press tab
Possible partitions are:
Device hd0: No known filesystem detected - Sector size 512B - Total size 20971520KiB
Partition hd0,gpt1: Filesystem type fat - Label `EFI', UUID 0DF5-3A76 - Partition start at 1024KiB - Total size 1048576KiB
Partition hd0,gpt2: No known filesystem detected - Partition start at 1049600KiB - Total size 4194304KiB
- If boot pool is encrypted:
Unlock it with ``cryptomount``::
grub> insmod luks
grub> cryptomount hd0,gpt2
Attempting to decrypt master key...
Enter passphrase for hd0,gpt2 (af5a240e13e24483acf02600d61e0f36):
Slot 1 opened
Unlocked LUKS container is ``(crypto0)``:
.. code-block:: text
grub> ls (crypto0)
Device crypto0: Filesystem type zfs - Label `bpool_ip3tdb' - Last modification
time 2021-05-03 12:14:08 Monday, UUID f14d7bdf89fe21fb - Sector size 512B -
Total size 4192256KiB
- If boot pool is not encrypted:
.. code-block:: text
grub> ls (hd0,gpt2)
Device hd0,gpt2: Filesystem type zfs - Label `bpool_ip3tdb' - Last modification
time 2021-05-03 12:14:08 Monday, UUID f14d7bdf89fe21fb - Sector size 512B -
Total size 4192256KiB
#. List boot environments nested inside ``bpool/$INST_ID/BOOT``::
grub> ls (crypto0)/sys/BOOT
@/ default/ be0/
#. Instruct GRUB to load configuration from ``be0`` boot environment::
grub> prefix=(crypto0)/sys/BOOT/be0/@/grub
grub> configfile $prefix/grub.cfg
#. GRUB menu should now appear.
#. After entering system, `reinstall GRUB <#grub-installation>`__.
Switch GRUB prefix when disk fails
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you are using LUKS encrypted boot pool with multiple disks,
the primary disk failed, GRUB will fail to load configuration.
If there's still enough redundancy for the boot pool, try fix
GRUB with the following method:
#. Ensure ``Slot 1 opened`` message
is shown
.. code-block:: text
Welcome to GRUB!
error: no such cryptodisk found.
Attempting to decrypt master key...
Enter passphrase for hd0,gpt2 (c0987ea1a51049e9b3056622804de62a):
Slot 1 opened
error: disk `cryptouuid/47ed1b7eb0014bc9a70aede3d8714faf' not found.
Entering rescue mode...
grub rescue>
If ``error: access denied.`` is shown,
try re-enter password with::
grub rescue> cryptomount hd0,gpt2
#. Check prefix::
grub rescue > set
# prefix=(cryptouuid/47ed1b7eb0014bc9a70aede3d8714faf)/sys/BOOT/be0@/grub
# root=cryptouuid/47ed1b7eb0014bc9a70aede3d8714faf
#. Set correct ``prefix`` and ``root`` by replacing
``cryptouuid/UUID`` with ``crypto0``::
grub rescue> prefix=(crypto0)/sys/BOOT/default@/grub
grub rescue> root=crypto0
#. Boot GRUB::
grub rescue> insmod normal
grub rescue> normal
GRUB should then boot normally.
#. After entering system, edit ``/etc/fstab`` to promote
one backup to ``/boot/efi``.
#. Make the change to ``prefix`` and ``root``
permanent by `reinstalling GRUB <#grub-installation>`__.
Access system in chroot
-----------------------
#. Go through `preparation <1-preparation.html>`__.
#. Import and unlock root and boot pool::
zpool import -NR /mnt rpool_$INST_UUID
zpool import -NR /mnt bpool_$INST_UUID
If using password::
zfs load-key rpool_$INST_UUID/$INST_ID
If using keyfile::
zfs load-key -L file:///path/to/keyfile rpool_$INST_UUID/$INST_ID
#. Find the current boot environment::
zfs list
BE=default
#. Mount root filesystem::
zfs mount rpool_$INST_UUID/$INST_ID/ROOT/$BE
#. chroot into the system::
arch-chroot /mnt /bin/bash --login
zfs mount -a
mount -a
#. Finish rescue. See `finish installation <#finish-installation>`__.
Backup and migrate existing installation
----------------------------------------
With the help of `zfs send
<https://openzfs.github.io/openzfs-docs/man/8/zfs-send.8.html>`__
it is relatively easy to perform a system backup and migration.
#. Create a snapshot of root file system::
zfs snapshot -r rpool/$INST_ID@backup
zfs snapshot -r bpool/$INST_ID@backup
#. Save snapshot to a file or pipe to SSH::
zfs send --options rpool/$INST_ID@backup > /backup/$INST_ID-rpool
zfs send --options bpool/$INST_ID@backup > /backup/$INST_ID-bpool
#. Re-create partitions and root/boot
pool on target system.
#. Restore backup::
zfs recv rpool_new/$INST_ID < /backup/$INST_ID-rpool
zfs recv bpool_new/$INST_ID < /backup/$INST_ID-bpool
#. Chroot and reinstall bootloader.
#. Update pool name in ``/etc/fstab``, ``/boot/grub/grub.cfg``
and ``/etc/zfs/zfs-list.cache/*``.
#. Update device name, etc, in ``/etc/fstab`` and ``/etc/crypttab``.

11
docs/Getting Started/RHEL-based distro/RHEL-based distro Root on ZFS.rst Normal file
View File

@@ -0,0 +1,11 @@
RHEL Root on ZFS
=======================================
`Start here <RHEL%20-based%20distro%20Root%20on%20ZFS/1-preparation.html>`__.
Contents
--------
.. toctree::
:maxdepth: 2
:glob:
RHEL-based distro Root on ZFS/*

78
docs/Getting Started/RHEL-based distro/RHEL-based distro Root on ZFS/1-preparation.rst Normal file
View File

@@ -0,0 +1,78 @@
.. highlight:: sh
Preparation
======================
.. contents:: Table of Contents
:local:
#. Disable Secure Boot. ZFS modules can not be loaded if Secure Boot is enabled.
#. Download a variant of `AlmaLinux Minimal Live ISO
<https://repo.almalinux.org/almalinux/9/live/x86_64/>`__ and boot from it.
#. Connect to the Internet.
#. Set root password or ``/root/.ssh/authorized_keys``.
#. Start SSH server::
echo PermitRootLogin yes >> /etc/ssh/sshd_config
systemctl restart sshd
#. Connect from another computer::
ssh root@192.168.1.19
#. Target disk
List available disks with::
ls /dev/disk/by-id/*
If using virtio as disk bus, use ``/dev/disk/by-path/*``.
Declare disk array::
DISK='/dev/disk/by-id/ata-FOO /dev/disk/by-id/nvme-BAR'
For single disk installation, use::
DISK='/dev/disk/by-id/disk1'
#. Set partition size:
Set swap size. It's `recommended <https://chrisdown.name/2018/01/02/in-defence-of-swap.html>`__
to setup a swap partition. If you intend to use hibernation,
the minimum should be no less than RAM size. Skip if swap is not needed::
INST_PARTSIZE_SWAP=8
Root pool size, use all remaining disk space if not set::
INST_PARTSIZE_RPOOL=
#. Temporarily set SELinux to permissive in live environment::
setenforce 0
SELinux will be enabled on the installed system.
#. Add ZFS repo::
dnf install -y https://zfsonlinux.org/epel/zfs-release-el-2-1.noarch.rpm
#. Check available repos::
dnf repolist --all
#. Install ZFS packages::
dnf config-manager --disable zfs
dnf config-manager --enable zfs-kmod
dnf install -y zfs
# if gpg import fails, add --nogpgcheck
#. Load kernel modules::
modprobe zfs
#. Install partition tool::
dnf install -y gdisk dosfstools

146
docs/Getting Started/RHEL-based distro/RHEL-based distro Root on ZFS/2-system-installation.rst Normal file
View File

@@ -0,0 +1,146 @@
.. highlight:: sh
System Installation
======================
.. contents:: Table of Contents
:local:
#. Partition the disks::
for i in ${DISK}; do
sgdisk --zap-all $i
sgdisk -n1:1M:+1G -t1:EF00 $i
sgdisk -n2:0:+4G -t2:BE00 $i
test -z $INST_PARTSIZE_SWAP || sgdisk -n4:0:+${INST_PARTSIZE_SWAP}G -t4:8200 $i
if test -z $INST_PARTSIZE_RPOOL; then
sgdisk -n3:0:0 -t3:BF00 $i
else
sgdisk -n3:0:+${INST_PARTSIZE_RPOOL}G -t3:BF00 $i
fi
sgdisk -a1 -n5:24K:+1000K -t5:EF02 $i
done
#. Create boot pool::
zpool create \
-o compatibility=grub2 \
-o ashift=12 \
-o autotrim=on \
-O acltype=posixacl \
-O canmount=off \
-O compression=lz4 \
-O devices=off \
-O normalization=formD \
-O relatime=on \
-O xattr=sa \
-O mountpoint=/boot \
-R /mnt \
bpool \
mirror \
$(for i in ${DISK}; do
printf "$i-part2 ";
done)
If not using a multi-disk setup, remove ``mirror``.
You should not need to customize any of the options for the boot pool.
GRUB does not support all of the zpool features. See ``spa_feature_names``
in `grub-core/fs/zfs/zfs.c
<http://git.savannah.gnu.org/cgit/grub.git/tree/grub-core/fs/zfs/zfs.c#n276>`__.
This step creates a separate boot pool for ``/boot`` with the features
limited to only those that GRUB supports, allowing the root pool to use
any/all features.
Features enabled with ``-o compatibility=grub2`` can be seen
`here <https://github.com/openzfs/zfs/blob/master/cmd/zpool/compatibility.d/grub2>`__.
#. Create root pool::
zpool create \
-o ashift=12 \
-o autotrim=on \
-R /mnt \
-O acltype=posixacl \
-O canmount=off \
-O compression=zstd \
-O dnodesize=auto \
-O normalization=formD \
-O relatime=on \
-O xattr=sa \
-O mountpoint=/ \
rpool \
mirror \
$(for i in ${DISK}; do
printf "$i-part3 ";
done)
If not using a multi-disk setup, remove ``mirror``.
#. This section implements dataset layout as described in `overview <1-preparation.html>`__.
Create root system container:
- Unencrypted::
zfs create \
-o canmount=off \
-o mountpoint=none \
rpool/redhat
- Encrypted:
Pick a strong password. Once compromised, changing password will not keep your
data safe. See ``zfs-change-key(8)`` for more info::
zfs create \
-o canmount=off \
-o mountpoint=none \
-o encryption=on \
-o keylocation=prompt \
-o keyformat=passphrase \
rpool/redhat
Create system datasets::
zfs create -o canmount=on -o mountpoint=/ rpool/redhat/root
zfs create -o canmount=on -o mountpoint=/home rpool/redhat/home
zfs create -o canmount=off -o mountpoint=/var rpool/redhat/var
zfs create -o canmount=on rpool/redhat/var/lib
zfs create -o canmount=on rpool/redhat/var/log
Create boot dataset::
zfs create -o canmount=on -o mountpoint=/boot bpool/redhat
#. Format and mount ESP::
for i in ${DISK}; do
mkfs.vfat -n EFI ${i}-part1
mkdir -p /mnt/boot/efis/${i##*/}-part1
mount -t vfat ${i}-part1 /mnt/boot/efis/${i##*/}-part1
done
mkdir -p /mnt/boot/efi
mount -t vfat $(echo $DISK | cut -f1 -d\ )-part1 /mnt/boot/efi
#. Install packages::
dnf --installroot=/mnt --releasever=$(source /etc/os-release ; echo $VERSION_ID) -y install \
@core grub2-efi-x64 grub2-pc-modules grub2-efi-x64-modules shim-x64 efibootmgr kernel
dnf --installroot=/mnt --releasever=$(source /etc/os-release ; echo $VERSION_ID) -y install \
https://zfsonlinux.org/epel/zfs-release-el-2-1.noarch.rpm
dnf config-manager --installroot=/mnt --disable zfs
dnf config-manager --installroot=/mnt --enable zfs-kmod
dnf --installroot=/mnt --releasever=$(source /etc/os-release ; echo $VERSION_ID) \
--nogpgcheck -y install zfs zfs-dracut

66
docs/Getting Started/RHEL-based distro/RHEL-based distro Root on ZFS/3-system-configuration.rst Normal file
View File

@@ -0,0 +1,66 @@
.. highlight:: sh
System Configuration
======================
.. contents:: Table of Contents
:local:
#. Generate fstab::
mkdir -p /mnt/etc/
for i in ${DISK}; do
echo UUID=$(blkid -s UUID -o value ${i}-part1) /boot/efis/${i##*/}-part1 vfat \
umask=0022,fmask=0022,dmask=0022 0 1 >> /mnt/etc/fstab
done
echo $(echo $DISK | cut -f1 -d\ )-part1 /boot/efi vfat \
noauto,umask=0022,fmask=0022,dmask=0022 0 1 >> /mnt/etc/fstab
#. Configure dracut::
echo 'add_dracutmodules+=" zfs "' > /mnt/etc/dracut.conf.d/zfs.conf
#. Force load mpt3sas module if used::
if grep mpt3sas /proc/modules; then
echo 'forced_drivers+=" mpt3sas "' >> /mnt/etc/dracut.conf.d/zfs.conf
fi
#. Set locale, keymap, timezone, hostname and root password::
rm -f /mnt/etc/localtime
systemd-firstboot --root=/mnt --prompt --root-password=PASSWORD --force
#. Generate host id::
zgenhostid -f -o /mnt/etc/hostid
#. Install locale package, example for English locale::
dnf --installroot=/mnt install -y glibc-minimal-langpack glibc-langpack-en
#. Enable ZFS services::
systemctl enable zfs-import-scan.service zfs-mount zfs-import.target zfs-zed zfs.target --root=/mnt
#. By default SSH server is enabled, allowing root login by password,
disable SSH server::
systemctl disable sshd --root=/mnt
systemctl enable firewalld --root=/mnt
#. Chroot::
m='/dev /proc /sys'
for i in $m; do mount --rbind $i /mnt/$i; done
history -w /mnt/home/sys-install-pre-chroot.txt
chroot /mnt /usr/bin/env DISK=$DISK bash --login
#. For SELinux, relabel filesystem on reboot::
fixfiles -F onboot
#. Set root password::
passwd

129
docs/Getting Started/RHEL-based distro/RHEL-based distro Root on ZFS/5-bootloader.rst Normal file
View File

@@ -0,0 +1,129 @@
.. highlight:: sh
Bootloader
======================
.. contents:: Table of Contents
:local:
Apply workarounds
~~~~~~~~~~~~~~~~~~~~
Currently GRUB has multiple compatibility problems with ZFS,
especially with regards to newer ZFS features.
Workarounds have to be applied.
#. grub2-probe fails to get canonical path
When persistent device names ``/dev/disk/by-id/*`` are used
with ZFS, GRUB will fail to resolve the path of the boot pool
device. Error::
# /usr/bin/grub2-probe: error: failed to get canonical path of `/dev/virtio-pci-0000:06:00.0-part3'.
Solution::
echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile.d/zpool_vdev_name_path.sh
source /etc/profile.d/zpool_vdev_name_path.sh
#. Pool name missing
See `this bug report <https://savannah.gnu.org/bugs/?59614>`__.
Root pool name is missing from ``root=ZFS=rpool_$INST_UUID/ROOT/default``
kernel cmdline in generated ``grub.cfg`` file.
A workaround is to replace the pool name detection with ``zdb``
command::
sed -i "s|rpool=.*|rpool=\`zdb -l \${GRUB_DEVICE} \| grep -E '[[:blank:]]name' \| cut -d\\\' -f 2\`|" /etc/grub.d/10_linux
Caution: this fix must be applied after every GRUB update and before generating the menu.
Install GRUB
~~~~~~~~~~~~~~~~~~~~
#. If using virtio disk, add driver to initrd::
echo 'filesystems+=" virtio_blk "' >> /etc/dracut.conf.d/fs.conf
#. Create empty cache file and generate initrd::
rm -f /etc/zfs/zpool.cache
touch /etc/zfs/zpool.cache
chmod a-w /etc/zfs/zpool.cache
chattr +i /etc/zfs/zpool.cache
for directory in /lib/modules/*; do
kernel_version=$(basename $directory)
dracut --force --kver $kernel_version
done
#. Load ZFS modules and disable BLS::
echo 'GRUB_ENABLE_BLSCFG=false' >> /etc/default/grub
#. If using legacy booting, install GRUB to every disk::
for i in ${DISK}; do
grub2-install --target=i386-pc $i
done
#. If using EFI::
for i in ${DISK}; do
efibootmgr -cgp 1 -l "\EFI\almalinux\shimx64.efi" \
-L "almalinux-${i##*/}" -d ${i}
done
cp -r /usr/lib/grub/x86_64-efi/ /boot/efi/EFI/almalinux/
#. Generate GRUB Menu:
Generate menu::
grub2-mkconfig -o /boot/grub2/grub.cfg
cp /boot/grub2/grub.cfg /boot/efi/EFI/almalinux/
#. For both legacy and EFI booting: mirror ESP content::
ESP_MIRROR=$(mktemp -d)
unalias -a
cp -r /boot/efi/EFI $ESP_MIRROR
for i in /boot/efis/*; do
cp -r $ESP_MIRROR/EFI $i
done
rm -rf $ESP_MIRROR
#. Notes for GRUB on RHEL
As bls is disabled, you will need to regenerate GRUB menu after each kernel upgrade.
Or else the new kernel will not be recognized and system will boot the old kernel
on reboot.
Finish Installation
~~~~~~~~~~~~~~~~~~~~
#. Exit chroot::
exit
#. Export pools::
umount -Rl /mnt
zpool export -a
#. Reboot::
reboot
#. On first reboot, the boot process will fail, with failure messages such
as "You are in Emergency Mode...Press Ctrl-D to continue".
Wait for the computer to automatically reboot and the problem will be resolved.
Post installaion
~~~~~~~~~~~~~~~~
#. Install package groups::
dnf group list --hidden -v # query package groups
dnf group install @gnome-desktop
#. Add new user, configure swap.

6
docs/Getting Started/RHEL-based distro/index.rst
View File

@@ -146,15 +146,15 @@ And for RHEL/CentOS 8 and newer::
Use *zfs-testing* for DKMS packages and *zfs-testing-kmod*
kABI-tracking kmod packages.
RHEL 8-based distro Root on ZFS
RHEL-based distro Root on ZFS
-------------------------------
`Start here <RHEL%208-based%20distro%20Root%20on%20ZFS/0-overview.html>`__.
`Start here <RHEL%20-based%20distro%20Root%20on%20ZFS/1-preparation.html>`__.
.. toctree::
:maxdepth: 1
:glob:
RHEL 8-based distro Root on ZFS/*
RHEL-based distro Root on ZFS/*
.. _kABI-tracking kmod: https://elrepoproject.blogspot.com/2016/02/kabi-tracking-kmod-packages.html
.. _DKMS: https://en.wikipedia.org/wiki/Dynamic_Kernel_Module_Support