jacobpascual/openzfs-docs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add support for RHEL Root on ZFS (#180)

Browse Source 
* Add support for RHEL Root on ZFS

Signed-off-by: Maurice Zhou <jasper@apvc.uk>

* require cryptsetup.target before mounting swap

Signed-off-by: Maurice Zhou <jasper@apvc.uk>

* disable sshd

Signed-off-by: Maurice Zhou <jasper@apvc.uk>

* fedora 34; grub workaround

Signed-off-by: Maurice Zhou <jasper@apvc.uk>

* add manual redirect to new page

Signed-off-by: Maurice Zhou <jasper@apvc.uk>

* orphan page

Signed-off-by: Maurice Zhou <jasper@apvc.uk>

* Remove `RHEL and CentOS` from index

Co-authored-by: George Melikov <mail@gmelikov.ru>
This commit is contained in:
ne9z
2021-08-03 22:55:33 +08:00
committed by GitHub
parent 6a60717816
commit bdb85fc137
11 changed files with 1191 additions and 168 deletions
Download Patch File Download Diff File Expand all files Collapse all files

11
docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS.rst Normal file
View File

@@ -0,0 +1,11 @@
RHEL 8-based distro Root on ZFS
=======================================
`Start here <Root%20on%20ZFS/0-overview.html>`__.
Contents
--------
.. toctree::
:maxdepth: 2
:glob:
RHEL 8-based distro Root on ZFS/*

112
docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS/0-overview.rst Normal file
View File

@@ -0,0 +1,112 @@
.. highlight:: sh
Overview
======================
This document describes how to install RHEL 8-based distro with ZFS as root
file system.
Caution
~~~~~~~
- With less than 4GB RAM, DKMS might fail to build
in live environment.
- This guide wipes entire physical disks. Back up existing data.
- `GRUB does not and
will not work on 4Kn drive with legacy (BIOS) booting.
<http://savannah.gnu.org/bugs/?46700>`__
Partition layout
~~~~~~~~~~~~~~~~
GUID partition table (GPT) is used.
EFI system partition will be referred to as **ESP** in this document.
+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+
| Name | legacy boot | ESP | Boot pool | swap | root pool | remaining space |
+======================+======================+=======================+======================+=====================+=======================+=================+
| File system | | vfat | ZFS | swap | ZFS | |
+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+
| Size | 1M | 2G | 4G | depends on RAM size | | |
+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+
| Optional encryption | | *Secure Boot* | | plain dm-crypt | ZFS native encryption | |
| | | | | | | |
+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+
| Partition no. | 5 | 1 | 2 | 4 | 3 | |
+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+
| Mount point | | /boot/efi | /boot | | / | |
| | | /boot/efis/disk-part1 | | | | |
+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+
Dataset layout
~~~~~~~~~~~~~~
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| Dataset | canmount | mountpoint | container | notes |
+===========================+======================+======================+=====================================+===========================================+
| bpool | off | /boot | contains sys | |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool | off | / | contains sys | |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| bpool/sys | off | none | contains BOOT | |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool/sys | off | none | contains ROOT | sys is encryptionroot |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| bpool/sys/BOOT | off | none | contains boot environments | |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool/sys/ROOT | off | none | contains boot environments | |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool/sys/DATA | off | none | contains placeholder "default" | |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool/sys/DATA/default | off | / | contains user datasets | child datsets inherits mountpoint |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool/sys/DATA/default/ | on | /home (inherited) | no | |
| home | | | | user datasets, also called "shared |
| | | | | datasets", "persistent datasets"; also |
| | | | | include /var/lib, /srv, ... |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| bpool/sys/BOOT/default | noauto | legacy /boot | no | noauto is used to switch BE. because of |
| | | | | noauto, must use fstab to mount |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool/sys/ROOT/default | noauto | / | no | mounted by initrd zfs hook |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| bpool/sys/BOOT/be1 | noauto | legacy /boot | no | see bpool/sys/BOOT/default |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
| rpool/sys/ROOT/be1 | noauto | / | no | see rpool/sys/ROOT/default |
+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+
Encryption
~~~~~~~~~~
- Swap
Swap is always encrypted. By default, swap is encrypted
with plain dm-crypt with key generated from ``/dev/urandom``
at every boot. Swap content does not persist between reboots.
- Root pool
ZFS native encryption can be optionally enabled for ``rpool/sys``
and child datasets.
User should be aware that, ZFS native encryption does not
encrypt some metadata of the datasets.
ZFS native encryption also does not change master key when ``zfs change-key`` is invoked.
Therefore, you should wipe the disk when password is compromised to protect confidentiality.
See `zfs-load-key.8 <https://openzfs.github.io/openzfs-docs/man/8/zfs-load-key.8.html>`__
and `zfs-change-key.8 <https://openzfs.github.io/openzfs-docs/man/8/zfs-change-key.8.html>`__
for more information regarding ZFS native encryption.
Encryption is enabled at dataset creation and can not be disabled later.
- Boot pool
Boot pool can not be encrypted.
- Bootloader
Bootloader can not be encrypted.
However, with Secure Boot, bootloader
can be verified by motherboard firmware to be untempered,
which should be sufficient for most purposes.
Secure Boot is supported out-of-the-box by RHEL 8-based distro.

129
docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS/1-preparation.rst Normal file
View File

@@ -0,0 +1,129 @@
.. highlight:: sh
Preparation
======================
.. contents:: Table of Contents
:local:
#. Download a variant of Fedora 34 live image
such as LXDE spin and boot from it.
#. Disable Secure Boot. ZFS modules can not be loaded of Secure Boot is enabled.
#. Set root password or ``/root/authorized_keys``.
#. Start SSH server::
echo PermitRootLogin yes >> /etc/ssh/sshd_config
systemctl start sshd
#. Connect from another computer::
ssh root@192.168.1.19
#. Set SELinux to permissive::
setenforce 0
#. Install ``kernel-devel``::
source /etc/os-release
dnf install -y https://dl.fedoraproject.org/pub/fedora/linux/releases/${VERSION_ID}/Everything/x86_64/os/Packages/k/kernel-devel-$(uname -r).rpm
#. Add ZFS repo::
dnf install -y https://zfsonlinux.org/fedora/zfs-release.fc${VERSION_ID}.noarch.rpm
#. Install ZFS packages::
dnf install -y zfs
#. Load kernel modules::
modprobe zfs
#. Install helper script and partition tool::
dnf install -y arch-install-scripts gdisk
#. Set RHEL version::
INST_RHEL_VER=8
#. Remove Fedora repo and install target distro repo, zfs repo::
rpm -qa | grep -E ^fedora | xargs -i{} rpm -e --nodeps {}
rpm -e zfs-release
ROCKY_REPO=https://dl.rockylinux.org/pub/rocky/8.4/BaseOS/x86_64/os/Packages/
rpm -ivh \
${ROCKY_REPO}rocky-gpg-keys-8.4-26.el8.noarch.rpm \
${ROCKY_REPO}rocky-release-8.4-26.el8.noarch.rpm \
${ROCKY_REPO}rocky-repos-8.4-26.el8.noarch.rpm
RHEL_ZFS_REPO=https://zfsonlinux.org/epel/zfs-release.el8_4.noarch.rpm
dnf install -y ${RHEL_ZFS_REPO} epel-release
#. Unique pool suffix. ZFS expects pool names to be
unique, therefore it's recommended to create
pools with a unique suffix::
INST_UUID=$(dd if=/dev/urandom bs=1 count=100 2>/dev/null | tr -dc 'a-z0-9' | cut -c-6)
#. Identify this installation in ZFS filesystem path::
INST_ID=rhel
#. Target disk
List available disks with::
ls /dev/disk/by-id/*
If using virtio as disk bus, use
``/dev/disk/by-path/*`` or ``/dev/vd*``.
Declare disk array::
DISK=(/dev/disk/by-id/ata-FOO /dev/disk/by-id/nvme-BAR)
For single disk installation, use::
DISK=(/dev/disk/by-id/disk1)
#. Choose a primary disk. This disk will be used
for primary EFI partition and hibernation, default to
first disk in the array::
INST_PRIMARY_DISK=${DISK[0]}
#. Set vdev topology, possible values are:
- (not set, single disk or striped; no redundancy)
- mirror
- raidz1
- raidz2
- raidz3
::
INST_VDEV=
#. Set partition size:
Set ESP size::
INST_PARTSIZE_ESP=2 # in GB
Set boot pool size. To avoid running out of space while using
boot environments, the minimum is 4GB. Adjust the size if you
intend to use multiple kernel/distros::
INST_PARTSIZE_BPOOL=4
Set swap size. It's `recommended <https://chrisdown.name/2018/01/02/in-defence-of-swap.html>`__
to setup a swap partition. If you intend to use hibernation,
the minimum should be no less than RAM size. Skip if swap is not needed::
INST_PARTSIZE_SWAP=8
Root pool size, use all remaining disk space if not set::
INST_PARTSIZE_RPOOL=

246
docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS/2-system-installation.rst Normal file
View File

@@ -0,0 +1,246 @@
.. highlight:: sh
System Installation
======================
.. contents:: Table of Contents
:local:
#. Partition the disks.
See `Overview <0-overview.html>`__ for details::
for i in ${DISK[@]}; do
sgdisk --zap-all $i
sgdisk -n1:1M:+${INST_PARTSIZE_ESP}G -t1:EF00 $i
sgdisk -n2:0:+${INST_PARTSIZE_BPOOL}G -t2:BE00 $i
if [ "${INST_PARTSIZE_SWAP}" != "" ]; then
sgdisk -n4:0:+${INST_PARTSIZE_SWAP}G -t4:8200 $i
fi
if [ "${INST_PARTSIZE_RPOOL}" = "" ]; then
sgdisk -n3:0:0 -t3:BF00 $i
else
sgdisk -n3:0:+${INST_PARTSIZE_RPOOL}G -t3:BF00 $i
fi
sgdisk -a1 -n5:24K:+1000K -t5:EF02 $i
done
#. Create boot pool::
zpool create \
-d -o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-o ashift=12 \
-o autotrim=on \
-O acltype=posixacl \
-O canmount=off \
-O compression=lz4 \
-O devices=off \
-O normalization=formD \
-O relatime=on \
-O xattr=sa \
-O mountpoint=/boot \
-R /mnt \
bpool_$INST_UUID \
$INST_VDEV \
$(for i in ${DISK[@]}; do
printf "$i-part2 ";
done)
You should not need to customize any of the options for the boot pool.
GRUB does not support all of the zpool features. See ``spa_feature_names``
in `grub-core/fs/zfs/zfs.c
<http://git.savannah.gnu.org/cgit/grub.git/tree/grub-core/fs/zfs/zfs.c#n276>`__.
This step creates a separate boot pool for ``/boot`` with the features
limited to only those that GRUB supports, allowing the root pool to use
any/all features.
Features enabled with ``-o compatibility=grub2`` can be seen
`here <https://github.com/openzfs/zfs/blob/master/cmd/zpool/compatibility.d/grub2>`__.
#. Create root pool::
zpool create \
-d \
-o feature@empty_bpobj=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-o feature@enabled_txg=enabled \
-o feature@hole_birth=enabled \
-o feature@extensible_dataset=enabled \
-o feature@embedded_data=enabled \
-o feature@large_dnode=enabled \
-o feature@userobj_accounting=enabled \
-o feature@encryption=enabled \
-o feature@project_quota=enabled \
-o feature@spacemap_v2=enabled \
-o ashift=12 \
-o autotrim=on \
-R /mnt \
-O acltype=posixacl \
-O canmount=off \
-O compression=lz4 \
-O dnodesize=auto \
-O normalization=formD \
-O relatime=on \
-O xattr=sa \
-O mountpoint=/ \
rpool_$INST_UUID \
$INST_VDEV \
$(for i in ${DISK[@]}; do
printf "$i-part3 ";
done)
**Notes:**
- The use of ``ashift=12`` is recommended here because many drives
today have 4 KiB (or larger) physical sectors, even though they
present 512 B logical sectors. Also, a future replacement drive may
have 4 KiB physical sectors (in which case ``ashift=12`` is desirable)
or 4 KiB logical sectors (in which case ``ashift=12`` is required).
- Setting ``-O acltype=posixacl`` enables POSIX ACLs globally. If you
do not want this, remove that option, but later add
``-o acltype=posixacl`` (note: lowercase “o”) to the ``zfs create``
for ``/var/log``, as `journald requires ACLs
<https://askubuntu.com/questions/970886/journalctl-says-failed-to-search-journal-acl-operation-not-supported>`__
- Setting ``normalization=formD`` eliminates some corner cases relating
to UTF-8 filename normalization. It also implies ``utf8only=on``,
which means that only UTF-8 filenames are allowed. If you care to
support non-UTF-8 filenames, do not use this option. For a discussion
of why requiring UTF-8 filenames may be a bad idea, see `The problems
with enforced UTF-8 only filenames
<http://utcc.utoronto.ca/~cks/space/blog/linux/ForcedUTF8Filenames>`__.
- ``recordsize`` is unset (leaving it at the default of 128 KiB). If you
want to tune it (e.g. ``-o recordsize=1M``), see `these
<https://jrs-s.net/2019/04/03/on-zfs-recordsize/>`__ `various
<http://blog.programster.org/zfs-record-size>`__ `blog
<https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSFileRecordsizeGrowth>`__
`posts
<https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSRecordsizeAndCompression>`__.
- Setting ``relatime=on`` is a middle ground between classic POSIX
``atime`` behavior (with its significant performance impact) and
``atime=off`` (which provides the best performance by completely
disabling atime updates). Since Linux 2.6.30, ``relatime`` has been
the default for other filesystems. See `RedHat’s documentation
<https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/power_management_guide/relatime>`__
for further information.
- Setting ``xattr=sa`` `vastly improves the performance of extended
attributes
<https://github.com/zfsonlinux/zfs/commit/82a37189aac955c81a59a5ecc3400475adb56355>`__.
Inside ZFS, extended attributes are used to implement POSIX ACLs.
Extended attributes can also be used by user-space applications.
`They are used by some desktop GUI applications.
<https://en.wikipedia.org/wiki/Extended_file_attributes#Linux>`__
`They can be used by Samba to store Windows ACLs and DOS attributes;
they are required for a Samba Active Directory domain controller.
<https://wiki.samba.org/index.php/Setting_up_a_Share_Using_Windows_ACLs>`__
Note that ``xattr=sa`` is `Linux-specific
<https://openzfs.org/wiki/Platform_code_differences>`__. If you move your
``xattr=sa`` pool to another OpenZFS implementation besides ZFS-on-Linux,
extended attributes will not be readable (though your data will be). If
portability of extended attributes is important to you, omit the
``-O xattr=sa`` above. Even if you do not want ``xattr=sa`` for the whole
pool, it is probably fine to use it for ``/var/log``.
- Make sure to include the ``-part3`` portion of the drive path. If you
forget that, you are specifying the whole disk, which ZFS will then
re-partition, and you will lose the bootloader partition(s).
#. This section implements dataset layout as described in `overview <0-overview.html>`__.
Create root system container:
- Unencrypted::
zfs create \
-o canmount=off \
-o mountpoint=none \
rpool_$INST_UUID/$INST_ID
- Encrypted:
Pick a strong password. Once compromised, changing password will not keep your
data safe. See ``zfs-change-key(8)`` for more info::
zfs create \
-o canmount=off \
-o mountpoint=none \
-o encryption=aes-256-gcm \
-o keylocation=prompt \
-o keyformat=passphrase \
rpool_$INST_UUID/$INST_ID
Create other system datasets::
zfs create -o canmount=off -o mountpoint=none bpool_$INST_UUID/$INST_ID
zfs create -o canmount=off -o mountpoint=none bpool_$INST_UUID/$INST_ID/BOOT
zfs create -o canmount=off -o mountpoint=none rpool_$INST_UUID/$INST_ID/ROOT
zfs create -o canmount=off -o mountpoint=none rpool_$INST_UUID/$INST_ID/DATA
zfs create -o mountpoint=legacy -o canmount=noauto bpool_$INST_UUID/$INST_ID/BOOT/default
zfs create -o mountpoint=/ -o canmount=off rpool_$INST_UUID/$INST_ID/DATA/default
zfs create -o mountpoint=/ -o canmount=noauto rpool_$INST_UUID/$INST_ID/ROOT/default
zfs mount rpool_$INST_UUID/$INST_ID/ROOT/default
mkdir /mnt/boot
mount -t zfs bpool_$INST_UUID/$INST_ID/BOOT/default /mnt/boot
for i in {usr,var,var/lib};
do
zfs create -o canmount=off rpool_$INST_UUID/$INST_ID/DATA/default/$i
done
for i in {home,root,srv,usr/local,var/log,var/spool};
do
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/$i
done
chmod 750 /mnt/root
#. Format and mount ESP::
for i in ${DISK[@]}; do
mkfs.vfat -n EFI ${i}-part1
mkdir -p /mnt/boot/efis/${i##*/}-part1
mount -t vfat ${i}-part1 /mnt/boot/efis/${i##*/}-part1
done
mkdir -p /mnt/boot/efi
mount -t vfat ${INST_PRIMARY_DISK}-part1 /mnt/boot/efi
#. Create optional user data datasets to omit data from rollback::
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/games
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/www
# for GNOME
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/AccountsService
# for Docker
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/docker
# for NFS
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/nfs
# for LXC
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/lxc
# for LibVirt
zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/libvirt
##other application
# zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/$name
Add other datasets when needed, such as PostgreSQL.
#. Install base packages::
dnf --installroot=/mnt --releasever=${INST_RHEL_VER} -y install dnf
#. Install bootloader, kernel and ZFS::
cp /etc/resolv.conf /mnt/etc
arch-chroot /mnt bash --login <<EOF
dnf --releasever=${INST_RHEL_VER} -y install ${RHEL_ZFS_REPO} \
@core epel-release \
grub2-efi-x64 grub2-pc-modules grub2-efi-x64-modules shim-x64 efibootmgr \
kernel kernel-devel
dnf --releasever=${INST_RHEL_VER} -y install zfs zfs-dracut
EOF

105
docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS/3-system-configuration.rst Normal file
View File

@@ -0,0 +1,105 @@
.. highlight:: sh
System Configuration
======================
.. contents:: Table of Contents
:local:
#. Generate list of datasets for `zfs-mount-generator
<https://manpages.ubuntu.com/manpages/focal/man8/zfs-mount-generator.8.html>`__ to mount them at boot::
# tab-separated zfs properties
# see /etc/zfs/zed.d/history_event-zfs-list-cacher.sh
export \
PROPS="name,mountpoint,canmount,atime,relatime,devices,exec\
,readonly,setuid,nbmand,encroot,keylocation"
mkdir -p /mnt/etc/zfs/zfs-list.cache
zfs list -H -t filesystem -o $PROPS -r rpool_$INST_UUID > /mnt/etc/zfs/zfs-list.cache/rpool_$INST_UUID
sed -Ei "s|/mnt/?|/|" /mnt/etc/zfs/zfs-list.cache/*
#. Generate fstab::
echo bpool_$INST_UUID/$INST_ID/BOOT/default /boot zfs rw,xattr,posixacl 0 0 >> /mnt/etc/fstab
for i in ${DISK[@]}; do
echo UUID=$(blkid -s UUID -o value ${i}-part1) /boot/efis/${i##*/}-part1 vfat \
x-systemd.idle-timeout=1min,x-systemd.automount,noauto,umask=0022,fmask=0022,dmask=0022 0 1 >> /mnt/etc/fstab
done
echo UUID=$(blkid -s UUID -o value ${INST_PRIMARY_DISK}-part1) /boot/efi vfat \
x-systemd.idle-timeout=1min,x-systemd.automount,noauto,umask=0022,fmask=0022,dmask=0022 0 1 >> /mnt/etc/fstab
if [ "${INST_PARTSIZE_SWAP}" != "" ]; then
for i in ${DISK[@]}; do
echo ${i##*/}-part4-swap ${i}-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256,discard >> /mnt/etc/crypttab
echo /dev/mapper/${i##*/}-part4-swap none swap x-systemd.requires=cryptsetup.target,defaults 0 0 >> /mnt/etc/fstab
done
fi
By default, systemd will halt boot process if any entry in ``/etc/fstab`` fails
to mount. This is unnecessary for mirrored EFI boot partitions.
With the above mount options, systemd will skip mounting them at boot,
only mount them on demand when accessed.
#. Configure dracut::
echo 'add_dracutmodules+=" zfs "' > /mnt/etc/dracut.conf.d/zfs.conf
#. Interactively set locale, keymap, timezone, hostname and root password::
rm -f /mnt/etc/localtime
systemd-firstboot --root=/mnt --force --prompt --root-password=PASSWORD
This can be non-interactive, see man page for details::
rm -f /mnt/etc/localtime
systemd-firstboot --root=/mnt --force \
--locale="en_US.UTF-8" --locale-messages="en_US.UTF-8" \
--keymap=us --timezone="Europe/Berlin" --hostname=myHost \
--root-password=PASSWORD --root-shell=/bin/bash
``systemd-firstboot`` have bugs, root password is set below.
#. Install locale package, example for English locale::
dnf --installroot=/mnt install -y glibc-minimal-langpack glibc-langpack-en
Program will show errors if not installed.
#. Enable ZFS services::
systemctl enable zfs-import-scan.service zfs-import.target zfs-mount zfs-zed zfs.target --root=/mnt
#. By default SSH server is enabled, allowing root login by password,
disable SSH server::
systemctl disable sshd --root=/mnt
#. Chroot::
echo "INST_PRIMARY_DISK=$INST_PRIMARY_DISK
INST_LINVAR=$INST_LINVAR
INST_UUID=$INST_UUID
INST_ID=$INST_ID
INST_VDEV=$INST_VDEV
TERM=xterm" > /mnt/root/chroot
echo DISK=\($(for i in ${DISK[@]}; do printf "$i "; done)\) >> /mnt/root/chroot
arch-chroot /mnt bash --login
unalias -a
#. Source variables::
source /root/chroot
#. Relabel filesystem on reboot::
fixfiles -F onboot
#. Set root password::
passwd
#. Build modules::
ls -1 /lib/modules \
| while read kernel_version; do
dkms autoinstall -k $kernel_version
done

20
docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS/4-optional-configuration.rst Normal file
View File

@@ -0,0 +1,20 @@
.. highlight:: sh
Optional Configuration
======================
.. contents:: Table of Contents
:local:
Skip to `bootloader <5-bootloader.html>`__ section if
no optional configuration is needed.
Boot environment manager
~~~~~~~~~~~~~~~~~~~~~~~~
A boot environment is a dataset which contains a bootable
instance of an operating system.
`bieaz <https://gitlab.com/m_zhou/bieaz/-/releases/>`__ can
be installed to manage boot environments. Download and install
prebuilt rpm file.

177
docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS/5-bootloader.rst Normal file
View File

@@ -0,0 +1,177 @@
.. highlight:: sh
Bootloader
======================
.. contents:: Table of Contents
:local:
Apply workarounds
~~~~~~~~~~~~~~~~~~~~
Currently GRUB has multiple compatibility problems with ZFS,
especially with regards to newer ZFS features.
Workarounds have to be applied.
#. grub2-probe fails to get canonical path
When persistent device names ``/dev/disk/by-id/*`` are used
with ZFS, GRUB will fail to resolve the path of the boot pool
device. Error::
# /usr/bin/grub2-probe: error: failed to get canonical path of `/dev/virtio-pci-0000:06:00.0-part3'.
Solution::
echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile.d/zpool_vdev_name_path.sh
source /etc/profile.d/zpool_vdev_name_path.sh
#. Pool name missing
See `this bug report <https://savannah.gnu.org/bugs/?59614>`__.
Root pool name is missing from ``root=ZFS=rpool_$INST_UUID/ROOT/default``
kernel cmdline in generated ``grub.cfg`` file.
A workaround is to replace the pool name detection with ``zdb``
command::
sed -i "s|rpool=.*|rpool=\`zdb -l \${GRUB_DEVICE} \| grep -E '[[:blank:]]name' \| cut -d\\\' -f 2\`|" /etc/grub.d/10_linux
Install GRUB
~~~~~~~~~~~~~~~~~~~~
#. If using virtio disk, add driver to initrd::
echo 'filesystems+=" virtio_blk "' >> /etc/dracut.conf.d/fs.conf
#. Generate initrd::
rm -f /etc/zfs/zpool.cache
touch /etc/zfs/zpool.cache
chmod a-w /etc/zfs/zpool.cache
chattr +i /etc/zfs/zpool.cache
ls -1 /lib/modules \
| while read kernel_version; do
dracut --force --kver $kernel_version
done
#. When in doubt, install both legacy boot
and EFI.
#. Load ZFS modules and disable BLS::
echo 'GRUB_ENABLE_BLSCFG=false' >> /etc/default/grub
#. Create GRUB boot directory, in ESP and boot pool::
mkdir -p /boot/efi/EFI/rocky # EFI GRUB dir
mkdir -p /boot/efi/EFI/rocky/grub2 # legacy GRUB dir
mkdir -p /boot/grub2
Boot environment-specific configuration (kernel, etc)
is stored in ``/boot/grub2/grub.cfg``, enabling rollback.
#. If using legacy booting, install GRUB to every disk::
for i in ${DISK[@]}; do
grub2-install --boot-directory /boot/efi/EFI/rocky --target=i386-pc $i
done
#. If using EFI::
for i in ${DISK[@]}; do
efibootmgr -cgp 1 -l "\EFI\rocky\shimx64.efi" \
-L "rocky-${i##*/}" -d ${i}
done
cp -r /usr/lib/grub/x86_64-efi/ /boot/efi/EFI/rocky
#. Generate GRUB Menu:
Apply workaround::
tee /etc/grub.d/09_fix_root_on_zfs <<EOF
#!/bin/sh
echo 'insmod zfs'
echo 'set root=(hd0,gpt2)'
EOF
chmod +x /etc/grub.d/09_fix_root_on_zfs
Generate menu::
grub2-mkconfig -o /boot/efi/EFI/rocky/grub.cfg
cp /boot/efi/EFI/rocky/grub.cfg /boot/efi/EFI/rocky/grub2/grub.cfg
cp /boot/efi/EFI/rocky/grub.cfg /boot/grub2/grub.cfg
#. For both legacy and EFI booting: mirror ESP content::
ESP_MIRROR=$(mktemp -d)
cp -r /boot/efi/EFI $ESP_MIRROR
for i in /boot/efis/*; do
cp -r $ESP_MIRROR/EFI $i
done
#. Notes for GRUB on RHEL
To support Secure Boot, GRUB has been heavily modified by Fedora,
namely:
- ``grub2-install`` is `disabled for UEFI <https://bugzilla.redhat.com/show_bug.cgi?id=1917213>`__
- Only a static, signed version of bootloader is copied to EFI system partition
- This signed bootloader does not have built-in support for either ZFS or LUKS containers
- This signed bootloader only loads configuration from ``/boot/efi/EFI/fedora/grub.cfg``
Unrelated to Secure Boot, GRUB has also been modified to provide optional
support for `systemd bootloader specification (bls) <https://systemd.io/BOOT_LOADER_SPECIFICATION/>`__.
Currently ``blscfg.mod`` is incompatible with root on ZFS.
Also see `Fedora docs for GRUB
<https://docs.fedoraproject.org/en-US/fedora/rawhide/system-administrators-guide/kernel-module-driver-configuration/Working_with_the_GRUB_2_Boot_Loader/>`__.
Finish Installation
~~~~~~~~~~~~~~~~~~~~
#. Exit chroot::
exit
#. Take a snapshot of the clean installation for future use::
zfs snapshot -r rpool_$INST_UUID/$INST_ID@install
zfs snapshot -r bpool_$INST_UUID/$INST_ID@install
#. Unmount EFI system partition::
umount /mnt/boot/efi
umount /mnt/boot/efis/*
#. Export pools::
zpool export bpool_$INST_UUID
zpool export rpool_$INST_UUID
#. Reboot::
reboot
#. After reboot, consider adding a normal user::
myUser=UserName
zfs create $(df --output=source /home | tail -n +2)/${myUser}
useradd -MUd /home/${myUser} -c 'My Name' ${myUser}
zfs allow -u ${myUser} mount,snapshot,destroy $(df --output=source /home | tail -n +2)/${myUser}
chown -R ${myUser}:${myUser} /home/${myUser}
chmod 700 /home/${myUser}
restorecon /home/${myUser}
passwd ${myUser}
Set up cron job to snapshot user home everyday::
dnf install cronie
systemctl enable --now crond
crontab -eu ${myUser}
#@daily zfs snap $(df --output=source /home/${myUser} | tail -n +2)@$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)
zfs list -t snapshot -S creation $(df --output=source /home/${myUser} | tail -n +2)
Install package groups::
dnf group list # query package groups
dnf group install 'Virtualization Host'

226
docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS/6-recovery.rst Normal file
View File

@@ -0,0 +1,226 @@
.. highlight:: sh
Recovery
======================
.. contents:: Table of Contents
:local:
GRUB Tips
-------------
Boot from GRUB rescue
~~~~~~~~~~~~~~~~~~~~~~~
If bootloader file is damaged, it's still possible
to boot computer with GRUB rescue image.
This section is also applicable if you are in
``grub rescue>``.
#. On another computer, generate rescue image with::
pacman -S --needed mtools libisoburn grub
grub-mkrescue -o grub-rescue.img
dd if=grub-rescue.img of=/dev/your-usb-stick
Boot computer from the rescue media.
Both legacy and EFI mode are supported.
Skip this step if you are in GRUB rescue.
#. List available disks with ``ls`` command::
grub> ls (hd # press tab
Possible devices are:
hd0 hd1 hd2 hd3
If you are dropped to GRUB rescue instead of
booting from GRUB rescue image, boot disk can be found
out with::
echo $root
# cryto0
# hd0,gpt2
GRUB configuration is loaded from::
echo $prefix
# (crypto0)/sys/BOOT/default@/grub
# (hd0,gpt2)/sys/BOOT/default@/grub
#. List partitions by pressing tab key:
.. code-block:: text
grub> ls (hd0 # press tab
Possible partitions are:
Device hd0: No known filesystem detected - Sector size 512B - Total size 20971520KiB
Partition hd0,gpt1: Filesystem type fat - Label `EFI', UUID 0DF5-3A76 - Partition start at 1024KiB - Total size 1048576KiB
Partition hd0,gpt2: No known filesystem detected - Partition start at 1049600KiB - Total size 4194304KiB
- If boot pool is encrypted:
Unlock it with ``cryptomount``::
grub> insmod luks
grub> cryptomount hd0,gpt2
Attempting to decrypt master key...
Enter passphrase for hd0,gpt2 (af5a240e13e24483acf02600d61e0f36):
Slot 1 opened
Unlocked LUKS container is ``(crypto0)``:
.. code-block:: text
grub> ls (crypto0)
Device crypto0: Filesystem type zfs - Label `bpool_ip3tdb' - Last modification
time 2021-05-03 12:14:08 Monday, UUID f14d7bdf89fe21fb - Sector size 512B -
Total size 4192256KiB
- If boot pool is not encrypted:
.. code-block:: text
grub> ls (hd0,gpt2)
Device hd0,gpt2: Filesystem type zfs - Label `bpool_ip3tdb' - Last modification
time 2021-05-03 12:14:08 Monday, UUID f14d7bdf89fe21fb - Sector size 512B -
Total size 4192256KiB
#. List boot environments nested inside ``bpool/$INST_ID/BOOT``::
grub> ls (crypto0)/sys/BOOT
@/ default/ be0/
#. Instruct GRUB to load configuration from ``be0`` boot environment
then enter normal mode::
grub> prefix=(crypto0)/sys/BOOT/be0/@/grub
grub> insmod normal
grub> normal
#. GRUB menu should now appear.
#. After entering system, `reinstall GRUB <#grub-installation>`__.
Switch GRUB prefix when disk fails
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you are using LUKS encrypted boot pool with multiple disks,
the primary disk failed, GRUB will fail to load configuration.
If there's still enough redundancy for the boot pool, try fix
GRUB with the following method:
#. Ensure ``Slot 1 opened`` message
is shown
.. code-block:: text
Welcome to GRUB!
error: no such cryptodisk found.
Attempting to decrypt master key...
Enter passphrase for hd0,gpt2 (c0987ea1a51049e9b3056622804de62a):
Slot 1 opened
error: disk `cryptouuid/47ed1b7eb0014bc9a70aede3d8714faf' not found.
Entering rescue mode...
grub rescue>
If ``error: access denied.`` is shown,
try re-enter password with::
grub rescue> cryptomount hd0,gpt2
#. Check prefix::
grub rescue > set
# prefix=(cryptouuid/47ed1b7eb0014bc9a70aede3d8714faf)/sys/BOOT/be0@/grub
# root=cryptouuid/47ed1b7eb0014bc9a70aede3d8714faf
#. Set correct ``prefix`` and ``root`` by replacing
``cryptouuid/UUID`` with ``crypto0``::
grub rescue> prefix=(crypto0)/sys/BOOT/default@/grub
grub rescue> root=crypto0
#. Boot GRUB::
grub rescue> insmod normal
grub rescue> normal
GRUB should then boot normally.
#. After entering system, edit ``/etc/fstab`` to promote
one backup to ``/boot/efi``.
#. Make the change to ``prefix`` and ``root``
permanent by `reinstalling GRUB <#grub-installation>`__.
Access system in chroot
-----------------------
#. Go through `preparation <1-preparation.html>`__.
#. Import and unlock root and boot pool::
zpool import -NR /mnt rpool_$INST_UUID
zpool import -NR /mnt bpool_$INST_UUID
If using password::
zfs load-key rpool_$INST_UUID/$INST_ID
If using keyfile::
zfs load-key -L file:///path/to/keyfile rpool_$INST_UUID/$INST_ID
#. Find the current boot environment::
zfs list
BE=default
#. Mount root filesystem::
zfs mount rpool_$INST_UUID/$INST_ID/ROOT/$BE
#. chroot into the system::
rhel-chroot /mnt /bin/bash --login
zfs mount -a
mount -a
#. Finish rescue. See `finish installation <#finish-installation>`__.
Backup and migrate existing installation
----------------------------------------
With the help of `zfs send
<https://openzfs.github.io/openzfs-docs/man/8/zfs-send.8.html>`__
it is relatively easy to perform a system backup and migration.
#. Create a snapshot of root file system::
zfs snapshot -r rpool/rhel@backup
zfs snapshot -r bpool/rhel@backup
#. Save snapshot to a file or pipe to SSH::
zfs send --options rpool/rhel@backup > /backup/rhel-rpool
zfs send --options bpool/rhel@backup > /backup/rhel-bpool
#. Re-create partitions and root/boot
pool on target system.
#. Restore backup::
zfs recv rpool_new/rhel < /backup/rhel-rpool
zfs recv bpool_new/rhel < /backup/rhel-bpool
#. Chroot and reinstall bootloader.
#. Update pool name in ``/etc/fstab``, ``/boot/grub/grub.cfg``
and ``/etc/zfs/zfs-list.cache/*``.
#. Update device name, etc, in ``/etc/fstab`` and ``/etc/crypttab``.

160
docs/Getting Started/RHEL-based distro/index.rst Normal file
View File

@@ -0,0 +1,160 @@
RHEL-based distro
=======================
Contents
--------
.. toctree::
:maxdepth: 1
:glob:
*
`DKMS`_ or `kABI-tracking kmod`_ style packages are provided for RHEL and
CentOS based distributions from the OpenZFS repository. These packages are
updated as new versions are released. Only the current repository for each
major release is updated with new packages. Packages are available for the
following configurations:
| **EL Releases:** 6, 7.9, 8.4
| **Architectures:** x86_64
To simplify installation a *zfs-release* package is provided which includes
a zfs.repo configuration file and public signing key. All official OpenZFS
packages are signed using this key, and by default yum or dnf will verify a
package's signature before allowing it be to installed. Users are strongly
encouraged to verify the authenticity of the ZFS on Linux public key using
the fingerprint listed here.
| **Location:** /etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
| **EL6 Package:** `zfs-release.el6.noarch.rpm`_ (zfs-0.8.x)
| **EL7.9 Package:** `zfs-release.el7_9.noarch.rpm`_ (zfs-0.8.x)
| **EL8.4 Package:** `zfs-release.el8_4.noarch.rpm`_ (zfs-2.0.5)
| **Archived Repositories:** `el7_5`_, `el7_6`_, `el7_7`_, `el7_8`_, `el8_0`_, `el8_1`_, `el8_2`_, `el8_3`_
| **Download from:**
`pgp.mit.edu <https://pgp.mit.edu/pks/lookup?search=0xF14AB620&op=index&fingerprint=on>`__
| **Fingerprint:** C93A FFFD 9F3F 7B03 C310 CEB6 A9D5 A1C0 F14A B620
For RHEL/CentOS versions 6 and 7 run::
yum install https://zfsonlinux.org/epel/zfs-release$(rpm -E %distro).noarch.rpm
rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
And for RHEL/CentOS 8 and newer::
dnf install https://zfsonlinux.org/epel/zfs-release$(rpm -E %distro).noarch.rpm
rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
After installing the *zfs-release* package and verifying the public key
users can opt to install either the DKMS or kABI-tracking kmod style packages.
DKMS packages are recommended for users running a non-distribution kernel or
for users who wish to apply local customizations to OpenZFS. For most users
the kABI-tracking kmod packages are recommended in order to avoid needing to
rebuild OpenZFS for every kernel update.
DKMS
----
To install DKMS style packages issue the following commands. First add the
`EPEL repository`_ which provides DKMS by installing the *epel-release*
package, then the *kernel-devel* and *zfs* packages. Note that it is
important to make sure that the matching *kernel-devel* package is installed
for the running kernel since DKMS requires it to build OpenZFS.
For RHEL/CentOS versions 6 and 7 run::
yum install epel-release kernel-devel zfs
And for RHEL/CentOS 8 and newer::
dnf install epel-release kernel-devel zfs
.. note::
When switching from DKMS to kABI-tracking kmods first uninstall the
existing DKMS packages. This should remove the kernel modules for all
installed kernels, then the kABI-tracking kmods can be installed as
described in the section below.
kABI-tracking kmod
------------------
By default the *zfs-release* package is configured to install DKMS style
packages so they will work with a wide range of kernels. In order to
install the kABI-tracking kmods the default repository must be switched
from *zfs* to *zfs-kmod*. Keep in mind that the kABI-tracking kmods are
only verified to work with the distribution provided kernel.
For RHEL/CentOS versions 6 and 7 run::
yum-config-manager --disable zfs
yum-config-manager --enable zfs-kmod
yum install zfs
And for RHEL/CentOS 8 and newer::
dnf config-manager --disable zfs
dnf config-manager --enable zfs-kmod
dnf install zfs
By default the OpenZFS kernel modules are automatically loaded when a ZFS
pool is detected. If you would prefer to always load the modules at boot
time you must create an ``/etc/modules-load.d/zfs.conf`` file::
echo zfs >/etc/modules-load.d/zfs.conf
.. note::
When updating to a new RHEL/CentOS minor release the existing kmod
packages may not work due to upstream kABI changes in the kernel.
After upgrading users must uninstall OpenZFS and then reinstall it
from the matching repository as described in this section.
Testing Repositories
--------------------
In addition to the primary *zfs* repository a *zfs-testing* repository
is available. This repository, which is disabled by default, contains
the latest version of OpenZFS which is under active development. These
packages are made available in order to get feedback from users regarding
the functionality and stability of upcoming releases. These packages
**should not** be used on production systems. Packages from the testing
repository can be installed as follows.
For RHEL/CentOS versions 6 and 7 run::
yum-config-manager --enable zfs-testing
yum install kernel-devel zfs
And for RHEL/CentOS 8 and newer::
dnf config-manager --enable zfs-testing
dnf install kernel-devel zfs
.. note::
Use *zfs-testing* for DKMS packages and *zfs-testing-kmod*
kABI-tracking kmod packages.
RHEL 8-based distro Root on ZFS
-------------------------------
`Start here <RHEL%208-based%20distro%20Root%20on%20ZFS/0-overview.html>`__.
.. toctree::
:maxdepth: 1
:glob:
RHEL 8-based distro Root on ZFS/*
.. _kABI-tracking kmod: https://elrepoproject.blogspot.com/2016/02/kabi-tracking-kmod-packages.html
.. _DKMS: https://en.wikipedia.org/wiki/Dynamic_Kernel_Module_Support
.. _zfs-release.el6.noarch.rpm: https://zfsonlinux.org/epel/zfs-release.el6.noarch.rpm
.. _zfs-release.el7_9.noarch.rpm: https://zfsonlinux.org/epel/zfs-release.el7_9.noarch.rpm
.. _zfs-release.el8_4.noarch.rpm: https://zfsonlinux.org/epel/zfs-release.el8_4.noarch.rpm
.. _el7_5: https://zfsonlinux.org/epel/zfs-release.el7_5.noarch.rpm
.. _el7_6: https://zfsonlinux.org/epel/zfs-release.el7_6.noarch.rpm
.. _el7_7: https://zfsonlinux.org/epel/zfs-release.el7_7.noarch.rpm
.. _el7_8: https://zfsonlinux.org/epel/zfs-release.el7_8.noarch.rpm
.. _el8_0: https://zfsonlinux.org/epel/zfs-release.el8_0.noarch.rpm
.. _el8_1: https://zfsonlinux.org/epel/zfs-release.el8_1.noarch.rpm
.. _el8_2: https://zfsonlinux.org/epel/zfs-release.el8_2.noarch.rpm
.. _el8_3: https://zfsonlinux.org/epel/zfs-release.el8_3.noarch.rpm
.. _EPEL repository: https://fedoraproject.org/wiki/EPEL