diff --git a/docs/Getting Started/Debian/Debian Buster Root on ZFS.rst b/docs/Getting Started/Debian/Debian Buster Root on ZFS.rst index ec1e846..70b0b3a 100644 --- a/docs/Getting Started/Debian/Debian Buster Root on ZFS.rst +++ b/docs/Getting Started/Debian/Debian Buster Root on ZFS.rst @@ -23,7 +23,7 @@ System Requirements iso) `__ - `A 64-bit kernel is strongly encouraged. `__ -- Installing on a drive which presents 4KiB logical sectors (a “4Kn” +- Installing on a drive which presents 4 KiB logical sectors (a “4Kn” drive) only works with UEFI booting. This not unique to ZFS. `GRUB does not and will not work on 4Kn with legacy (BIOS) booting. `__ @@ -39,7 +39,7 @@ Support If you need help, reach out to the community using the :doc:`zfs-discuss mailing list <../../Project and Community/Mailing Lists>` or IRC at -`#zfsonlinux `__. on `freenode +`#zfsonlinux `__ on `freenode `__. If you have a bug report or feature request related to this HOWTO, please `file a new issue and mention @rlaager `__. @@ -74,20 +74,13 @@ Contributing Encryption ~~~~~~~~~~ -This guide supports three different encryption options: unencrypted, -LUKS (full-disk encryption), and ZFS native encryption. With any option, -all ZFS features are fully available. +This guide supports three different encryption options: unencrypted, ZFS +native encryption, and LUKS. With any option, all ZFS features are fully +available. Unencrypted does not encrypt anything, of course. With no encryption happening, this option naturally has the best performance. -LUKS encrypts almost everything: the OS, swap, home directories, and -anything else. The only unencrypted data is the bootloader, kernel, and -initrd. The system cannot boot without the passphrase being entered at -the console. Performance is good, but LUKS sits underneath ZFS, so if -multiple disks (mirror or raidz topologies) are used, the data has to be -encrypted once per disk. - ZFS native encryption encrypts the data and most metadata in the root pool. It does not encrypt dataset or snapshot names or properties. The boot pool is not encrypted at all, but it only contains the bootloader, @@ -97,6 +90,12 @@ without the passphrase being entered at the console. Performance is good. As the encryption happens in ZFS, even if multiple disks (mirror or raidz topologies) are used, the data only has to be encrypted once. +LUKS encrypts almost everything. The only unencrypted data is the bootloader, +kernel, and initrd. The system cannot boot without the passphrase being +entered at the console. Performance is good, but LUKS sits underneath ZFS, so +if multiple disks (mirror or raidz topologies) are used, the data has to be +encrypted once per disk. + Step 1: Prepare The Install Environment --------------------------------------- @@ -212,12 +211,7 @@ commands for all the disks which will be part of the pool. -o feature@large_blocks=enabled \ -o feature@lz4_compress=enabled \ -o feature@spacemap_histogram=enabled \ - -o feature@userobj_accounting=enabled \ -o feature@zpool_checkpoint=enabled \ - -o feature@spacemap_v2=enabled \ - -o feature@project_quota=enabled \ - -o feature@resilver_defer=enabled \ - -o feature@allocation_classes=enabled \ -O acltype=posixacl -O canmount=off -O compression=lz4 -O devices=off \ -O normalization=formD -O relatime=on -O xattr=sa \ -O mountpoint=/ -R /mnt bpool ${DISK}-part3 @@ -230,7 +224,7 @@ GRUB does not support all of the zpool features. See This step creates a separate boot pool for ``/boot`` with the features limited to only those that GRUB supports, allowing the root pool to use any/all features. Note that GRUB opens the pool read-only, so all -read-only compatible features are "supported" by GRUB. +read-only compatible features are “supported” by GRUB. **Hints:** @@ -241,6 +235,24 @@ read-only compatible features are "supported" by GRUB. - The pool name is arbitrary. If changed, the new name must be used consistently. The ``bpool`` convention originated in this HOWTO. +**Feature Notes:** + +- The ``allocation_classes`` feature should be safe to use. However, unless + one is using it (i.e. a ``special`` vdev), there is no point to enabling it. + It is extremely unlikely that someone would use this feature for a boot + pool. If one cares about speeding up the boot pool, it would make more sense + to put the whole pool on the faster disk rather than using it as a + ``special`` vdev. +- The ``project_quota`` feature has been tested and is safe to use. This + feature is extremely unlikely to matter for the boot pool. +- The ``resilver_defer`` should be safe but the boot pool is small enough that + it is unlikely to be necessary. +- The ``spacemap_v2`` feature has been tested and is safe to use. The boot + pool is small, so this does not matter in practice. +- As a read-only compatible feature, the ``userobj_accounting`` feature should + be compatible in theory, but in practice, GRUB can fail with an “invalid + dnode type” error. This feature does not matter for ``/boot`` anyway. + 2.5 Create the root pool: Choose one of the following options: @@ -252,7 +264,15 @@ Choose one of the following options: -O dnodesize=auto -O normalization=formD -O relatime=on -O xattr=sa \ -O mountpoint=/ -R /mnt rpool ${DISK}-part4 -2.5b LUKS:: +2.5b ZFS native encryption:: + + zpool create -o ashift=12 \ + -O acltype=posixacl -O canmount=off -O compression=lz4 \ + -O dnodesize=auto -O normalization=formD -O relatime=on -O xattr=sa \ + -O encryption=aes-256-gcm -O keylocation=prompt -O keyformat=passphrase \ + -O mountpoint=/ -R /mnt rpool ${DISK}-part4 + +2.5c LUKS:: apt install --yes cryptsetup cryptsetup luksFormat -c aes-xts-plain64 -s 512 -h sha256 ${DISK}-part4 @@ -262,22 +282,16 @@ Choose one of the following options: -O dnodesize=auto -O normalization=formD -O relatime=on -O xattr=sa \ -O mountpoint=/ -R /mnt rpool /dev/mapper/luks1 -2.5c ZFS native encryption:: - - zpool create -o ashift=12 \ - -O acltype=posixacl -O canmount=off -O compression=lz4 \ - -O dnodesize=auto -O normalization=formD -O relatime=on -O xattr=sa \ - -O encryption=aes-256-gcm -O keylocation=prompt -O keyformat=passphrase \ - -O mountpoint=/ -R /mnt rpool ${DISK}-part4 +**Notes:** - The use of ``ashift=12`` is recommended here because many drives - today have 4KiB (or larger) physical sectors, even though they - present 512B logical sectors. Also, a future replacement drive may - have 4KiB physical sectors (in which case ``ashift=12`` is desirable) - or 4KiB logical sectors (in which case ``ashift=12`` is required). + today have 4 KiB (or larger) physical sectors, even though they + present 512 B logical sectors. Also, a future replacement drive may + have 4 KiB physical sectors (in which case ``ashift=12`` is desirable) + or 4 KiB logical sectors (in which case ``ashift=12`` is required). - Setting ``-O acltype=posixacl`` enables POSIX ACLs globally. If you do not want this, remove that option, but later add - ``-o acltype=posixacl`` (note: lowercase "o") to the ``zfs create`` + ``-o acltype=posixacl`` (note: lowercase “o”) to the ``zfs create`` for ``/var/log``, as `journald requires ACLs `__ - Setting ``normalization=formD`` eliminates some corner cases relating @@ -287,11 +301,18 @@ Choose one of the following options: of why requiring UTF-8 filenames may be a bad idea, see `The problems with enforced UTF-8 only filenames `__. +- ``recordsize`` is unset (leaving it at the default of 128 KiB). If you want to + tune it (e.g. ``-o recordsize=1M``), see `these + `__ `various + `__ `blog + `__ + `posts + `__. - Setting ``relatime=on`` is a middle ground between classic POSIX ``atime`` behavior (with its significant performance impact) and ``atime=off`` (which provides the best performance by completely disabling atime updates). Since Linux 2.6.30, ``relatime`` has been - the default for other filesystems. See `RedHat's + the default for other filesystems. See `RedHat’s documentation `__ for further information. - Setting ``xattr=sa`` `vastly improves the performance of extended @@ -314,16 +335,17 @@ Choose one of the following options: - Make sure to include the ``-part4`` portion of the drive path. If you forget that, you are specifying the whole disk, which ZFS will then re-partition, and you will lose the bootloader partition(s). +- ZFS native encryption defaults to ``aes-256-ccm``, but `the default has + changed upstream `__ + to ``aes-256-gcm``. `AES-GCM seems to be generally preferred over AES-CCM + `__, + `is faster now + `__, and + `will be even faster in the future + `__. - For LUKS, the key size chosen is 512 bits. However, XTS mode requires two keys, so the LUKS key is split in half. Thus, ``-s 512`` means AES-256. -- ZFS native encryption uses ``aes-256-ccm`` by default. `AES-GCM seems - to be generally preferred over - AES-CCM `__, - `is faster - now `__, - and `will be even faster in the - future `__. - Your passphrase will likely be the weakest link. Choose wisely. See `section 5 of the cryptsetup FAQ `__ @@ -351,9 +373,10 @@ Step 3: System Installation On Solaris systems, the root filesystem is cloned and the suffix is incremented for major system changes through ``pkg image-update`` or -``beadm``. Similar functionality for APT is possible but currently -unimplemented. Even without such a tool, it can still be used for -manually created clones. +``beadm``. Similar functionality has been implemented in Ubuntu 20.04 with the +``zsys`` tool, though its dataset layout is more complicated. Even without +such a tool, the `rpool/ROOT` and `bpool/BOOT` containers can still be used +for manually created clones. 3.2 Create filesystem datasets for the root and boot filesystems:: @@ -428,17 +451,15 @@ If this system will use NFS (locking):: zfs create -o com.sun:auto-snapshot=false rpool/var/lib/nfs A tmpfs is recommended later, but if you want a separate dataset for -/tmp:: +``/tmp``:: zfs create -o com.sun:auto-snapshot=false rpool/tmp chmod 1777 /mnt/tmp -The primary goal of this dataset layout is to separate the OS from user -data. This allows the root filesystem to be rolled back without rolling -back user data such as logs (in ``/var/log``). This will be especially -important if/when a ``beadm`` or similar utility is integrated. The -``com.sun.auto-snapshot`` setting is used by some ZFS snapshot utilities -to exclude transient data. +The primary goal of this dataset layout is to separate the OS from user data. +This allows the root filesystem to be rolled back without rolling back user +data. The ``com.sun.auto-snapshot`` setting is used by some ZFS +snapshot utilities to exclude transient data. If you do nothing extra, ``/tmp`` will be stored as part of the root filesystem. Alternatively, you can create a separate dataset for @@ -459,8 +480,9 @@ of a working system into the new ZFS root. Step 4: System Configuration ---------------------------- -4.1 Configure the hostname (change ``HOSTNAME`` to the desired -hostname):: +4.1 Configure the hostname: + +Replace ``HOSTNAME`` with the desired hostname:: echo HOSTNAME > /mnt/etc/hostname vi /mnt/etc/hosts @@ -491,9 +513,7 @@ Adjust NAME below to match your interface name:: Customize this file if the system is not a DHCP client. -4.3 Configure the package sources: - -:: +4.3 Configure the package sources:: vi /mnt/etc/apt/sources.list @@ -550,16 +570,15 @@ Even if you prefer a non-English system language, always ensure that apt install --yes zfs-initramfs echo REMAKE_INITRD=yes > /etc/dkms/zfs.conf -4.7 For LUKS installs only, setup crypttab:: +4.7 For LUKS installs only, setup ``/etc/crypttab``:: apt install --yes cryptsetup echo luks1 UUID=$(blkid -s UUID -o value ${DISK}-part4) none \ luks,discard,initramfs > /etc/crypttab -- The use of ``initramfs`` is a work-around for `cryptsetup does not - support - ZFS `__. +The use of ``initramfs`` is a work-around for `cryptsetup does not support ZFS +`__.\ **Hint:** If you are creating a mirror or raidz topology, repeat the ``/etc/crypttab`` entries for ``luks2``, etc. adjusting for each disk. @@ -572,7 +591,7 @@ Choose one of the following options: apt install --yes grub-pc -Install GRUB to the disk(s), not the partition(s). +Select (using the space bar) all of the disks (not partitions) in your pool. 4.8b Install GRUB for UEFI booting:: @@ -584,20 +603,27 @@ Install GRUB to the disk(s), not the partition(s). mount /boot/efi apt install --yes grub-efi-amd64 shim-signed -- The ``-s 1`` for ``mkdosfs`` is only necessary for drives which - present 4 KiB logical sectors (“4Kn” drives) to meet the minimum - cluster size (given the partition size of 512 MiB) for FAT32. It also - works fine on drives which present 512 B sectors. +**Notes:** -**Note:** If you are creating a mirror or raidz topology, this step only -installs GRUB on the first disk. The other disk(s) will be handled -later. +- The ``-s 1`` for ``mkdosfs`` is only necessary for drives which present + 4 KiB logical sectors (“4Kn” drives) to meet the minimum cluster size + (given the partition size of 512 MiB) for FAT32. It also works fine on + drives which present 512 B sectors. +- For a mirror or raidz topology, this step only installs GRUB on the + first disk. The other disk(s) will be handled later. -4.9 Set a root password:: +4.9 (Optional): Remove os-prober:: + + dpkg --purge os-prober + +This avoids error messages from `update-grub`. `os-prober` is only necessary +in dual-boot configurations. + +4.10 Set a root password:: passwd -4.10 Enable importing bpool +4.11 Enable importing bpool This ensures that ``bpool`` is always imported, regardless of whether ``/etc/zfs/zpool.cache`` exists, whether it is in the cachefile or not, @@ -626,7 +652,7 @@ or whether ``zfs-import-scan.service`` is enabled. systemctl enable zfs-import-bpool.service -4.11 Optional (but recommended): Mount a tmpfs to /tmp +4.12 Optional (but recommended): Mount a tmpfs to ``/tmp`` If you chose to create a ``/tmp`` dataset above, skip this step, as they are mutually exclusive choices. Otherwise, you can put ``/tmp`` on a @@ -637,7 +663,7 @@ tmpfs (RAM filesystem) by enabling the ``tmp.mount`` unit. cp /usr/share/systemd/tmp.mount /etc/systemd/system/ systemctl enable tmp.mount -4.12 Optional (but kindly requested): Install popcon +4.13 Optional (but kindly requested): Install popcon The ``popularity-contest`` package reports the list of packages install on your system. Showing that ZFS is popular may be helpful in terms of @@ -658,12 +684,12 @@ Step 5: GRUB Installation 5.2 Refresh the initrd files:: - update-initramfs -u -k all + update-initramfs -c -k all -**Note:** When using LUKS, this will print "WARNING could not determine -root device from /etc/fstab". This is because `cryptsetup does not -support -ZFS `__. +**Note:** When using LUKS, this will print “WARNING could not determine +root device from /etc/fstab”. This is because `cryptsetup does not +support ZFS +`__. 5.3 Workaround GRUB's missing zpool-features support:: @@ -686,7 +712,7 @@ working, you can undo these changes, if desired. **Note:** Ignore errors from ``osprober``, if present. -5.6 Install the boot loader +5.6 Install the boot loader: 5.6a For legacy (BIOS) booting, install GRUB to the MBR:: @@ -705,11 +731,7 @@ If you are creating a mirror or raidz topology, repeat the It is not necessary to specify the disk here. If you are creating a mirror or raidz topology, the additional disks will be handled later. -5.7 Verify that the ZFS module is installed:: - - ls /boot/grub/*/zfs.mod - -5.8 Fix filesystem mount ordering +5.7 Fix filesystem mount ordering: Until there is support for mounting ``/boot`` in the initramfs, we also need to mount that, because it was marked ``canmount=noauto``. Also, @@ -738,7 +760,7 @@ Everything else applies to both BIOS and UEFI booting:: ln -s /usr/lib/zfs-linux/zed.d/history_event-zfs-list-cacher.sh /etc/zfs/zed.d zed -F & -Verify that zed updated the cache by making sure this is not empty:: +Verify that ``zed`` updated the cache by making sure this is not empty:: cat /etc/zfs/zfs-list.cache/rpool @@ -746,12 +768,12 @@ If it is empty, force a cache update and check again:: zfs set canmount=noauto rpool/ROOT/debian -Stop zed:: +Stop ``zed``:: fg Press Ctrl-C. -Fix the paths to eliminate /mnt:: +Fix the paths to eliminate ``/mnt``:: sed -Ei "s|/mnt/?|/|" /etc/zfs/zfs-list.cache/rpool @@ -781,32 +803,31 @@ filesystems:: reboot -6.5 Wait for the newly installed system to boot normally. Login as root. +Wait for the newly installed system to boot normally. Login as root. -6.6 Create a user account:: +6.5 Create a user account: - zfs create rpool/home/YOURUSERNAME - adduser YOURUSERNAME - cp -a /etc/skel/. /home/YOURUSERNAME - chown -R YOURUSERNAME:YOURUSERNAME /home/YOURUSERNAME +Replace ``username`` with your desired username:: -6.7 Add your user account to the default set of groups for an -administrator:: + zfs create rpool/home/username + adduser username - usermod -a -G audio,cdrom,dip,floppy,netdev,plugdev,sudo,video YOURUSERNAME + cp -a /etc/skel/. /home/username + chown -R username:username /home/username + usermod -a -G audio,cdrom,dip,floppy,netdev,plugdev,sudo,video username -6.8 Mirror GRUB +6.6 Mirror GRUB If you installed to multiple disks, install GRUB on the additional disks: -6.8a For legacy (BIOS) booting:: +6.6a For legacy (BIOS) booting:: dpkg-reconfigure grub-pc Hit enter until you get to the device selection screen. Select (using the space bar) all of the disks (not partitions) in your pool. -6.8b For UEFI booting:: +6.6b For UEFI booting:: umount /boot/efi @@ -992,8 +1013,8 @@ not hotplug pool members. See `https://github.com/zfsonlinux/zfs/issues/330 `__. Most LSI cards are perfectly compatible with ZoL. If your card has this -glitch, try setting ZFS_INITRD_PRE_MOUNTROOT_SLEEP=X in -/etc/default/zfs. The system will wait X seconds for all drives to +glitch, try setting ``ZFS_INITRD_PRE_MOUNTROOT_SLEEP=X`` in +``/etc/default/zfs``. The system will wait ``X`` seconds for all drives to appear before importing the pool. Areca @@ -1001,7 +1022,7 @@ Areca Systems that require the ``arcsas`` blob driver should add it to the ``/etc/initramfs-tools/modules`` file and run -``update-initramfs -u -k all``. +``update-initramfs -c -k all``. Upgrade or downgrade the Areca driver if something like ``RIP: 0010:[] [] native_read_tsc+0x6/0x20`` diff --git a/docs/Getting Started/Debian/Debian Stretch Root on ZFS.rst b/docs/Getting Started/Debian/Debian Stretch Root on ZFS.rst index 1c0dafc..bf311a3 100644 --- a/docs/Getting Started/Debian/Debian Stretch Root on ZFS.rst +++ b/docs/Getting Started/Debian/Debian Stretch Root on ZFS.rst @@ -43,7 +43,7 @@ Support If you need help, reach out to the community using the :doc:`zfs-discuss mailing list <../../Project and Community/Mailing Lists>` or IRC at -`#zfsonlinux `__. on `freenode +`#zfsonlinux `__ on `freenode `__. If you have a bug report or feature request related to this HOWTO, please `file a new issue and mention @rlaager `__. diff --git a/docs/Getting Started/Ubuntu/Ubuntu 16.04 Root on ZFS.rst b/docs/Getting Started/Ubuntu/Ubuntu 16.04 Root on ZFS.rst index d333a58..4e9e132 100644 --- a/docs/Getting Started/Ubuntu/Ubuntu 16.04 Root on ZFS.rst +++ b/docs/Getting Started/Ubuntu/Ubuntu 16.04 Root on ZFS.rst @@ -10,7 +10,7 @@ Overview Newer release available ~~~~~~~~~~~~~~~~~~~~~~~ -- See :doc:`Ubuntu 18.04 Root on ZFS <./Ubuntu 18.04 Root on ZFS>` for new +- See :doc:`Ubuntu 20.04 Root on ZFS <./Ubuntu 20.04 Root on ZFS>` for new installs. Caution @@ -41,12 +41,12 @@ deduplication is a permanent change that cannot be easily reverted. Support ~~~~~~~ -If you need help, reach out to the community using the `zfs-discuss -mailing list `__ -or IRC at #zfsonlinux on `freenode `__. If you -have a bug report or feature request related to this HOWTO, please `file -a new issue `__ and -mention @rlaager. +If you need help, reach out to the community using the :doc:`zfs-discuss +mailing list <../../Project and Community/Mailing Lists>` or IRC at +`#zfsonlinux `__ on `freenode +`__. If you have a bug report or feature request +related to this HOWTO, please `file a new issue and mention @rlaager +`__. Contributing ~~~~~~~~~~~~ diff --git a/docs/Getting Started/Ubuntu/Ubuntu 18.04 Root on ZFS.rst b/docs/Getting Started/Ubuntu/Ubuntu 18.04 Root on ZFS.rst index d7ecae4..aa3e8d4 100644 --- a/docs/Getting Started/Ubuntu/Ubuntu 18.04 Root on ZFS.rst +++ b/docs/Getting Started/Ubuntu/Ubuntu 18.04 Root on ZFS.rst @@ -9,6 +9,12 @@ Ubuntu 18.04 Root on ZFS Overview -------- +Newer release available +~~~~~~~~~~~~~~~~~~~~~~~ + +- See :doc:`Ubuntu 20.04 Root on ZFS <./Ubuntu 20.04 Root on ZFS>` for new + installs. + Caution ~~~~~~~ @@ -22,7 +28,7 @@ System Requirements - `Ubuntu 18.04.3 ("Bionic") Desktop CD `__ (*not* any server images) -- Installing on a drive which presents 4KiB logical sectors (a “4Kn” +- Installing on a drive which presents 4 KiB logical sectors (a “4Kn” drive) only works with UEFI booting. This not unique to ZFS. `GRUB does not and will not work on 4Kn with legacy (BIOS) booting. `__ @@ -38,7 +44,7 @@ Support If you need help, reach out to the community using the :doc:`zfs-discuss mailing list <../../Project and Community/Mailing Lists>` or IRC at -`#zfsonlinux `__. on `freenode +`#zfsonlinux `__ on `freenode `__. If you have a bug report or feature request related to this HOWTO, please `file a new issue and mention @rlaager `__. @@ -74,17 +80,16 @@ Encryption ~~~~~~~~~~ This guide supports two different encryption options: unencrypted and -LUKS (full-disk encryption). ZFS native encryption has not yet been -released. With either option, all ZFS features are fully available. +LUKS (full-disk encryption). With either option, all ZFS features are fully +available. ZFS native encryption is not available in Ubuntu 18.04. Unencrypted does not encrypt anything, of course. With no encryption happening, this option naturally has the best performance. -LUKS encrypts almost everything: the OS, swap, home directories, and -anything else. The only unencrypted data is the bootloader, kernel, and -initrd. The system cannot boot without the passphrase being entered at -the console. Performance is good, but LUKS sits underneath ZFS, so if -multiple disks (mirror or raidz topologies) are used, the data has to be +LUKS encrypts almost everything. The only unencrypted data is the bootloader, +kernel, and initrd. The system cannot boot without the passphrase being +entered at the console. Performance is good, but LUKS sits underneath ZFS, so +if multiple disks (mirror or raidz topologies) are used, the data has to be encrypted once per disk. Step 1: Prepare The Install Environment @@ -192,7 +197,6 @@ commands for all the disks which will be part of the pool. -o feature@large_blocks=enabled \ -o feature@lz4_compress=enabled \ -o feature@spacemap_histogram=enabled \ - -o feature@userobj_accounting=enabled \ -O acltype=posixacl -O canmount=off -O compression=lz4 -O devices=off \ -O normalization=formD -O relatime=on -O xattr=sa \ -O mountpoint=/ -R /mnt bpool ${DISK}-part3 @@ -205,7 +209,7 @@ GRUB does not support all of the zpool features. See This step creates a separate boot pool for ``/boot`` with the features limited to only those that GRUB supports, allowing the root pool to use any/all features. Note that GRUB opens the pool read-only, so all -read-only compatible features are "supported" by GRUB. +read-only compatible features are “supported” by GRUB. **Hints:** @@ -216,6 +220,12 @@ read-only compatible features are "supported" by GRUB. - The pool name is arbitrary. If changed, the new name must be used consistently. The ``bpool`` convention originated in this HOWTO. +**Feature Notes:** + +- As a read-only compatible feature, the ``userobj_accounting`` feature should + be compatible in theory, but in practice, GRUB can fail with an “invalid + dnode type” error. This feature does not matter for ``/boot`` anyway. + 2.5 Create the root pool: Choose one of the following options: @@ -236,14 +246,16 @@ Choose one of the following options: -O dnodesize=auto -O normalization=formD -O relatime=on -O xattr=sa \ -O mountpoint=/ -R /mnt rpool /dev/mapper/luks1 +**Notes:** + - The use of ``ashift=12`` is recommended here because many drives - today have 4KiB (or larger) physical sectors, even though they - present 512B logical sectors. Also, a future replacement drive may - have 4KiB physical sectors (in which case ``ashift=12`` is desirable) - or 4KiB logical sectors (in which case ``ashift=12`` is required). + today have 4 KiB (or larger) physical sectors, even though they + present 512 B logical sectors. Also, a future replacement drive may + have 4 KiB physical sectors (in which case ``ashift=12`` is desirable) + or 4 KiB logical sectors (in which case ``ashift=12`` is required). - Setting ``-O acltype=posixacl`` enables POSIX ACLs globally. If you do not want this, remove that option, but later add - ``-o acltype=posixacl`` (note: lowercase "o") to the ``zfs create`` + ``-o acltype=posixacl`` (note: lowercase “o”) to the ``zfs create`` for ``/var/log``, as `journald requires ACLs `__ - Setting ``normalization=formD`` eliminates some corner cases relating @@ -253,11 +265,18 @@ Choose one of the following options: of why requiring UTF-8 filenames may be a bad idea, see `The problems with enforced UTF-8 only filenames `__. +- ``recordsize`` is unset (leaving it at the default of 128 KiB). If you want to + tune it (e.g. ``-o recordsize=1M``), see `these + `__ `various + `__ `blog + `__ + `posts + `__. - Setting ``relatime=on`` is a middle ground between classic POSIX ``atime`` behavior (with its significant performance impact) and ``atime=off`` (which provides the best performance by completely disabling atime updates). Since Linux 2.6.30, ``relatime`` has been - the default for other filesystems. See `RedHat's + the default for other filesystems. See `RedHat’s documentation `__ for further information. - Setting ``xattr=sa`` `vastly improves the performance of extended @@ -310,9 +329,10 @@ Step 3: System Installation On Solaris systems, the root filesystem is cloned and the suffix is incremented for major system changes through ``pkg image-update`` or -``beadm``. Similar functionality for APT is possible but currently -unimplemented. Even without such a tool, it can still be used for -manually created clones. +``beadm``. Similar functionality has been implemented in Ubuntu 20.04 with the +``zsys`` tool, though its dataset layout is more complicated. Even without +such a tool, the `rpool/ROOT` and `bpool/BOOT` containers can still be used +for manually created clones. 3.2 Create filesystem datasets for the root and boot filesystems:: @@ -387,17 +407,15 @@ If this system will use NFS (locking):: zfs create -o com.sun:auto-snapshot=false rpool/var/lib/nfs A tmpfs is recommended later, but if you want a separate dataset for -/tmp:: +``/tmp``:: zfs create -o com.sun:auto-snapshot=false rpool/tmp chmod 1777 /mnt/tmp -The primary goal of this dataset layout is to separate the OS from user -data. This allows the root filesystem to be rolled back without rolling -back user data such as logs (in ``/var/log``). This will be especially -important if/when a ``beadm`` or similar utility is integrated. The -``com.sun.auto-snapshot`` setting is used by some ZFS snapshot utilities -to exclude transient data. +The primary goal of this dataset layout is to separate the OS from user data. +This allows the root filesystem to be rolled back without rolling back user +data. The ``com.sun.auto-snapshot`` setting is used by some ZFS +snapshot utilities to exclude transient data. If you do nothing extra, ``/tmp`` will be stored as part of the root filesystem. Alternatively, you can create a separate dataset for @@ -418,8 +436,9 @@ of a working system into the new ZFS root. Step 4: System Configuration ---------------------------- -4.1 Configure the hostname (change ``HOSTNAME`` to the desired -hostname):: +4.1 Configure the hostname: + +Replace ``HOSTNAME`` with the desired hostname:: echo HOSTNAME > /mnt/etc/hostname vi /mnt/etc/hosts @@ -459,14 +478,10 @@ Customize this file if the system is not a DHCP client. .. code-block:: sourceslist - deb http://archive.ubuntu.com/ubuntu bionic main universe - deb-src http://archive.ubuntu.com/ubuntu bionic main universe - - deb http://security.ubuntu.com/ubuntu bionic-security main universe - deb-src http://security.ubuntu.com/ubuntu bionic-security main universe - - deb http://archive.ubuntu.com/ubuntu bionic-updates main universe - deb-src http://archive.ubuntu.com/ubuntu bionic-updates main universe + deb http://archive.ubuntu.com/ubuntu bionic main restricted universe multiverse + deb http://archive.ubuntu.com/ubuntu bionic-updates main restricted universe multiverse + deb http://archive.ubuntu.com/ubuntu bionic-backports main restricted universe multiverse + deb http://security.ubuntu.com/ubuntu bionic-security main restricted universe multiverse 4.4 Bind the virtual filesystems from the LiveCD environment to the new system and ``chroot`` into it:: @@ -490,7 +505,7 @@ Even if you prefer a non-English system language, always ensure that dpkg-reconfigure tzdata -If you prefer nano over vi, install it:: +If you prefer ``nano`` over ``vi``, install it:: apt install --yes nano @@ -502,16 +517,15 @@ If you prefer nano over vi, install it:: **Hint:** For the HWE kernel, install ``linux-image-generic-hwe-18.04`` instead of ``linux-image-generic``. -4.7 For LUKS installs only, setup crypttab:: +4.7 For LUKS installs only, setup ``/etc/crypttab``:: apt install --yes cryptsetup echo luks1 UUID=$(blkid -s UUID -o value ${DISK}-part4) none \ luks,discard,initramfs > /etc/crypttab -- The use of ``initramfs`` is a work-around for `cryptsetup does not - support - ZFS `__. +The use of ``initramfs`` is a work-around for `cryptsetup does not support ZFS +`__. **Hint:** If you are creating a mirror or raidz topology, repeat the ``/etc/crypttab`` entries for ``luks2``, etc. adjusting for each disk. @@ -524,7 +538,7 @@ Choose one of the following options: apt install --yes grub-pc -Install GRUB to the disk(s), not the partition(s). +Select (using the space bar) all of the disks (not partitions) in your pool. 4.8b Install GRUB for UEFI booting:: @@ -536,20 +550,27 @@ Install GRUB to the disk(s), not the partition(s). mount /boot/efi apt install --yes grub-efi-amd64-signed shim-signed -- The ``-s 1`` for ``mkdosfs`` is only necessary for drives which - present 4 KiB logical sectors (“4Kn” drives) to meet the minimum - cluster size (given the partition size of 512 MiB) for FAT32. It also - works fine on drives which present 512 B sectors. +**Notes:** -**Note:** If you are creating a mirror or raidz topology, this step only -installs GRUB on the first disk. The other disk(s) will be handled -later. +- The ``-s 1`` for ``mkdosfs`` is only necessary for drives which present + 4 KiB logical sectors (“4Kn” drives) to meet the minimum cluster size + (given the partition size of 512 MiB) for FAT32. It also works fine on + drives which present 512 B sectors. +- For a mirror or raidz topology, this step only installs GRUB on the + first disk. The other disk(s) will be handled later. -4.9 Set a root password:: +4.9 (Optional): Remove os-prober:: + + dpkg --purge os-prober + +This avoids error messages from `update-grub`. `os-prober` is only necessary +in dual-boot configurations. + +4.10 Set a root password:: passwd -4.10 Enable importing bpool +4.11 Enable importing bpool This ensures that ``bpool`` is always imported, regardless of whether ``/etc/zfs/zpool.cache`` exists, whether it is in the cachefile or not, @@ -578,7 +599,7 @@ or whether ``zfs-import-scan.service`` is enabled. systemctl enable zfs-import-bpool.service -4.11 Optional (but recommended): Mount a tmpfs to /tmp +4.12 Optional (but recommended): Mount a tmpfs to ``/tmp`` If you chose to create a ``/tmp`` dataset above, skip this step, as they are mutually exclusive choices. Otherwise, you can put ``/tmp`` on a @@ -589,9 +610,7 @@ tmpfs (RAM filesystem) by enabling the ``tmp.mount`` unit. cp /usr/share/systemd/tmp.mount /etc/systemd/system/ systemctl enable tmp.mount -4.12 Setup system groups: - -:: +4.13 Setup system groups:: addgroup --system lpadmin addgroup --system sambashare @@ -605,12 +624,12 @@ Step 5: GRUB Installation 5.2 Refresh the initrd files:: - update-initramfs -u -k all + update-initramfs -c -k all -**Note:** When using LUKS, this will print "WARNING could not determine -root device from /etc/fstab". This is because `cryptsetup does not -support -ZFS `__. +**Note:** When using LUKS, this will print “WARNING could not determine +root device from /etc/fstab”. This is because `cryptsetup does not +support ZFS +`__. 5.3 Workaround GRUB's missing zpool-features support:: @@ -636,7 +655,7 @@ working, you can undo these changes, if desired. **Note:** Ignore errors from ``osprober``, if present. -5.6 Install the boot loader +5.6 Install the boot loader: 5.6a For legacy (BIOS) booting, install GRUB to the MBR:: @@ -655,11 +674,7 @@ If you are creating a mirror or raidz topology, repeat the It is not necessary to specify the disk here. If you are creating a mirror or raidz topology, the additional disks will be handled later. -5.7 Verify that the ZFS module is installed:: - - ls /boot/grub/*/zfs.mod - -5.8 Fix filesystem mount ordering +5.7 Fix filesystem mount ordering: `Until ZFS gains a systemd mount generator `__, there are @@ -734,32 +749,31 @@ filesystems:: reboot -6.5 Wait for the newly installed system to boot normally. Login as root. +Wait for the newly installed system to boot normally. Login as root. -6.6 Create a user account:: +6.5 Create a user account: - zfs create rpool/home/YOURUSERNAME - adduser YOURUSERNAME - cp -a /etc/skel/. /home/YOURUSERNAME - chown -R YOURUSERNAME:YOURUSERNAME /home/YOURUSERNAME +Replace ``username`` with your desired username:: -6.7 Add your user account to the default set of groups for an -administrator:: + zfs create rpool/home/username + adduser username - usermod -a -G adm,cdrom,dip,lpadmin,plugdev,sambashare,sudo YOURUSERNAME + cp -a /etc/skel/. /home/username + chown -R username:username /home/username + usermod -a -G audio,cdrom,dip,floppy,netdev,plugdev,sudo,video username -6.8 Mirror GRUB +6.6 Mirror GRUB If you installed to multiple disks, install GRUB on the additional disks: -6.8a For legacy (BIOS) booting:: +6.6a For legacy (BIOS) booting:: dpkg-reconfigure grub-pc Hit enter until you get to the device selection screen. Select (using the space bar) all of the disks (not partitions) in your pool. -6.8b For UEFI booting:: +6.6b For UEFI booting:: umount /boot/efi @@ -841,7 +855,8 @@ Choose one of the following options: **Hint**: If you are installing a full GUI environment, you will likely want to manage your network with NetworkManager:: - vi /etc/netplan/01-netcfg.yaml + rm /mnt/etc/netplan/01-netcfg.yaml + vi /etc/netplan/01-network-manager-all.yaml .. code-block:: yaml @@ -965,8 +980,8 @@ not hotplug pool members. See `https://github.com/zfsonlinux/zfs/issues/330 `__. Most LSI cards are perfectly compatible with ZoL. If your card has this -glitch, try setting ZFS_INITRD_PRE_MOUNTROOT_SLEEP=X in -/etc/default/zfs. The system will wait X seconds for all drives to +glitch, try setting ``ZFS_INITRD_PRE_MOUNTROOT_SLEEP=X`` in +``/etc/default/zfs``. The system will wait ``X`` seconds for all drives to appear before importing the pool. Areca @@ -974,7 +989,7 @@ Areca Systems that require the ``arcsas`` blob driver should add it to the ``/etc/initramfs-tools/modules`` file and run -``update-initramfs -u -k all``. +``update-initramfs -c -k all``. Upgrade or downgrade the Areca driver if something like ``RIP: 0010:[] [] native_read_tsc+0x6/0x20`` diff --git a/docs/Getting Started/Ubuntu/Ubuntu 20.04 Root on ZFS.rst b/docs/Getting Started/Ubuntu/Ubuntu 20.04 Root on ZFS.rst new file mode 100644 index 0000000..ef74425 --- /dev/null +++ b/docs/Getting Started/Ubuntu/Ubuntu 20.04 Root on ZFS.rst @@ -0,0 +1,1054 @@ +.. highlight:: sh + +Ubuntu 20.04 Root on ZFS +======================== + +.. contents:: Table of Contents + :local: + +Overview +-------- + +Ubuntu Installer +~~~~~~~~~~~~~~~~ + +The Ubuntu installer has `support for root-on-ZFS +`__. +This HOWTO produces nearly identical results as the Ubuntu installer because of +`bidirectional collaboration +`__. + +If you want a single-disk, unencrypted, desktop install, use the installer. It +is far easier and faster than doing everything by hand. + +If you want a ZFS native encrypted, desktop install, you can `trivially edit +the installer +`__. +The ``-o recordsize=1M`` there is unrelated to encryption; omit that unless you +understand it. `Hopefully the installer will gain encryption support in the +future `__. + +If you want to setup a mirror or raidz topology, use LUKS encryption, and/or +install a server (no desktop GUI), use this HOWTO. + +Caution +~~~~~~~ + +- This HOWTO uses a whole physical disk. +- Do not use these instructions for dual-booting. +- Backup your data. Any existing data will be lost. + +System Requirements +~~~~~~~~~~~~~~~~~~~ + +- `Ubuntu 20.04 (“Focal”) Desktop CD + `__ + (*not* any server images) +- Installing on a drive which presents 4 KiB logical sectors (a “4Kn” + drive) only works with UEFI booting. This not unique to ZFS. `GRUB + does not and will not work on 4Kn with legacy (BIOS) + booting. `__ + +Computers that have less than 2 GiB of memory run ZFS slowly. 4 GiB of +memory is recommended for normal performance in basic workloads. If you +wish to use deduplication, you will need `massive amounts of +RAM `__. Enabling +deduplication is a permanent change that cannot be easily reverted. + +Support +~~~~~~~ + +If you need help, reach out to the community using the :doc:`zfs-discuss +mailing list <../../Project and Community/Mailing Lists>` or IRC at +`#zfsonlinux `__ on `freenode +`__. If you have a bug report or feature request +related to this HOWTO, please `file a new issue and mention @rlaager +`__. + +Contributing +~~~~~~~~~~~~ + +1. Fork and clone: https://github.com/openzfs/openzfs-docs + +2. Install the tools:: + + # On Debian 11 / Ubuntu 20.04 or later: + sudo apt install python3-sphinx python3-sphinx-issues python3-sphinx-rtd-theme + + # On earlier releases: + sudo apt install pip3 + pip3 install -r requirements.txt + # Add ~/.local/bin to your $PATH, e.g. by adding this to ~/.bashrc: + PATH=$HOME/.local/bin:$PATH + +3. Make your changes. + +4. Test:: + + cd docs + make html + sensible-browser _build/html/index.html + +5. ``git commit --signoff`` to a branch, ``git push``, and create a pull + request. Mention @rlaager. + +Encryption +~~~~~~~~~~ + +This guide supports three different encryption options: unencrypted, ZFS +native encryption, and LUKS. With any option, all ZFS features are fully +available. + +Unencrypted does not encrypt anything, of course. With no encryption +happening, this option naturally has the best performance. + +ZFS native encryption encrypts the data and most metadata in the root +pool. It does not encrypt dataset or snapshot names or properties. The +boot pool is not encrypted at all, but it only contains the bootloader, +kernel, and initrd. (Unless you put a password in ``/etc/fstab``, the +initrd is unlikely to contain sensitive data.) The system cannot boot +without the passphrase being entered at the console. Performance is +good. As the encryption happens in ZFS, even if multiple disks (mirror +or raidz topologies) are used, the data only has to be encrypted once. + +LUKS encrypts almost everything. The only unencrypted data is the bootloader, +kernel, and initrd. The system cannot boot without the passphrase being +entered at the console. Performance is good, but LUKS sits underneath ZFS, so +if multiple disks (mirror or raidz topologies) are used, the data has to be +encrypted once per disk. + +Step 1: Prepare The Install Environment +--------------------------------------- + +1.1 Boot the Ubuntu Live CD. Select Try Ubuntu. Connect your system to +the Internet as appropriate (e.g. join your WiFi network). Open a +terminal (press Ctrl-Alt-T). + +1.2 Setup and update the repositories:: + + sudo apt-add-repository universe + sudo apt update + +1.3 Optional: Install and start the OpenSSH server in the Live CD +environment: + +If you have a second system, using SSH to access the target system can +be convenient:: + + passwd + # There is no current password; hit enter at that prompt. + sudo apt install --yes openssh-server + +**Hint:** You can find your IP address with +``ip addr show scope global | grep inet``. Then, from your main machine, +connect with ``ssh ubuntu@IP``. + +1.4 Become root:: + + sudo -i + +1.5 Install ZFS in the Live CD environment:: + + apt install --yes debootstrap gdisk zfs-initramfs + systemctl stop zed + +Step 2: Disk Formatting +----------------------- + +2.1 Set a variable with the disk name:: + + DISK=/dev/disk/by-id/scsi-SATA_disk1 + +Always use the long ``/dev/disk/by-id/*`` aliases with ZFS. Using the +``/dev/sd*`` device nodes directly can cause sporadic import failures, +especially on systems that have more than one storage pool. + +**Hints:** + +- ``ls -la /dev/disk/by-id`` will list the aliases. +- Are you doing this in a virtual machine? If your virtual disk is + missing from ``/dev/disk/by-id``, use ``/dev/vda`` if you are using + KVM with virtio; otherwise, read the + `troubleshooting <#troubleshooting>`__ section. + +2.2 If you are re-using a disk, clear it as necessary: + +If the disk was previously used in an MD array, zero the superblock:: + + apt install --yes mdadm + mdadm --zero-superblock --force $DISK + +Clear the partition table:: + + sgdisk --zap-all $DISK + +2.3 Create bootloader partition(s):: + + sgdisk -n1:1M:+512M -t1:EF00 $DISK + +**Note:** This partition is setup for UEFI support. For legacy (BIOS) booting, +this will allow you to move the disk(s) to a new system/motherboard in the +future without having to rebuild the pool (and restore your data from a +backup). Additionally, this is used for `/boot/grub` in single-disk scenarios, +as discussed below. + +For legacy (BIOS) booting:: + + sgdisk -a1 -n5:24K:+1000K -t5:EF02 $DISK + +**Note:** For simplicity and forward compatibility, this HOWTO uses GPT +partition labels for both UEFI and legacy (BIOS) booting. The Ubuntu installer +uses an MBR label for legacy (BIOS) booting. + +2.4 Create a partition for swap: + +Previous versions of this HOWTO put swap on a zvol. `Ubuntu recommends against +this configuration due to deadlocks. +`__ There is +`a bug report upstream `__. + +Putting swap on a partition gives up the benefit of ZFS checksums (for your +swap). That is probably the right trade-off given the reports of ZFS deadlocks +with swap. If you are bothered by this, simply do not enable swap. + +Choose one of the following options if you want swap: + +2.4a For a single-disk install:: + + sgdisk -n2:0:+500M -t2:8200 $DISK + +2.4b For a mirror or raidz topology:: + + sgdisk -n2:0:+500M -t2:FD00 $DISK + +2.5 Create a boot pool partition:: + + sgdisk -n3:0:+2G -t3:BE00 $DISK + +The Ubuntu installer uses 5% of the disk space constrained to a minimum of +500 MiB and a maximum of 2 GiB. `Making this too small (and 500 MiB might be +too small) can result in an inability to upgrade the kernel. +`__ + +2.6 Create a root pool partition: + +Choose one of the following options: + +2.6a Unencrypted or ZFS native encryption:: + + sgdisk -n4:0:0 -t4:BF00 $DISK + +2.6b LUKS:: + + sgdisk -n4:0:0 -t4:8309 $DISK + +If you are creating a mirror or raidz topology, repeat the partitioning +commands for all the disks which will be part of the pool. + +2.7 Create the boot pool:: + + zpool create -o ashift=12 -d \ + -o feature@async_destroy=enabled \ + -o feature@bookmarks=enabled \ + -o feature@embedded_data=enabled \ + -o feature@empty_bpobj=enabled \ + -o feature@enabled_txg=enabled \ + -o feature@extensible_dataset=enabled \ + -o feature@filesystem_limits=enabled \ + -o feature@hole_birth=enabled \ + -o feature@large_blocks=enabled \ + -o feature@lz4_compress=enabled \ + -o feature@spacemap_histogram=enabled \ + -o feature@zpool_checkpoint=enabled \ + -O acltype=posixacl -O canmount=off -O compression=lz4 -O devices=off \ + -O normalization=formD -O relatime=on -O xattr=sa \ + -O mountpoint=/boot -R /mnt bpool ${DISK}-part3 + +You should not need to customize any of the options for the boot pool. + +GRUB does not support all of the zpool features. See +``spa_feature_names`` in +`grub-core/fs/zfs/zfs.c `__. +This step creates a separate boot pool for ``/boot`` with the features +limited to only those that GRUB supports, allowing the root pool to use +any/all features. Note that GRUB opens the pool read-only, so all +read-only compatible features are “supported” by GRUB. + +**Hints:** + +- If you are creating a mirror or raidz topology, create the pool using + ``zpool create ... bpool mirror /dev/disk/by-id/scsi-SATA_disk1-part3 /dev/disk/by-id/scsi-SATA_disk2-part3`` + (or replace ``mirror`` with ``raidz``, ``raidz2``, or ``raidz3`` and + list the partitions from additional disks). +- The pool name is arbitrary. If changed, the new name must be used + consistently. The ``bpool`` convention originated in this HOWTO. + +**Feature Notes:** + +- The ``allocation_classes`` feature should be safe to use. However, unless + one is using it (i.e. a ``special`` vdev), there is no point to enabling it. + It is extremely unlikely that someone would use this feature for a boot + pool. If one cares about speeding up the boot pool, it would make more sense + to put the whole pool on the faster disk rather than using it as a + ``special`` vdev. +- The ``project_quota`` feature has been tested and is safe to use. This + feature is extremely unlikely to matter for the boot pool. +- The ``resilver_defer`` should be safe but the boot pool is small enough that + it is unlikely to be necessary. +- The ``spacemap_v2`` feature has been tested and is safe to use. The boot + pool is small, so this does not matter in practice. +- As a read-only compatible feature, the ``userobj_accounting`` feature should + be compatible in theory, but in practice, GRUB can fail with an “invalid + dnode type” error. This feature does not matter for ``/boot`` anyway. +- The ``zpool_checkpoint`` feature has been tested and is safe to use. The + Ubuntu installer does not use it. This HOWTO does, as the feature may be + desirable for the boot pool. + +2.8 Create the root pool: + +Choose one of the following options: + +2.8a Unencrypted:: + + zpool create -o ashift=12 \ + -O acltype=posixacl -O canmount=off -O compression=lz4 \ + -O dnodesize=auto -O normalization=formD -O relatime=on -O xattr=sa \ + -O mountpoint=/ -R /mnt rpool ${DISK}-part4 + +2.8b ZFS native encryption:: + + zpool create -o ashift=12 \ + -O acltype=posixacl -O canmount=off -O compression=lz4 \ + -O dnodesize=auto -O normalization=formD -O relatime=on -O xattr=sa \ + -O encryption=aes-256-gcm -O keylocation=prompt -O keyformat=passphrase \ + -O mountpoint=/ -R /mnt rpool ${DISK}-part4 + +2.8c LUKS:: + + cryptsetup luksFormat -c aes-xts-plain64 -s 512 -h sha256 ${DISK}-part4 + cryptsetup luksOpen ${DISK}-part4 luks1 + zpool create -o ashift=12 \ + -O acltype=posixacl -O canmount=off -O compression=lz4 \ + -O dnodesize=auto -O normalization=formD -O relatime=on -O xattr=sa \ + -O mountpoint=/ -R /mnt rpool /dev/mapper/luks1 + +**Notes:** + +- The use of ``ashift=12`` is recommended here because many drives + today have 4 KiB (or larger) physical sectors, even though they + present 512 B logical sectors. Also, a future replacement drive may + have 4 KiB physical sectors (in which case ``ashift=12`` is desirable) + or 4 KiB logical sectors (in which case ``ashift=12`` is required). +- Setting ``-O acltype=posixacl`` enables POSIX ACLs globally. If you + do not want this, remove that option, but later add + ``-o acltype=posixacl`` (note: lowercase “o”) to the ``zfs create`` + for ``/var/log``, as `journald requires + ACLs `__ +- Setting ``normalization=formD`` eliminates some corner cases relating + to UTF-8 filename normalization. It also implies ``utf8only=on``, + which means that only UTF-8 filenames are allowed. If you care to + support non-UTF-8 filenames, do not use this option. For a discussion + of why requiring UTF-8 filenames may be a bad idea, see `The problems + with enforced UTF-8 only + filenames `__. +- ``recordsize`` is unset (leaving it at the default of 128 KiB). If you want to + tune it (e.g. ``-o recordsize=1M``), see `these + `__ `various + `__ `blog + `__ + `posts + `__. +- Setting ``relatime=on`` is a middle ground between classic POSIX + ``atime`` behavior (with its significant performance impact) and + ``atime=off`` (which provides the best performance by completely + disabling atime updates). Since Linux 2.6.30, ``relatime`` has been + the default for other filesystems. See `RedHat’s + documentation `__ + for further information. +- Setting ``xattr=sa`` `vastly improves the performance of extended + attributes `__. + Inside ZFS, extended attributes are used to implement POSIX ACLs. + Extended attributes can also be used by user-space applications. + `They are used by some desktop GUI + applications. `__ + `They can be used by Samba to store Windows ACLs and DOS attributes; + they are required for a Samba Active Directory domain + controller. `__ + Note that ``xattr=sa`` is + `Linux-specific `__. + If you move your ``xattr=sa`` pool to another OpenZFS implementation + besides ZFS-on-Linux, extended attributes will not be readable + (though your data will be). If portability of extended attributes is + important to you, omit the ``-O xattr=sa`` above. Even if you do not + want ``xattr=sa`` for the whole pool, it is probably fine to use it + for ``/var/log``. +- Make sure to include the ``-part4`` portion of the drive path. If you + forget that, you are specifying the whole disk, which ZFS will then + re-partition, and you will lose the bootloader partition(s). +- ZFS native encryption defaults to ``aes-256-ccm``, but `the default has + changed upstream `__ + to ``aes-256-gcm``. `AES-GCM seems to be generally preferred over AES-CCM + `__, + `is faster now + `__, and + `will be even faster in the future + `__. +- For LUKS, the key size chosen is 512 bits. However, XTS mode requires + two keys, so the LUKS key is split in half. Thus, ``-s 512`` means + AES-256. +- Your passphrase will likely be the weakest link. Choose wisely. See + `section 5 of the cryptsetup + FAQ `__ + for guidance. + +**Hints:** + +- If you are creating a mirror or raidz topology, create the pool using + ``zpool create ... rpool mirror /dev/disk/by-id/scsi-SATA_disk1-part4 /dev/disk/by-id/scsi-SATA_disk2-part4`` + (or replace ``mirror`` with ``raidz``, ``raidz2``, or ``raidz3`` and + list the partitions from additional disks). For LUKS, use + ``/dev/mapper/luks1``, ``/dev/mapper/luks2``, etc., which you will + have to create using ``cryptsetup``. +- The pool name is arbitrary. If changed, the new name must be used + consistently. On systems that can automatically install to ZFS, the + root pool is named ``rpool`` by default. + +Step 3: System Installation +--------------------------- + +3.1 Create filesystem datasets to act as containers:: + + zfs create -o canmount=off -o mountpoint=none rpool/ROOT + zfs create -o canmount=off -o mountpoint=none bpool/BOOT + +3.2 Create filesystem datasets for the root and boot filesystems:: + + UUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null | + tr -dc 'a-z0-9' | cut -c-6) + + zfs create -o canmount=noauto -o mountpoint=/ \ + -o com.ubuntu.zsys:bootfs=yes \ + -o com.ubuntu.zsys:last-used=$(date +%s) rpool/ROOT/ubuntu_$UUID + zfs mount rpool/ROOT/ubuntu_$UUID + + zfs create -o canmount=noauto -o mountpoint=/boot \ + bpool/BOOT/ubuntu_$UUID + zfs mount bpool/BOOT/ubuntu_$UUID + +With ZFS, it is not normally necessary to use a mount command (either +``mount`` or ``zfs mount``). This situation is an exception because of +``canmount=noauto``. + +3.3 Create datasets:: + + zfs create -o com.ubuntu.zsys:bootfs=no \ + rpool/ROOT/ubuntu_$UUID/srv + zfs create -o com.ubuntu.zsys:bootfs=no -o canmount=off \ + rpool/ROOT/ubuntu_$UUID/usr + zfs create rpool/ROOT/ubuntu_$UUID/usr/local + zfs create -o com.ubuntu.zsys:bootfs=no -o canmount=off \ + rpool/ROOT/ubuntu_$UUID/var + zfs create rpool/ROOT/ubuntu_$UUID/var/games + zfs create rpool/ROOT/ubuntu_$UUID/var/lib + zfs create rpool/ROOT/ubuntu_$UUID/var/lib/AccountServices + zfs create rpool/ROOT/ubuntu_$UUID/var/lib/apt + zfs create rpool/ROOT/ubuntu_$UUID/var/lib/dpkg + zfs create rpool/ROOT/ubuntu_$UUID/var/lib/NetworkManager + zfs create rpool/ROOT/ubuntu_$UUID/var/log + zfs create rpool/ROOT/ubuntu_$UUID/var/mail + zfs create rpool/ROOT/ubuntu_$UUID/var/snap + zfs create rpool/ROOT/ubuntu_$UUID/var/spool + zfs create rpool/ROOT/ubuntu_$UUID/var/www + + zfs create -o canmount=off -o mountpoint=/ \ + rpool/USERDATA + zfs create -o com.ubuntu.zsys:bootfs-datasets=rpool/ROOT/ubuntu_$UUID \ + -o canmount=on -o mountpoint=/root \ + rpool/USERDATA/root_$UUID + +For a mirror or raidz topology, create a dataset for ``/boot/grub``:: + + zfs create -o com.ubuntu.zsys:bootfs=no bpool/grub + +A tmpfs is recommended later, but if you want a separate dataset for +``/tmp``:: + + zfs create -o com.ubuntu.zsys:bootfs=no \ + rpool/ROOT/ubuntu_$UUID/tmp + chmod 1777 /mnt/tmp + +The primary goal of this dataset layout is to separate the OS from user data. +This allows the root filesystem to be rolled back without rolling back user +data. + +If you do nothing extra, ``/tmp`` will be stored as part of the root +filesystem. Alternatively, you can create a separate dataset for +``/tmp``, as shown above. This keeps the ``/tmp`` data out of snapshots +of your root filesystem. It also allows you to set a quota on +``rpool/tmp``, if you want to limit the maximum space used. Otherwise, +you can use a tmpfs (RAM filesystem) later. + +3.4 Install the minimal system:: + + debootstrap focal /mnt + +The ``debootstrap`` command leaves the new system in an unconfigured +state. An alternative to using ``debootstrap`` is to copy the entirety +of a working system into the new ZFS root. + +Step 4: System Configuration +---------------------------- + +4.1 Configure the hostname: + +Replace ``HOSTNAME`` with the desired hostname:: + + echo HOSTNAME > /mnt/etc/hostname + vi /mnt/etc/hosts + +.. code-block:: text + + Add a line: + 127.0.1.1 HOSTNAME + or if the system has a real name in DNS: + 127.0.1.1 FQDN HOSTNAME + +**Hint:** Use ``nano`` if you find ``vi`` confusing. + +4.2 Configure the network interface: + +Find the interface name:: + + ip addr show + +Adjust NAME below to match your interface name:: + + vi /mnt/etc/netplan/01-netcfg.yaml + +.. code-block:: yaml + + network: + version: 2 + ethernets: + NAME: + dhcp4: true + +Customize this file if the system is not a DHCP client. + +4.3 Configure the package sources:: + + vi /mnt/etc/apt/sources.list + +.. code-block:: sourceslist + + deb http://archive.ubuntu.com/ubuntu focal main restricted universe multiverse + deb http://archive.ubuntu.com/ubuntu focal-updates main restricted universe multiverse + deb http://archive.ubuntu.com/ubuntu focal-backports main restricted universe multiverse + deb http://security.ubuntu.com/ubuntu focal-security main restricted universe multiverse + +4.4 Bind the virtual filesystems from the LiveCD environment to the new +system and ``chroot`` into it:: + + mount --rbind /dev /mnt/dev + mount --rbind /proc /mnt/proc + mount --rbind /sys /mnt/sys + chroot /mnt /usr/bin/env DISK=$DISK UUID=$UUID bash --login + +**Note:** This is using ``--rbind``, not ``--bind``. + +4.5 Configure a basic system environment:: + + apt update + + dpkg-reconfigure locales + +Even if you prefer a non-English system language, always ensure that +``en_US.UTF-8`` is available:: + + dpkg-reconfigure tzdata + +If you prefer ``nano`` over ``vi``, install it:: + + apt install --yes nano + +4.6 For LUKS installs only, setup ``/etc/crypttab``:: + + apt install --yes cryptsetup + + echo luks1 UUID=$(blkid -s UUID -o value ${DISK}-part4) none \ + luks,discard,initramfs > /etc/crypttab + +The use of ``initramfs`` is a work-around for `cryptsetup does not support ZFS +`__. + +**Hint:** If you are creating a mirror or raidz topology, repeat the +``/etc/crypttab`` entries for ``luks2``, etc. adjusting for each disk. + +4.7 Create the EFI filesystem: + +Perform these steps for both UEFI and legacy (BIOS) booting:: + + apt install --yes dosfstools + + mkdosfs -F 32 -s 1 -n EFI ${DISK}-part1 + mkdir /boot/efi + echo UUID=$(blkid -s UUID -o value ${DISK}-part1) \ + /boot/efi vfat umask=0022,fmask=0022,dmask=0022 0 1 >> /etc/fstab + mount /boot/efi + +For a mirror or raidz topology, repeat these steps for the additional disks, +using ``/boot/efi2``, ``/boot/efi3``, etc. + +**Note:** The ``-s 1`` for ``mkdosfs`` is only necessary for drives which +present 4 KiB logical sectors (“4Kn” drives) to meet the minimum cluster size +(given the partition size of 512 MiB) for FAT32. It also works fine on drives +which present 512 B sectors. + +4.8 Install GRUB/Linux/ZFS in the chroot environment for the new system: + +For a single-disk install only:: + + mkdir /boot/efi/grub /boot/grub + echo /boot/efi/grub /boot/grub none defaults,bind 0 0 >> /etc/fstab + mount /boot/grub + +**Note:** This puts `/boot/grub` on the EFI System Partition. This allows GRUB +to write to it, which means that `/boot/grub/grubenv` and the `recordfail` +feature works as expected: if the boot fails, the normally hidden GRUB menu +will be shown on the next boot. For a mirror or raidz topology, we do not want +GRUB writing to the EFI System Partition. This is becase we duplicate it at +install without a mechanism to update the copies when the GRUB configuration +changes (e.g. as the kernel is upgraded). Thus, we keep `/boot/grub` on the +boot pool for the mirror or raidz topologies. This preserves correct +mirroring/raidz behavior, at the expense of being able to write to +`/boot/grub/grubenv` and thus the `recordfail` behavior. + +Choose one of the following options: + +4.8a Install GRUB/Linux/ZFS for legacy (BIOS) booting:: + + apt install --yes grub-pc linux-image-generic zfs-initramfs zsys + +Select (using the space bar) all of the disks (not partitions) in your pool. + +4.8b Install GRUB/Linux/ZFS for UEFI booting:: + + apt install --yes \ + grub-efi-amd64 grub-efi-amd64-signed linux-image-generic shim-signed \ + zfs-initramfs zsys + +**Note:** For a mirror or raidz topology, this step only installs GRUB on the +first disk. The other disk(s) will be handled later. + +4.9 (Optional): Remove os-prober:: + + dpkg --purge os-prober + +This avoids error messages from `update-grub`. `os-prober` is only necessary +in dual-boot configurations. + +4.10 Set a root password:: + + passwd + +4.11 Configure swap: + +Choose one of the following options if you want swap: + +4.11a For an unencrypted single-disk install:: + + mkswap -f ${DISK}-part2 + echo UUID=$(blkid -s UUID -o value ${DISK}-part2) \ + none swap discard 0 0 >> /etc/fstab + swapon -a + +4.11b For an unencrypted mirror or raidz topology:: + + apt install --yes mdadm + # Adjust the level (ZFS raidz = MD raid5, raidz2 = raid6) and raid-devices + # if necessary and specify the actual devices. + mdadm --create /dev/md0 --metadata=1.2 --level=mirror --raid-devices=2 \ + ${DISK1}-part2 ${DISK2}-part2 + mkswap -f /dev/md0 + echo UUID=$(blkid -s UUID -o value /dev/md0) \ + none swap discard 0 0 >> /etc/fstab + swapon -a + +4.11c For an encrypted (LUKS or ZFS native encryption) single-disk install:: + + apt install --yes cryptsetup + echo swap ${DISK}-part2 /dev/urandom \ + swap,cipher=aes-xts-plain64:sha256,size=512 >> /etc/crypttab + echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab + +4.11d For an encrypted (LUKS or ZFS native encryption) mirror or raidz +topology:: + + apt install --yes cryptsetup mdadm + # Adjust the level (ZFS raidz = MD raid5, raidz2 = raid6) and raid-devices + # if necessary and specify the actual devices. + mdadm --create /dev/md0 --metadata=1.2 --level=mirror --raid-devices=2 \ + ${DISK1}-part2 ${DISK2}-part2 + echo swap /dev/md0 /dev/urandom \ + swap,cipher=aes-xts-plain64:sha256,size=512 >> /etc/crypttab + echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab + +4.12 Optional (but recommended): Mount a tmpfs to ``/tmp`` + +If you chose to create a ``/tmp`` dataset above, skip this step, as they +are mutually exclusive choices. Otherwise, you can put ``/tmp`` on a +tmpfs (RAM filesystem) by enabling the ``tmp.mount`` unit. + +:: + + cp /usr/share/systemd/tmp.mount /etc/systemd/system/ + systemctl enable tmp.mount + +4.13 Setup system groups:: + + addgroup --system lpadmin + addgroup --system lxd + addgroup --system sambashare + +Step 5: GRUB Installation +------------------------- + +5.1 Verify that the ZFS boot filesystem is recognized:: + + grub-probe /boot + +5.2 Refresh the initrd files:: + + update-initramfs -c -k all + +**Note:** When using LUKS, this will print “WARNING could not determine +root device from /etc/fstab”. This is because `cryptsetup does not +support ZFS +`__. + +5.3 Disable memory zeroing:: + + vi /etc/default/grub + # Add init_on_alloc=0 to: GRUB_CMDLINE_LINUX_DEFAULT + # Save and quit. + +This is to address `performance regressions +`__. + +5.4 Optional (but highly recommended): Make debugging GRUB easier:: + + vi /etc/default/grub + # Comment out: GRUB_TIMEOUT_STYLE=hidden + # Set: GRUB_TIMEOUT=5 + # Below GRUB_TIMEOUT, add: GRUB_RECORDFAIL_TIMEOUT=5 + # Remove quiet and splash from: GRUB_CMDLINE_LINUX_DEFAULT + # Uncomment: GRUB_TERMINAL=console + # Save and quit. + +Later, once the system has rebooted twice and you are sure everything is +working, you can undo these changes, if desired. + +5.5 Update the boot configuration:: + + update-grub + +**Note:** Ignore errors from ``osprober``, if present. + +5.6 Install the boot loader: + +5.6a For legacy (BIOS) booting, install GRUB to the MBR:: + + grub-install $DISK + +Note that you are installing GRUB to the whole disk, not a partition. + +If you are creating a mirror or raidz topology, repeat the +``grub-install`` command for each disk in the pool. + +5.6b For UEFI booting, install GRUB:: + + grub-install --target=x86_64-efi --efi-directory=/boot/efi \ + --bootloader-id=ubuntu --recheck --no-floppy + +For a mirror or raidz topology, run this for the additional disk(s), +incrementing the “2” to “3” and so on for both ``/boot/efi2`` and +``ubuntu-2``:: + + cp -a /boot/efi/EFI /boot/efi2 + grub-install --target=x86_64-efi --efi-directory=/boot/efi2 \ + --bootloader-id=ubuntu-2 --recheck --no-floppy + +5.7 Fix filesystem mount ordering: + +We need to activate ``zfs-mount-generator``. This makes systemd aware of +the separate mountpoints, which is important for things like +``/var/log`` and ``/var/tmp``. In turn, ``rsyslog.service`` depends on +``var-log.mount`` by way of ``local-fs.target`` and services using the +``PrivateTmp`` feature of systemd automatically use +``After=var-tmp.mount``. + +:: + + mkdir /etc/zfs/zfs-list.cache + touch /etc/zfs/zfs-list.cache/bpool + touch /etc/zfs/zfs-list.cache/rpool + ln -s /usr/lib/zfs-linux/zed.d/history_event-zfs-list-cacher.sh /etc/zfs/zed.d + zed -F & + +Verify that ``zed`` updated the cache by making sure these are not empty:: + + cat /etc/zfs/zfs-list.cache/bpool + cat /etc/zfs/zfs-list.cache/rpool + +If either is empty, force a cache update and check again:: + + zfs set canmount=noauto bpool/BOOT/ubuntu_$UUID + zfs set canmount=noauto rpool/ROOT/ubuntu_$UUID + +Stop ``zed``:: + + fg + Press Ctrl-C. + +Fix the paths to eliminate ``/mnt``:: + + sed -Ei "s|/mnt/?|/|" /etc/zfs/zfs-list.cache/* + +Step 6: First Boot +------------------ + +6.1 (Optional): Install SSH:: + + apt install --yes openssh-server + +If you want to login as root via SSH, set ``PermitRootLogin yes`` in +``/etc/ssh/sshd_config``. For security, undo this as soon as possible (i.e. +once you have your regular user account setup). + +6.2 Exit from the ``chroot`` environment back to the LiveCD environment:: + + exit + +6.3 Run these commands in the LiveCD environment to unmount all +filesystems:: + + mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {} + zpool export -a + +6.4 Reboot:: + + reboot + +Wait for the newly installed system to boot normally. Login as root. + +6.5 Create a user account: + +Replace ``username`` with your desired username:: + + UUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null | + tr -dc 'a-z0-9' | cut -c-6) + ROOT_DS=$(zfs list -o name | awk '/ROOT\/ubuntu_/{print $1;exit}') + zfs create -o com.ubuntu.zsys:bootfs-datasets=$ROOT_DS \ + -o canmount=on -o mountpoint=/home/username \ + rpool/USERDATA/username_$UUID + adduser username + + cp -a /etc/skel/. /home/username + chown -R username:username /home/username + usermod -a -G adm,cdrom,dip,lpadmin,lxd,plugdev,sambashare,sudo username + +Step 7: Full Software Installation +---------------------------------- + +7.1 Upgrade the minimal system:: + + apt dist-upgrade --yes + +7.2 Install a regular set of software: + +Choose one of the following options: + +7.2a Install a command-line environment only:: + + apt install --yes ubuntu-standard + +7.2b Install a full GUI environment:: + + apt install --yes ubuntu-desktop + vi /etc/gdm3/custom.conf + # In the [daemon] section, add: InitialSetupEnable=false + +**Hint**: If you are installing a full GUI environment, you will likely +want to manage your network with NetworkManager:: + + rm /mnt/etc/netplan/01-netcfg.yaml + vi /etc/netplan/01-network-manager-all.yaml + +.. code-block:: yaml + + network: + version: 2 + renderer: NetworkManager + +7.3 Optional: Disable log compression: + +As ``/var/log`` is already compressed by ZFS, logrotate’s compression is +going to burn CPU and disk I/O for (in most cases) very little gain. +Also, if you are making snapshots of ``/var/log``, logrotate’s +compression will actually waste space, as the uncompressed data will +live on in the snapshot. You can edit the files in ``/etc/logrotate.d`` +by hand to comment out ``compress``, or use this loop (copy-and-paste +highly recommended):: + + for file in /etc/logrotate.d/* ; do + if grep -Eq "(^|[^#y])compress" "$file" ; then + sed -i -r "s/(^|[^#y])(compress)/\1#\2/" "$file" + fi + done + +7.4 Reboot:: + + reboot + +Step 8: Final Cleanup +--------------------- + +8.1 Wait for the system to boot normally. Login using the account you +created. Ensure the system (including networking) works normally. + +8.2 Optional: Disable the root password:: + + sudo usermod -p '*' root + +8.3 Optional: Re-enable the graphical boot process: + +If you prefer the graphical boot process, you can re-enable it now. If +you are using LUKS, it makes the prompt look nicer. + +:: + + sudo vi /etc/default/grub + # Uncomment: GRUB_TIMEOUT_STYLE=hidden + # Add quiet and splash to: GRUB_CMDLINE_LINUX_DEFAULT + # Comment out: GRUB_TERMINAL=console + # Save and quit. + + sudo update-grub + +**Note:** Ignore errors from ``osprober``, if present. + +8.4 Optional: For LUKS installs only, backup the LUKS header:: + + sudo cryptsetup luksHeaderBackup /dev/disk/by-id/scsi-SATA_disk1-part4 \ + --header-backup-file luks1-header.dat + +Store that backup somewhere safe (e.g. cloud storage). It is protected +by your LUKS passphrase, but you may wish to use additional encryption. + +**Hint:** If you created a mirror or raidz topology, repeat this for +each LUKS volume (``luks2``, etc.). + +Troubleshooting +--------------- + +Rescuing using a Live CD +~~~~~~~~~~~~~~~~~~~~~~~~ + +Go through `Step 1: Prepare The Install +Environment <#step-1-prepare-the-install-environment>`__. + +For LUKS, first unlock the disk(s):: + + cryptsetup luksOpen /dev/disk/by-id/scsi-SATA_disk1-part4 luks1 + # Repeat for additional disks, if this is a mirror or raidz topology. + +Mount everything correctly:: + + zpool export -a + zpool import -N -R /mnt rpool + zpool import -N -R /mnt bpool + # Replace “UUID” as appropriate; use zfs list to find it: + zfs mount rpool/ROOT/ubuntu_UUID + zfs mount -a + +If needed, you can chroot into your installed environment:: + + mount --rbind /dev /mnt/dev + mount --rbind /proc /mnt/proc + mount --rbind /sys /mnt/sys + chroot /mnt /bin/bash --login + mount /boot + mount -a + +Do whatever you need to do to fix your system. + +When done, cleanup:: + + exit + mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {} + zpool export -a + reboot + +MPT2SAS +~~~~~~~ + +Most problem reports for this tutorial involve ``mpt2sas`` hardware that +does slow asynchronous drive initialization, like some IBM M1015 or +OEM-branded cards that have been flashed to the reference LSI firmware. + +The basic problem is that disks on these controllers are not visible to +the Linux kernel until after the regular system is started, and ZoL does +not hotplug pool members. See +`https://github.com/zfsonlinux/zfs/issues/330 `__. + +Most LSI cards are perfectly compatible with ZoL. If your card has this +glitch, try setting ``ZFS_INITRD_PRE_MOUNTROOT_SLEEP=X`` in +``/etc/default/zfs``. The system will wait ``X`` seconds for all drives to +appear before importing the pool. + +Areca +~~~~~ + +Systems that require the ``arcsas`` blob driver should add it to the +``/etc/initramfs-tools/modules`` file and run +``update-initramfs -c -k all``. + +Upgrade or downgrade the Areca driver if something like +``RIP: 0010:[] [] native_read_tsc+0x6/0x20`` +appears anywhere in kernel log. ZoL is unstable on systems that emit +this error message. + +VMware +~~~~~~ + +- Set ``disk.EnableUUID = "TRUE"`` in the vmx file or vsphere + configuration. Doing this ensures that ``/dev/disk`` aliases are + created in the guest. + +QEMU/KVM/XEN +~~~~~~~~~~~~ + +Set a unique serial number on each virtual disk using libvirt or qemu +(e.g. ``-drive if=none,id=disk1,file=disk1.qcow2,serial=1234567890``). + +To be able to use UEFI in guests (instead of only BIOS booting), run +this on the host:: + + sudo apt install ovmf + sudo vi /etc/libvirt/qemu.conf + +Uncomment these lines: + +.. code-block:: text + + nvram = [ + "/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd", + "/usr/share/OVMF/OVMF_CODE.secboot.fd:/usr/share/OVMF/OVMF_VARS.fd", + "/usr/share/AAVMF/AAVMF_CODE.fd:/usr/share/AAVMF/AAVMF_VARS.fd", + "/usr/share/AAVMF/AAVMF32_CODE.fd:/usr/share/AAVMF/AAVMF32_VARS.fd", + "/usr/share/OVMF/OVMF_CODE.ms.fd:/usr/share/OVMF/OVMF_VARS.ms.fd" + ] + +:: + + sudo systemctl restart libvirtd.service