diff --git a/docs/Getting Started/Ubuntu/Ubuntu 20.04 Root on ZFS for Raspberry Pi.rst b/docs/Getting Started/Ubuntu/Ubuntu 20.04 Root on ZFS for Raspberry Pi.rst index a4d6a59..0c2a497 100644 --- a/docs/Getting Started/Ubuntu/Ubuntu 20.04 Root on ZFS for Raspberry Pi.rst +++ b/docs/Getting Started/Ubuntu/Ubuntu 20.04 Root on ZFS for Raspberry Pi.rst @@ -9,6 +9,12 @@ Ubuntu 20.04 Root on ZFS for Raspberry Pi Overview -------- +Newer release available +~~~~~~~~~~~~~~~~~~~~~~~ + +- See :doc:`Ubuntu 22.04 Root on ZFS for Raspberry Pi <./Ubuntu 22.04 Root on ZFS for Raspberry Pi>` for new + installs. + Caution ~~~~~~~ diff --git a/docs/Getting Started/Ubuntu/Ubuntu 22.04 Root on ZFS for Raspberry Pi.rst b/docs/Getting Started/Ubuntu/Ubuntu 22.04 Root on ZFS for Raspberry Pi.rst new file mode 100644 index 0000000..8fa8076 --- /dev/null +++ b/docs/Getting Started/Ubuntu/Ubuntu 22.04 Root on ZFS for Raspberry Pi.rst @@ -0,0 +1,850 @@ +.. highlight:: sh + +Ubuntu 22.04 Root on ZFS for Raspberry Pi +========================================= + +.. contents:: Table of Contents + :local: + +Overview +-------- + +.. note:: + These are beta instructions. The author still needs to test them. + Additionally, it may be possible to use U-Boot now, which would eliminate + some of the customizations. + +Caution +~~~~~~~ + +- This HOWTO uses a whole physical disk. +- Backup your data. Any existing data will be lost. + +System Requirements +~~~~~~~~~~~~~~~~~~~ + +- A Raspberry Pi 4 B. (If you are looking to install on a regular PC, see + :doc:`Ubuntu 22.04 Root on ZFS`.) +- `Ubuntu Server 22.04 (“Jammy”) for Raspberry Pi 4 + `__ +- A microSD card or USB disk. For microSD card recommendations, see Jeff + Geerling's `performance comparison + `__. + When using a USB enclosure, `ensure it supports UASP + `__. +- An Ubuntu system (with the ability to write to the microSD card or USB disk) + other than the target Raspberry Pi. + +4 GiB of memory is recommended. Do not use deduplication, as it needs `massive +amounts of RAM `__. +Enabling deduplication is a permanent change that cannot be easily reverted. + +A Raspberry Pi 3 B/B+ would probably work (as the Pi 3 is 64-bit, though it +has less RAM), but has not been tested. Please report your results (good or +bad) using the issue link below. + +Support +~~~~~~~ + +If you need help, reach out to the community using the :ref:`mailing_lists` or IRC at +`#zfsonlinux `__ on `Libera Chat +`__. If you have a bug report or feature request +related to this HOWTO, please `file a new issue and mention @rlaager +`__. + +Contributing +~~~~~~~~~~~~ + +#. Fork and clone: https://github.com/openzfs/openzfs-docs + +#. Install the tools:: + + sudo apt install python3-pip + + pip3 install -r docs/requirements.txt + + # Add ~/.local/bin to your $PATH, e.g. by adding this to ~/.bashrc: + PATH=$HOME/.local/bin:$PATH + +#. Make your changes. + +#. Test:: + + cd docs + make html + sensible-browser _build/html/index.html + +#. ``git commit --signoff`` to a branch, ``git push``, and create a pull + request. Mention @rlaager. + +Encryption +~~~~~~~~~~ + +**WARNING:** Encryption has not yet been tested on the Raspberry Pi. + +This guide supports three different encryption options: unencrypted, ZFS +native encryption, and LUKS. With any option, all ZFS features are fully +available. + +Unencrypted does not encrypt anything, of course. With no encryption +happening, this option naturally has the best performance. + +ZFS native encryption encrypts the data and most metadata in the root +pool. It does not encrypt dataset or snapshot names or properties. The +boot pool is not encrypted at all, but it only contains the bootloader, +kernel, and initrd. (Unless you put a password in ``/etc/fstab``, the +initrd is unlikely to contain sensitive data.) The system cannot boot +without the passphrase being entered at the console. Performance is +good. As the encryption happens in ZFS, even if multiple disks (mirror +or raidz topologies) are used, the data only has to be encrypted once. + +LUKS encrypts almost everything. The only unencrypted data is the bootloader, +kernel, and initrd. The system cannot boot without the passphrase being +entered at the console. Performance is good, but LUKS sits underneath ZFS, so +if multiple disks (mirror or raidz topologies) are used, the data has to be +encrypted once per disk. + +USB Disks +~~~~~~~~~ + +The Raspberry Pi 4 runs much faster using a USB Solid State Drive (SSD) than +a microSD card. These instructions can also be used to install Ubuntu on a +USB-connected SSD or other USB disk. USB disks have three requirements that +do not apply to microSD cards: + +#. The Raspberry Pi's Bootloader EEPROM must be dated 2020-09-03 or later. + + To check the bootloader version, power up the Raspberry Pi without an SD + card inserted or a USB boot device attached; the date will be on the + ``bootloader`` line. (If you do not see the ``bootloader`` line, the + bootloader is too old.) Alternatively, run ``sudo rpi-eeprom-update`` + on an existing OS on the Raspberry Pi (which on Ubuntu requires + ``apt install rpi-eeprom``). + + If needed, the bootloader can be updated from an existing OS on the + Raspberry Pi using ``rpi-eeprom-update -a`` and rebooting. + For other options, see `Updating the Bootloader + `_. + +#. The Raspberry Pi must configured for USB boot. The bootloader will show a + ``boot`` line; if ``order`` includes ``4``, USB boot is enabled. + + If not already enabled, it can be enabled from an existing OS on the + Raspberry Pi using ``rpi-eeprom-config -e``: set ``BOOT_ORDER=0xf41`` + and reboot to apply the change. On subsequent reboots, USB boot will be + enabled. + + Otherwise, it can be enabled without an existing OS as follows: + + - Download the `Raspberry Pi Imager Utility + `_. + - Flash the ``USB Boot`` image to a microSD card. The ``USB Boot`` image is + listed under ``Bootload`` in the ``Misc utility images`` folder. + - Boot the Raspberry Pi from the microSD card. USB Boot should be enabled + automatically. + +#. U-Boot on Ubuntu 20.04 does not seem to support the Raspberry Pi USB. + `Ubuntu 20.10 may work + `_. As a + work-around, the Raspberry Pi bootloader is configured to directly boot + Linux. For this to work, the Linux kernel must not be compressed. These + instructions decompress the kernel and add a script to + ``/etc/kernel/postinst.d`` to handle kernel upgrades. + +Step 1: Disk Formatting +----------------------- + +The commands in this step are run on the system other than the Raspberry Pi. + +This guide has you go to some extra work so that the stock ext4 partition can +be deleted. + +#. Download and unpack the official image:: + + curl -O https://cdimage.ubuntu.com/releases/22.04/release/ubuntu-22.04-preinstalled-server-arm64+raspi.img.xz + xz -d ubuntu-22.04-preinstalled-server-arm64+raspi.img.xz + + # or combine them to decompress as you download: + curl https://cdimage.ubuntu.com/releases/22.04/release/ubuntu-22.04-preinstalled-server-arm64+raspi.img.xz | \ + xz -d > ubuntu-22.04-preinstalled-server-arm64+raspi.img + +#. Dump the partition table for the image:: + + sfdisk -d ubuntu-22.04-preinstalled-server-arm64+raspi.img + + That will output this:: + + label: dos + label-id: 0x638274e3 + device: ubuntu-22.04-preinstalled-server-arm64+raspi.img + unit: sectors + + .img1 : start= 2048, size= 524288, type=c, bootable + .img2 : start= 526336, size= 7129360, type=83 + + The important numbers are 524288 and 7129360. Store those in variables:: + + BOOT=524288 + ROOT=7129360 + +#. Create a partition script:: + + cat > partitions << EOF + label: dos + unit: sectors + + 1 : start= 2048, size=$BOOT, type=c, bootable + 2 : start=$((2048+BOOT)), size=$ROOT, type=83 + 3 : start=$((2048+BOOT+ROOT)), size=$ROOT, type=83 + EOF + +#. Connect the disk: + + Connect the disk to a machine other than the target Raspberry Pi. If any + filesystems are automatically mounted (e.g. by GNOME) unmount them. + Determine the device name. For SD, the device name is almost certainly + ``/dev/mmcblk0``. For USB SSDs, the device name is ``/dev/sdX``, where + ``X`` is a lowercase letter. ``lsblk`` can help determine the device name. + Set the ``DISK`` environment variable to the device name:: + + DISK=/dev/mmcblk0 # microSD card + DISK=/dev/sdX # USB disk + + Because partitions are named differently for ``/dev/mmcblk0`` and ``/dev/sdX`` + devices, set a second variable used when working with partitions:: + + export DISKP=${DISK}p # microSD card + export DISKP=${DISK} # USB disk ($DISKP == $DISK for /dev/sdX devices) + + **Hint**: microSD cards connected using a USB reader also have ``/dev/sdX`` + names. + + **WARNING**: The following steps destroy the existing data on the disk. Ensure + ``DISK`` and ``DISKP`` are correct before proceeding. + +#. Ensure swap partitions are not in use:: + + swapon -v + # If a partition is in use from the disk, disable it: + sudo swapoff THAT_PARTITION + +#. Clear old ZFS labels:: + + sudo zpool labelclear -f ${DISK} + + If a ZFS label still exists from a previous system/attempt, expanding the + pool will result in an unbootable system. + + **Hint:** If you do not already have the ZFS utilities installed, you can + install them with: ``sudo apt install zfsutils-linux`` Alternatively, you + can zero the entire disk with: + ``sudo dd if=/dev/zero of=${DISK} bs=1M status=progress`` + +#. Delete existing partitions:: + + echo "label: dos" | sudo sfdisk ${DISK} + sudo partprobe + ls ${DISKP}* + + Make sure there are no partitions, just the file for the disk itself. This + step is not strictly necessary; it exists to catch problems. + +#. Create the partitions:: + + sudo sfdisk $DISK < partitions + +#. Loopback mount the image:: + + IMG=$(sudo losetup -fP --show \ + ubuntu-22.04-preinstalled-server-arm64+raspi.img) + +#. Copy the bootloader data:: + + sudo dd if=${IMG}p1 of=${DISKP}1 bs=1M + +#. Clear old label(s) from partition 2:: + + sudo wipefs -a ${DISKP}2 + + If a filesystem with the ``writable`` label from the Ubuntu image is still + present in partition 2, the system will not boot initially. + +#. Copy the root filesystem data:: + + # NOTE: the destination is p3, not p2. + sudo dd if=${IMG}p2 of=${DISKP}3 bs=1M status=progress conv=fsync + +#. Unmount the image:: + + sudo losetup -d $IMG + +#. If setting up a USB disk: + + Decompress the kernel:: + + sudo -sE + + MNT=$(mktemp -d /mnt/XXXXXXXX) + mkdir -p $MNT/boot $MNT/root + mount ${DISKP}1 $MNT/boot + mount ${DISKP}3 $MNT/root + + zcat -qf $MNT/boot/vmlinuz >$MNT/boot/vmlinux + + Modify boot config:: + + cat >> $MNT/boot/usercfg.txt << EOF + kernel=vmlinux + initramfs initrd.img followkernel + boot_delay + EOF + + Create a script to automatically decompress the kernel after an upgrade:: + + cat >$MNT/root/etc/kernel/postinst.d/zz-decompress-kernel << 'EOF' + #!/bin/sh + + set -eu + + echo "Updating decompressed kernel..." + [ -e /boot/firmware/vmlinux ] && \ + cp /boot/firmware/vmlinux /boot/firmware/vmlinux.bak + vmlinuxtmp=$(mktemp /boot/firmware/vmlinux.XXXXXXXX) + zcat -qf /boot/vmlinuz > "$vmlinuxtmp" + mv "$vmlinuxtmp" /boot/firmware/vmlinux + EOF + + chmod +x $MNT/root/etc/kernel/postinst.d/zz-decompress-kernel + + Cleanup:: + + umount $MNT/* + rm -rf $MNT + exit + +#. Boot the Raspberry Pi. + + Move the SD/USB disk to the Raspberry Pi. Boot it and login (e.g. via SSH) + with ``ubuntu`` as the username and password. If you are using SSH, note + that it takes a little bit for cloud-init to enable password logins on the + first boot. Set a new password when prompted and login again using that + password. If you have your local SSH configured to use ``ControlPersist``, + you will have to kill the existing SSH process before logging in the second + time. + +Step 2: Setup ZFS +----------------- + +#. Become root:: + + sudo -i + +#. Set the DISK and DISKP variables again:: + + DISK=/dev/mmcblk0 # microSD card + DISKP=${DISK}p # microSD card + + DISK=/dev/sdX # USB disk + DISKP=${DISK} # USB disk + + **WARNING:** Device names can change when moving a device to a different + computer or switching the microSD card from a USB reader to a built-in + slot. Double check the device name before continuing. + +#. Install ZFS:: + + apt update + + apt install pv zfs-initramfs + + **Note:** Since this is the first boot, you may get ``Waiting for cache + lock`` because ``unattended-upgrades`` is running in the background. + Wait for it to finish. + +#. Create the root pool: + + Choose one of the following options: + + - Unencrypted:: + + zpool create \ + -o ashift=12 \ + -O acltype=posixacl -O canmount=off -O compression=lz4 \ + -O dnodesize=auto -O normalization=formD -O relatime=on \ + -O xattr=sa -O mountpoint=/ -R /mnt \ + rpool ${DISKP}2 + + **WARNING:** Encryption has not yet been tested on the Raspberry Pi. + + - ZFS native encryption:: + + zpool create \ + -o ashift=12 \ + -O encryption=aes-256-gcm \ + -O keylocation=prompt -O keyformat=passphrase \ + -O acltype=posixacl -O canmount=off -O compression=lz4 \ + -O dnodesize=auto -O normalization=formD -O relatime=on \ + -O xattr=sa -O mountpoint=/ -R /mnt \ + rpool ${DISKP}2 + + - LUKS:: + + cryptsetup luksFormat -c aes-xts-plain64 -s 512 -h sha256 ${DISKP}2 + cryptsetup luksOpen ${DISK}-part4 luks1 + zpool create \ + -o ashift=12 \ + -O acltype=posixacl -O canmount=off -O compression=lz4 \ + -O dnodesize=auto -O normalization=formD -O relatime=on \ + -O xattr=sa -O mountpoint=/ -R /mnt \ + rpool /dev/mapper/luks1 + + **Notes:** + + - The use of ``ashift=12`` is recommended here because many drives + today have 4 KiB (or larger) physical sectors, even though they + present 512 B logical sectors. Also, a future replacement drive may + have 4 KiB physical sectors (in which case ``ashift=12`` is desirable) + or 4 KiB logical sectors (in which case ``ashift=12`` is required). + - Setting ``-O acltype=posixacl`` enables POSIX ACLs globally. If you + do not want this, remove that option, but later add + ``-o acltype=posixacl`` (note: lowercase “o”) to the ``zfs create`` + for ``/var/log``, as `journald requires ACLs + `__ + Also, `disabling ACLs apparently breaks umask handling with NFSv4 + `__. + - Setting ``normalization=formD`` eliminates some corner cases relating + to UTF-8 filename normalization. It also implies ``utf8only=on``, + which means that only UTF-8 filenames are allowed. If you care to + support non-UTF-8 filenames, do not use this option. For a discussion + of why requiring UTF-8 filenames may be a bad idea, see `The problems + with enforced UTF-8 only filenames + `__. + - ``recordsize`` is unset (leaving it at the default of 128 KiB). If you + want to tune it (e.g. ``-o recordsize=1M``), see `these + `__ `various + `__ `blog + `__ + `posts + `__. + - Setting ``relatime=on`` is a middle ground between classic POSIX + ``atime`` behavior (with its significant performance impact) and + ``atime=off`` (which provides the best performance by completely + disabling atime updates). Since Linux 2.6.30, ``relatime`` has been + the default for other filesystems. See `RedHat’s documentation + `__ + for further information. + - Setting ``xattr=sa`` `vastly improves the performance of extended + attributes + `__. + Inside ZFS, extended attributes are used to implement POSIX ACLs. + Extended attributes can also be used by user-space applications. + `They are used by some desktop GUI applications. + `__ + `They can be used by Samba to store Windows ACLs and DOS attributes; + they are required for a Samba Active Directory domain controller. + `__ + Note that ``xattr=sa`` is `Linux-specific + `__. If you move your + ``xattr=sa`` pool to another OpenZFS implementation besides ZFS-on-Linux, + extended attributes will not be readable (though your data will be). If + portability of extended attributes is important to you, omit the + ``-O xattr=sa`` above. Even if you do not want ``xattr=sa`` for the whole + pool, it is probably fine to use it for ``/var/log``. + - Make sure to include the ``-part4`` portion of the drive path. If you + forget that, you are specifying the whole disk, which ZFS will then + re-partition, and you will lose the bootloader partition(s). + - ZFS native encryption defaults to ``aes-256-ccm``, but `the default has + changed upstream + `__ + to ``aes-256-gcm``. `AES-GCM seems to be generally preferred over AES-CCM + `__, + `is faster now + `__, + and `will be even faster in the future + `__. + - For LUKS, the key size chosen is 512 bits. However, XTS mode requires two + keys, so the LUKS key is split in half. Thus, ``-s 512`` means AES-256. + - Your passphrase will likely be the weakest link. Choose wisely. See + `section 5 of the cryptsetup FAQ + `__ + for guidance. + +Step 3: System Installation +--------------------------- + +#. Create a filesystem dataset to act as a container:: + + zfs create -o canmount=off -o mountpoint=none rpool/ROOT + +#. Create a filesystem dataset for the root filesystem:: + + UUID=$(dd if=/dev/urandom bs=1 count=100 2>/dev/null | + tr -dc 'a-z0-9' | cut -c-6) + + zfs create -o canmount=noauto -o mountpoint=/ \ + -o com.ubuntu.zsys:bootfs=yes \ + -o com.ubuntu.zsys:last-used=$(date +%s) rpool/ROOT/ubuntu_$UUID + zfs mount rpool/ROOT/ubuntu_$UUID + + With ZFS, it is not normally necessary to use a mount command (either + ``mount`` or ``zfs mount``). This situation is an exception because of + ``canmount=noauto``. + +#. Create datasets:: + + zfs create -o com.ubuntu.zsys:bootfs=no \ + rpool/ROOT/ubuntu_$UUID/srv + zfs create -o com.ubuntu.zsys:bootfs=no -o canmount=off \ + rpool/ROOT/ubuntu_$UUID/usr + zfs create rpool/ROOT/ubuntu_$UUID/usr/local + zfs create -o com.ubuntu.zsys:bootfs=no -o canmount=off \ + rpool/ROOT/ubuntu_$UUID/var + zfs create rpool/ROOT/ubuntu_$UUID/var/games + zfs create rpool/ROOT/ubuntu_$UUID/var/lib + zfs create rpool/ROOT/ubuntu_$UUID/var/lib/AccountsService + zfs create rpool/ROOT/ubuntu_$UUID/var/lib/apt + zfs create rpool/ROOT/ubuntu_$UUID/var/lib/dpkg + zfs create rpool/ROOT/ubuntu_$UUID/var/lib/NetworkManager + zfs create rpool/ROOT/ubuntu_$UUID/var/log + zfs create rpool/ROOT/ubuntu_$UUID/var/mail + zfs create rpool/ROOT/ubuntu_$UUID/var/snap + zfs create rpool/ROOT/ubuntu_$UUID/var/spool + zfs create rpool/ROOT/ubuntu_$UUID/var/www + + zfs create -o canmount=off -o mountpoint=/ \ + rpool/USERDATA + zfs create -o com.ubuntu.zsys:bootfs-datasets=rpool/ROOT/ubuntu_$UUID \ + -o canmount=on -o mountpoint=/root \ + rpool/USERDATA/root_$UUID + + If you want a separate dataset for ``/tmp``:: + + zfs create -o com.ubuntu.zsys:bootfs=no \ + rpool/ROOT/ubuntu_$UUID/tmp + chmod 1777 /mnt/tmp + + The primary goal of this dataset layout is to separate the OS from user + data. This allows the root filesystem to be rolled back without rolling + back user data. + + If you do nothing extra, ``/tmp`` will be stored as part of the root + filesystem. Alternatively, you can create a separate dataset for ``/tmp``, + as shown above. This keeps the ``/tmp`` data out of snapshots of your root + filesystem. It also allows you to set a quota on ``rpool/tmp``, if you want + to limit the maximum space used. Otherwise, you can use a tmpfs (RAM + filesystem) later. + +#. Optional: Ignore synchronous requests: + + microSD cards are relatively slow. If you want to increase performance + (especially when installing packages) at the cost of some safety, you can + disable flushing of synchronous requests (e.g. ``fsync()``, ``O_[D]SYNC``): + + Choose one of the following options: + + - For the root filesystem, but not user data:: + + zfs set sync=disabled rpool/ROOT + + - For everything:: + + zfs set sync=disabled rpool + + ZFS is transactional, so it will still be crash consistent. However, you + should leave ``sync`` at its default of ``standard`` if this system needs + to guarantee persistence (e.g. if it is a database or NFS server). + +#. Copy the system into the ZFS filesystems:: + + (cd /; tar -cf - --one-file-system --warning=no-file-ignored .) | \ + pv -p -bs $(du -sxm --apparent-size / | cut -f1)m | \ + (cd /mnt ; tar -x) + +Step 4: System Configuration +---------------------------- + +#. Configure the hostname: + + Replace ``HOSTNAME`` with the desired hostname:: + + hostname HOSTNAME + hostname > /mnt/etc/hostname + vi /mnt/etc/hosts + + .. code-block:: text + + Add a line: + 127.0.1.1 HOSTNAME + or if the system has a real name in DNS: + 127.0.1.1 FQDN HOSTNAME + + **Hint:** Use ``nano`` if you find ``vi`` confusing. + +#. Stop ``zed``:: + + systemctl stop zed + +#. Bind the virtual filesystems from the running environment to the new + ZFS environment and ``chroot`` into it:: + + mount --make-private --rbind /boot/firmware /mnt/boot/firmware + mount --make-private --rbind /dev /mnt/dev + mount --make-private --rbind /proc /mnt/proc + mount --make-private --rbind /run /mnt/run + mount --make-private --rbind /sys /mnt/sys + chroot /mnt /usr/bin/env DISK=$DISK UUID=$UUID bash --login + +#. Configure a basic system environment:: + + apt update + + Even if you prefer a non-English system language, always ensure that + ``en_US.UTF-8`` is available:: + + dpkg-reconfigure locales + dpkg-reconfigure tzdata + +#. For LUKS installs only, setup ``/etc/crypttab``:: + + # cryptsetup is already installed, but this marks it as manually + # installed so it is not automatically removed. + apt install --yes cryptsetup + + echo luks1 UUID=$(blkid -s UUID -o value ${DISK}-part4) none \ + luks,discard,initramfs > /etc/crypttab + + The use of ``initramfs`` is a work-around for `cryptsetup does not support + ZFS `__. + +#. Optional: Mount a tmpfs to ``/tmp`` + + If you chose to create a ``/tmp`` dataset above, skip this step, as they + are mutually exclusive choices. Otherwise, you can put ``/tmp`` on a + tmpfs (RAM filesystem) by enabling the ``tmp.mount`` unit. + + :: + + cp /usr/share/systemd/tmp.mount /etc/systemd/system/ + systemctl enable tmp.mount + +#. Setup system groups:: + + addgroup --system lpadmin + addgroup --system sambashare + +#. Fix filesystem mount ordering: + + We need to activate ``zfs-mount-generator``. This makes systemd aware of + the separate mountpoints, which is important for things like ``/var/log`` + and ``/var/tmp``. In turn, ``rsyslog.service`` depends on ``var-log.mount`` + by way of ``local-fs.target`` and services using the ``PrivateTmp`` feature + of systemd automatically use ``After=var-tmp.mount``. + + :: + + mkdir /etc/zfs/zfs-list.cache + touch /etc/zfs/zfs-list.cache/rpool + zed -F & + + Force a cache update:: + + zfs set canmount=noauto rpool/ROOT/ubuntu_$UUID + + Verify that ``zed`` updated the cache by making sure this is not empty, + which will take a few seconds:: + + cat /etc/zfs/zfs-list.cache/rpool + + Stop ``zed``:: + + fg + Press Ctrl-C. + + Fix the paths to eliminate ``/mnt``:: + + sed -Ei "s|/mnt/?|/|" /etc/zfs/zfs-list.cache/* + +#. Remove old filesystem from ``/etc/fstab``:: + + vi /etc/fstab + # Remove the old root filesystem line: + # LABEL=writable / ext4 ... + +#. Configure kernel command line:: + + cp /boot/firmware/cmdline.txt /boot/firmware/cmdline.txt.bak + sed -i "s|root=LABEL=writable rootfstype=ext4|root=ZFS=rpool/ROOT/ubuntu_$UUID|" \ + /boot/firmware/cmdline.txt + sed -i "s| fixrtc||" /boot/firmware/cmdline.txt + sed -i "s|$| init_on_alloc=0|" /boot/firmware/cmdline.txt + + The ``fixrtc`` script is not compatible with ZFS and will cause the boot + to hang for 180 seconds. + + The ``init_on_alloc=0`` is to address `performance regressions + `__. + +#. Optional (but highly recommended): Make debugging booting easier:: + + sed -i "s|$| nosplash|" /boot/firmware/cmdline.txt + +#. Reboot:: + + exit + reboot + + Wait for the newly installed system to boot normally. Login as ``ubuntu``. + +Step 5: First Boot +------------------ + +#. Become root:: + + sudo -i + +#. Set the DISK variable again:: + + DISK=/dev/mmcblk0 # microSD card + + DISK=/dev/sdX # USB disk + +#. Delete the ext4 partition and expand the ZFS partition:: + + sfdisk $DISK --delete 3 + echo ", +" | sfdisk --no-reread -N 2 $DISK + + **Note:** This does not automatically expand the pool. That will be happen + on reboot. + +#. Create a user account: + + Replace ``YOUR_USERNAME`` with your desired username:: + + username=YOUR_USERNAME + + UUID=$(dd if=/dev/urandom bs=1 count=100 2>/dev/null | + tr -dc 'a-z0-9' | cut -c-6) + ROOT_DS=$(zfs list -o name | awk '/ROOT\/ubuntu_/{print $1;exit}') + zfs create -o com.ubuntu.zsys:bootfs-datasets=$ROOT_DS \ + -o canmount=on -o mountpoint=/home/$username \ + rpool/USERDATA/${username}_$UUID + adduser $username + + cp -a /etc/skel/. /home/$username + chown -R $username:$username /home/$username + usermod -a -G adm,cdrom,dip,lpadmin,lxd,plugdev,sambashare,sudo $username + +#. Reboot:: + + reboot + + Wait for the system to boot normally. Login using the account you + created. + +#. Become root:: + + sudo -i + +#. Expand the ZFS pool: + + Verify the pool expanded:: + + zfs list rpool + + If it did not automatically expand, try to expand it manually:: + + DISK=/dev/mmcblk0 # microSD card + DISKP=${DISK}p # microSD card + + DISK=/dev/sdX # USB disk + DISKP=${DISK} # USB disk + + zpool online -e rpool ${DISKP}2 + +#. Delete the ``ubuntu`` user:: + + deluser --remove-home ubuntu + +Step 6: Full Software Installation +---------------------------------- + +#. Optional: Remove cloud-init:: + + vi /etc/netplan/01-netcfg.yaml + + .. code-block:: yaml + + network: + version: 2 + ethernets: + eth0: + dhcp4: true + + :: + + rm /etc/netplan/50-cloud-init.yaml + apt purge --autoremove ^cloud-init + rm -rf /etc/cloud + +#. Optional: Remove other storage packages:: + + apt purge --autoremove bcache-tools btrfs-progs cloud-guest-utils lvm2 \ + mdadm multipath-tools open-iscsi overlayroot xfsprogs + +#. Upgrade the minimal system:: + + apt dist-upgrade --yes + +#. Optional: Install a full GUI environment:: + + apt install --yes ubuntu-desktop + echo dtoverlay=vc4-fkms-v3d >> /boot/firmware/usercfg.txt + + **Hint**: If you are installing a full GUI environment, you will likely + want to remove cloud-init as discussed above but manage your network with + NetworkManager:: + + rm /etc/netplan/*.yaml + vi /etc/netplan/01-network-manager-all.yaml + + .. code-block:: yaml + + network: + version: 2 + renderer: NetworkManager + +#. Optional (but recommended): Disable log compression: + + As ``/var/log`` is already compressed by ZFS, logrotate’s compression is + going to burn CPU and disk I/O for (in most cases) very little gain. Also, + if you are making snapshots of ``/var/log``, logrotate’s compression will + actually waste space, as the uncompressed data will live on in the + snapshot. You can edit the files in ``/etc/logrotate.d`` by hand to comment + out ``compress``, or use this loop (copy-and-paste highly recommended):: + + for file in /etc/logrotate.d/* ; do + if grep -Eq "(^|[^#y])compress" "$file" ; then + sed -i -r "s/(^|[^#y])(compress)/\1#\2/" "$file" + fi + done + +#. Reboot:: + + reboot + +Step 7: Final Cleanup +--------------------- + +#. Wait for the system to boot normally. Login using the account you + created. Ensure the system (including networking) works normally. + +#. Optional: For LUKS installs only, backup the LUKS header:: + + sudo cryptsetup luksHeaderBackup /dev/disk/by-id/scsi-SATA_disk1-part4 \ + --header-backup-file luks1-header.dat + + Store that backup somewhere safe (e.g. cloud storage). It is protected by + your LUKS passphrase, but you may wish to use additional encryption. + + **Hint:** If you created a mirror or raidz topology, repeat this for each + LUKS volume (``luks2``, etc.).