Ubuntu Raspberry Pi: Support USB disks

Signed-off-by: Scott G. Ainsworth <scott@ainsworth.us>
Co-authored-by: Richard Laager <rlaager@wiktel.com>
Signed-off-by: Richard Laager <rlaager@wiktel.com>

docs/Getting Started/Ubuntu/Ubuntu 20.04 Root on ZFS for Raspberry Pi.rst

@@ -12,7 +12,7 @@ Overview
 Caution
 ~~~~~~~
 
-- This HOWTO uses a whole physical SD card.
+- This HOWTO uses a whole physical disk.
 - Backup your data. Any existing data will be lost.
 
 System Requirements
@@ -22,11 +22,13 @@ System Requirements
   :doc:`Ubuntu 20.04 Root on ZFS`.)
 - `Ubuntu Server 20.04.3 (“Focal”) for Raspberry Pi 4
   <https://cdimage.ubuntu.com/releases/20.04.3/release/ubuntu-20.04.3-preinstalled-server-arm64+raspi.img.xz>`__
-- A microSD card. For recommendations, see Jeff Geerling's `performance
+- A microSD card or USB disk. For microSD card recommendations, see Jeff
-  comparison
+  Geerling's `performance comparison
   <https://www.jeffgeerling.com/blog/2019/raspberry-pi-microsd-card-performance-comparison-2019>`__.
-- An Ubuntu system (with the ability to write to the SD card) other than the
+  When using a USB enclosure, `ensure it supports UASP
-  target Raspberry Pi.
+  <https://github.com/geerlingguy/turing-pi-cluster/issues/11#issuecomment-647726561>`__.
+- An Ubuntu system (with the ability to write to the microSD card or USB disk)
+  other than the target Raspberry Pi.
 
 4 GiB of memory is recommended. Do not use deduplication, as it needs `massive
 amounts of RAM <http://wiki.freebsd.org/ZFSTuningGuide#Deduplication>`__.
@@ -97,6 +99,53 @@ entered at the console. Performance is good, but LUKS sits underneath ZFS, so
 if multiple disks (mirror or raidz topologies) are used, the data has to be
 encrypted once per disk.
 
+USB Disks
+~~~~~~~~~
+
+The Raspberry Pi 4 runs much faster using a USB Solid State Drive (SSD) than
+a microSD card. These instructions can also be used to install Ubuntu on a
+USB-connected SSD or other USB disk. USB disks have three requirements that
+do not apply to microSD cards:
+
+#. The Raspberry Pi's Bootloader EEPROM must be dated 2020-09-03 or later.
+
+   To check the bootloader version, power up the Raspberry Pi without an SD
+   card inserted or a USB boot device attached; the date will be on the
+   ``bootloader`` line. (If you do not see the ``bootloader`` line, the
+   bootloader is too old.) Alternatively, run ``sudo rpi-eeprom-update``
+   on an existing OS on the Raspberry Pi (which on Ubuntu requires
+   ``apt install rpi-eeprom``).
+
+   If needed, the bootloader can be updated from an existing OS on the
+   Raspberry Pi using ``rpi-eeprom-update -a`` and rebooting.
+   For other options, see `Updating the Bootloader
+   <https://www.raspberrypi.com/documentation/computers/raspberry-pi.html#updating-the-bootloader>`_.
+
+#. The Raspberry Pi must configured for USB boot. The bootloader will show a
+   ``boot`` line; if ``order`` includes ``4``, USB boot is enabled.
+
+   If not already enabled, it can be enabled from an existing OS on the
+   Raspberry Pi using ``rpi-eeprom-config -e``: set ``BOOT_ORDER=0xf41``
+   and reboot to apply the change. On subsequent reboots, USB boot will be
+   enabled.
+
+   Otherwise, it can be enabled without an existing OS as follows:
+
+   - Download the `Raspberry Pi Imager Utility
+     <https://www.raspberrypi.com/news/raspberry-pi-imager-imaging-utility/>`_.
+   - Flash the ``USB Boot`` image to a microSD card. The ``USB Boot`` image is
+     listed under ``Bootload`` in the ``Misc utility images`` folder.
+   - Boot the Raspberry Pi from the microSD card. USB Boot should be enabled
+     automatically.
+
+#. U-Boot on Ubuntu 20.04 does not seem to support the Raspberry Pi USB.
+   `Ubuntu 20.10 may work
+   <https://forums.raspberrypi.com/viewtopic.php?t=295609>`_.  As a
+   work-around, the Raspberry Pi bootloader is configured to directly boot
+   Linux.  For this to work, the Linux kernel must not be compressed. These
+   instructions decompress the kernel and add a script to
+   ``/etc/kernel/postinst.d`` to handle kernel upgrades.
+
 Step 1: Disk Formatting
 -----------------------
 
@@ -143,19 +192,34 @@ be deleted.
      3 : start=$((2048+BOOT+ROOT)), size=$ROOT, type=83
      EOF
 
-#. Connect the SD card:
+#. Connect the disk:
 
-   Connect the SD card to a machine other than the target Raspberry Pi.  If
+   Connect the disk to a machine other than the target Raspberry Pi.  If any
-   any filesystems are automatically mounted (e.g. by GNOME), unmount them.
+   filesystems are automatically mounted (e.g. by GNOME) unmount them.
-   Determine the device name (which is almost certainly as shown below) and
+   Determine the device name. For SD, the device name is almost certainly
-   set it in a variable::
+   ``/dev/mmcblk0``. For USB SSDs, the device name is ``/dev/sdX``, where
+   ``X`` is a lowercase letter. ``lsblk`` can help determine the device name.
+   Set the ``DISK`` environment variable to the device name::
 
-     DISK=/dev/mmcblk0
+     DISK=/dev/mmcblk0    # microSD card
+     DISK=/dev/sdX        # USB disk
+
+   Because partitions are named differently for ``/dev/mmcblk0`` and ``/dev/sdX``
+   devices, set a second variable used when working with partitions::
+
+     export DISKP=${DISK}p # microSD card
+     export DISKP=${DISK}  # USB disk ($DISKP == $DISK for /dev/sdX devices)
+
+   **Hint**: microSD cards connected using a USB reader also have ``/dev/sdX``
+   names.
+
+   **WARNING**: The following steps destroy the existing data on the disk. Ensure
+   ``DISK`` and ``DISKP`` are correct before proceeding.
 
 #. Ensure swap partitions are not in use::
 
      swapon -v
-     # If a partition is in use from the SD card, disable it:
+     # If a partition is in use from the disk, disable it:
      sudo swapoff THAT_PARTITION
 
 #. Clear old ZFS labels::
@@ -167,14 +231,14 @@ be deleted.
 
    **Hint:** If you do not already have the ZFS utilities installed, you can
    install them with: ``sudo apt install zfsutils-linux``  Alternatively, you
-   can zero the entire SD card with:
+   can zero the entire disk with:
    ``sudo dd if=/dev/zero of=${DISK} bs=1M status=progress``
 
 #. Delete existing partitions::
 
      echo "label: dos" | sudo sfdisk ${DISK}
      sudo partprobe
-     ls ${DISK}*
+     ls ${DISKP}*
 
    Make sure there are no partitions, just the file for the disk itself.  This
    step is not strictly necessary; it exists to catch problems.
@@ -190,11 +254,11 @@ be deleted.
 
 #. Copy the bootloader data::
 
-     sudo dd if=${IMG}p1 of=${DISK}p1 bs=1M
+     sudo dd if=${IMG}p1 of=${DISKP}1 bs=1M
 
 #. Clear old label(s) from partition 2::
 
-     sudo wipefs -a ${DISK}p2
+     sudo wipefs -a ${DISKP}2
 
    If a filesystem with the ``writable`` label from the Ubuntu image is still
    present in partition 2, the system will not boot initially.
@@ -202,15 +266,59 @@ be deleted.
 #. Copy the root filesystem data::
 
      # NOTE: the destination is p3, not p2.
-     sudo dd if=${IMG}p2 of=${DISK}p3 bs=1M status=progress conv=fsync
+     sudo dd if=${IMG}p2 of=${DISKP}3 bs=1M status=progress conv=fsync
 
 #. Unmount the image::
 
      sudo losetup -d $IMG
 
+#. If setting up a USB disk:
+
+   Decompress the kernel::
+
+     sudo -sE
+
+     MNT=$(mktemp -d /mnt/XXXXXXXX)
+     mkdir -p $MNT/boot $MNT/root
+     mount ${DISKP}1 $MNT/boot
+     mount ${DISKP}3 $MNT/root
+
+     zcat -qf $MNT/boot/vmlinuz >$MNT/boot/vmlinux
+
+   Modify boot config::
+
+     cat >> $MNT/boot/usercfg.txt << EOF
+     kernel=vmlinux
+     initramfs initrd.img followkernel
+     boot_delay
+     EOF
+
+   Create a script to automatically decompress the kernel after an upgrade::
+
+     cat >$MNT/root/etc/kernel/postinst.d/zz-decompress-kernel << 'EOF'
+     #!/bin/sh
+
+     set -eu
+
+     echo "Updating decompressed kernel..."
+     [ -e /boot/firmware/vmlinux ] && \
+         cp /boot/firmware/vmlinux /boot/firmware/vmlinux.bak
+     vmlinuxtmp=$(mktemp /boot/firmware/vmlinux.XXXXXXXX)
+     zcat -qf /boot/vmlinuz > "$vmlinuxtmp"
+     mv "$vmlinuxtmp" /boot/firmware/vmlinux
+     EOF
+
+     chmod +x $MNT/root/etc/kernel/postinst.d/zz-decompress-kernel
+
+   Cleanup::
+
+     umount $MNT/*
+     rm -rf $MNT
+     exit
+
 #. Boot the Raspberry Pi.
 
-   Move the SD card into the Raspberry Pi. Boot it and login (e.g. via SSH)
+   Move the SD/USB disk to the Raspberry Pi. Boot it and login (e.g. via SSH)
    with ``ubuntu`` as the username and password.  If you are using SSH, note
    that it takes a little bit for cloud-init to enable password logins on the
    first boot.  Set a new password when prompted and login again using that
@@ -225,11 +333,17 @@ Step 2: Setup ZFS
 
      sudo -i
 
-#. Set a variable with the disk name::
+#. Set the DISK and DISKP variables again::
 
-     DISK=/dev/mmcblk0
+     DISK=/dev/mmcblk0    # microSD card
+     DISKP=${DISK}p       # microSD card
 
-   On the Pi, this is always ``mmcblk0``.
+     DISK=/dev/sdX        # USB disk
+     DISKP=${DISK}        # USB disk
+
+   **WARNING:** Device names can change when moving a device to a different
+   computer or switching the microSD card from a USB reader to a built-in
+   slot. Double check the device name before continuing.
 
 #. Install ZFS::
 
@@ -252,7 +366,7 @@ Step 2: Setup ZFS
            -O acltype=posixacl -O canmount=off -O compression=lz4 \
            -O dnodesize=auto -O normalization=formD -O relatime=on \
            -O xattr=sa -O mountpoint=/ -R /mnt \
-           rpool ${DISK}p2
+           rpool ${DISKP}2
 
    **WARNING:** Encryption has not yet been tested on the Raspberry Pi.
 
@@ -265,11 +379,11 @@ Step 2: Setup ZFS
            -O acltype=posixacl -O canmount=off -O compression=lz4 \
            -O dnodesize=auto -O normalization=formD -O relatime=on \
            -O xattr=sa -O mountpoint=/ -R /mnt \
-           rpool ${DISK}p2
+           rpool ${DISKP}2
 
    - LUKS::
 
-       cryptsetup luksFormat -c aes-xts-plain64 -s 512 -h sha256 ${DISK}p2
+       cryptsetup luksFormat -c aes-xts-plain64 -s 512 -h sha256 ${DISKP}2
        cryptsetup luksOpen ${DISK}-part4 luks1
        zpool create \
            -o ashift=12 \
@@ -416,7 +530,7 @@ Step 3: System Installation
 
 #. Optional: Ignore synchronous requests:
 
-   SD cards are relatively slow.  If you want to increase performance
+   microSD cards are relatively slow.  If you want to increase performance
    (especially when installing packages) at the cost of some safety, you can
    disable flushing of synchronous requests (e.g. ``fsync()``, ``O_[D]SYNC``):
 
@@ -509,7 +623,6 @@ Step 4: System Configuration
 #. Setup system groups::
 
      addgroup --system lpadmin
-     addgroup --system lxd
      addgroup --system sambashare
 
 #. Patch a dependency loop:
@@ -598,10 +711,16 @@ Step 5: First Boot
 
      sudo -i
 
+#. Set the DISK variable again::
+
+     DISK=/dev/mmcblk0    # microSD card
+
+     DISK=/dev/sdX        # USB disk
+
 #. Delete the ext4 partition and expand the ZFS partition::
 
-     sfdisk /dev/mmcblk0 --delete 3
+     sfdisk $DISK --delete 3
-     echo ", +" | sfdisk --no-reread -N 2 /dev/mmcblk0
+     echo ", +" | sfdisk --no-reread -N 2 $DISK
 
    **Note:** This does not automatically expand the pool.  That will be happen
    on reboot.
@@ -643,7 +762,13 @@ Step 5: First Boot
 
    If it did not automatically expand, try to expand it manually::
 
-     zpool online -e rpool mmcblk0p2
+     DISK=/dev/mmcblk0    # microSD card
+     DISKP=${DISK}p       # microSD card
+
+     DISK=/dev/sdX        # USB disk
+     DISKP=${DISK}        # USB disk
+
+     zpool online -e rpool ${DISKP}2
 
 #. Delete the ``ubuntu`` user::
 
@@ -658,11 +783,13 @@ Step 6: Full Software Installation
 
    .. code-block:: yaml
 
-     network:
+    network:
-       version: 2
+      version: 2
-       ethernets:
+      ethernets:
-         eth0:
+        eth0:
-           dhcp4: true
+          dhcp4: true
+
+   ::
 
     rm /etc/netplan/50-cloud-init.yaml
     apt purge --autoremove ^cloud-init