Upgrading to FreeBSD 14 - how to fix a broken BIOS bootcode
Published on 2023-11-22. Modified on 2023-11-27.
A lot of people running ZFS zroot have managed to break their FreeBSD systems upgrading from 13.2 to the new 14.0 release because of a broken BIOS bootcode. In this tutorial I'll show you how you can fix that without having to reinstall.
In my humble opinion, the release notes for FreeBSD 14.0 is lacking relevant information to a lot of users and I do not like how these issues have been handled.
On the blog of Colin Percival, who is the new FreeBSD Release Engineering Lead, he writes that:
I assumed the role of FreeBSD Release Engineering Lead a few days ago, and one of my first duties in the role was to write and send out the FreeBSD 14.0-RELEASE announcement. (To be clear: Glen Barber did all of the work of getting the release ready; the final bits had already been copied out to mirrors at the point that I took over.) FreeBSD 14 is a great release, but there are a few last-minute issues which deserve to be documented — probably somewhere on the FreeBSD website, but I can post to my blog much faster and hopefully we'll get these onto the FreeBSD website later.
Again, in my humble opinion, and with all due respect, that is just not good enough.
A project should NEVER release anything until ALL issues have been fully and clearly documented on the project website, in the release notes, and mailing list posts, etc.
The most devastating issue has been people running FreeBSD 13.2 on a BIOS setup rather than EFI who managed to break their systems completely, making them unable to boot.
This issue could have been avoided very easily with the correct and relevant information, but instead the release notes only deal with issues related to EFI systems, not BIOS systems.
The release notes says the following:
Note for systems that boot via EFI, using either binary or source upgrades: There are one or more copies of the boot loader on the MS-DOS EFI System Partition (ESP), used by the firmware to boot the kernel. If the root file system is ZFS, the boot loader must be able to support reading from the ZFS boot file system. After a system upgrade, but before doing a zpool upgrade, the boot loader on the ESP must be updated, or the system may become unbootable.
After finishing up the upgrade using the freebsd-update
tool, most ZFS users typically upgrade their ZFS pools too. When that is done on the zroot pool the following message is supplied:
# zpool upgrade zroot This system supports ZFS pool feature flags. Enabled the following features on 'zroot': edonr zilsaxattr head_errlog blake3 block_cloning vdev_zaps_v2 Pool 'zroot' has the bootfs property set, you might need to update the boot code. See gptzfsboot(8) and loader.efi(8) for details.
Now, I am sure most users ran man gptzfsboot
and man loader.efi
or perhaps looked at the FreeBSD online manual pages to get the relevant information, but for many users, dealing with "boot code" has just not been relevant before.
When you install FreeBSD using the installation tool, you do not have to deal with any bootcode related stuff, regardless of whether you use a BIOS or EFI setup. So you can basically have been running FreeBSD for many years without ever looking at anything bootcode related.
In any case, looking at the EXAMPLES section at the man page for gptzfsboot
doesn't really help either unless you truly know what this is all about.
gptzfsboot is typically installed in combination with a "protective MBR" (see gpart(8)). To install gptzfsboot on the ada0 drive:
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
gptzfsboot can also be installed without the PMBR:
gpart bootcode -p /boot/gptzfsboot -i 1 ada0
The man page for loader.efi
is completely useless in this regard as that is only related to the UEFI kernel loader.
If you reboot the computer without installing a new gptzfsboot
bootcode, you will see something like the following during boot:
ZFS: unsupported feature: com.delphix:head_errlog ZFS: pool zroot is not supported Can't find /boot/zfsloader Can't find /boot/loader Can't find /boot/kernel/kernel
And the machine will then halt.
The solution
The reason for the problem with BIOS based systems is that gptzfsboot
is used to boot from a filesystem in a ZFS pool. gptzfsboot
is installed in a freebsd-boot partition of a GPT-partitioned disk with gpart
. When the computer boots, gptzfsboot
tries to find all ZFS pools that are composed of BIOS visible hard disks or partitions. gptzfsboot
then looks for ZFS device labels on all visible disks and in discovered supported partitions for all supported partition scheme types.
Because ZFS was upgraded to version 2.2 a new gptzfsboot
bootcode needs to be installed on the freebsd-boot partition.
So what you can do to solve the issue is:
- Boot from a FreeBSD 14 installation media
- Choose
<LiveCD>
- Log in as root (no password is required)
Then list your disks with the gpart show
command. The disk I run zroot on in this example is ada1
so I'll use this here. If you are running your zroot on a mirror of e.g. two disks, you need to locate both of them.
Be careful that you choose the correct disks.
# gpart show => 40 234441568 ada1 GPT (112G) 40 1024 1 freebsd-boot (512K) 1064 984 - free - (492K) 2048 4194304 2 freebsd-swap (2.0G) 4196352 230244352 3 freebsd-zfs (110G) 234440704 904 - free - (452K)
Then you just install the new bootcode from the install media with the following command (change ada1
to match your disk, and also pay attention to the partition number which is addressed with the -i
option - you need the "freebsd-boot" partition, which is the first partition in this example):
# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1 partcode written to ada1p1 bootcode written to ada1
Then repeat the command for the mirror disks, if you have that.
That's it. You can now reboot the machine and your zpool will work again.
Email from Aldo Gonzalez (shortened)
I actually had an issue with booting up a VM using EFI and had to boot from a live cd and run:
# zpool import zroot # mkdir -p /tmp/esp/ # mount -t msdosfs /dev/da0p1 /tmp/esp # cp /tmp/esp/EFI/BOOT/bootx64.efi /tmp/esp/EFI/BOOT/bootx64.efi.bak # cp /boot/loader.efi /tmp/esp/EFI/BOOT/bootx64.efi # reboot