Can’t believe I finally did it. Give the website a decent look, hugo new posts/freebsd-aws/index.md, kick off Vim and finally start writing something. Woo-hoo. Let’s celebrate by jumping straight into it 🎉

This short post is about my latest adventure at work: switching from Ubuntu to FreeBSD, in Amazon Web Services, using ZFS instead of the standard UFS as the root filesystem. Such a sexy setup, but not as straightforward as expected. Turns out it’s not difficult either if you know where to look, so I hope this post is useful to anyone considering giving it a try.

FreeBSD, quick

FreeBSD is a server-oriented operating system whose roots go back through the BSD family to the first UNIXes. Although it has way less development manpower than, say, Linux, its principled and conservative philosophy make it a wonderful choice as a rock-solid OS for business operation. Big companies such as Netflix or WhatsApp rely on it for both its stability and impressive networking performance.

Current versions of FreeBSD support booting from either of these filesystems:

• UFS/FFS (Unix File System / Fast File System), which is the filesystem traditionally used in FreeBSD and other BSDs. It comes from the original UNIX implementation, it’s battle tested but lacks some modern features such as journaling.
• ZFS (Z File System). On the other side, developed originally for Solaris but ported to FreeBSD in 2007, ZFS has some fairly advanced features that make it worth considering, such as atomic snapshots, compression, self-healing and volume management (à la RAID+LVM).

… in AWS

There are official AMIs for the latest FreeBSD versions in the AWS marketplace (12, 13) that you can instantiate right away. The catch is that, at least for now, all the official AMIs are based on UFS as root filesystem. It may be the case that you want need to use ZFS for the greater nerd fun for some of the crazy stuff things you can do with it.

If so, then you’ll need to build a custom AMI. Thankfully, the hard part has already been done by former FreeBSD Security Officer Colin Percival, who provides Builder AMIs that make it much easier to make small (or not so small) modifications to the standard FreeBSD release and generate a custom AMI with it. Colin frequently updates his blog and Patreon with his progress on FreeBSD @ AWS, so if you’re interested about this platform you should definitely keep an eye on both¹.

So, here are the amd64 Builder AMIs for the latest FreeBSD releases:

You can launch one of these, make some changes to the system mounted in /mnt, shut down the instance, and create a new AMI from the instance (vía the AWS Console). Magic.

Just keep in mind that you’ll need a beefy Nitro instance with at least 8GB of RAM to launch one of these AMIs. t2.2xlarge worked fine for me at the time of writing this.

… over ZFS

Thankfully again, most of the work to adapt a Builder Instance to generate a ZFS-rooted AMIs is documented in Colin’s Blog, so we basically just need to follow his steps. While I was doing it I needed to adapt just a couple of things:

• The main disk name is nvd0 instead of ada0
• The growfs fix (step 8) can be skipped altogether, as it has already been applied upstream

Here is the final recipe:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

mdconfig -a -t swap -s 3G -u 2
newfs /dev/md2
mkdir /mdisk
mount /dev/md2 /mdisk
tar -czf /mdisk/base.tgz --exclude .snap -C /mnt .
umount /mnt

gpart destroy -F nvd0
dd if=/dev/zero bs=128k of=/dev/nvd0
gpart create -s gpt nvd0
gpart add -a 4k -s 512K -t freebsd-boot nvd0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 nvd0
gpart add -a 1m -t freebsd-zfs -l disk0 nvd0

zpool create -o altroot=/mnt -O compress=lz4 -O atime=off -m none -f zroot nvd0p2
zfs create -o mountpoint=none zroot/ROOT
zfs create -o mountpoint=/ -o canmount=noauto zroot/ROOT/default
mount -t zfs zroot/ROOT/default /mnt
zfs create -o mountpoint=/tmp -o exec=on -o setuid=off zroot/tmp
zfs create -o mountpoint=/usr -o canmount=off zroot/usr
zfs create zroot/usr/home
zfs create -o setuid=off zroot/usr/ports
zfs create zroot/usr/src
zfs create -o mountpoint=/var -o canmount=off zroot/var
zfs create -o exec=off -o setuid=off zroot/var/audit
zfs create -o exec=off -o setuid=off zroot/var/crash
zfs create -o exec=off -o setuid=off zroot/var/log
zfs create -o atime=on zroot/var/mail
zfs create -o setuid=off zroot/var/tmp
zpool set bootfs=zroot/ROOT/default zroot

tar -xf /mdisk/base.tgz -C /mnt

: > /mnt/etc/fstab
echo 'zfs_load="YES"' >> /mnt/boot/loader.conf
echo 'kern.geom.label.disk_ident.enable="0"' >> /mnt/boot/loader.conf
echo 'kern.geom.label.gptid.enable="0"' >> /mnt/boot/loader.conf
echo 'vfs.zfs.min_auto_ashift=12' >> /mnt/etc/sysctl.conf
echo 'zfs_enable="YES"' >> /mnt/etc/rc.conf

shutdown -p now

To wrap up:

1. Find the ID of the correct Builder AMI you need for your AWS region, architecture and FreeBSD version.
2. Make a new instance from that AMI and wait for it to start.
3. Connect to it, using the ec2-user account, and follow the steps above to move the root disk from UFS to ZFS.
4. Stop the instance and generate a new image from the AWS Console.
5. Delete the intermediate instance.

At the end of this process we should finally have our shiny new AMI ready to launch FreeBSD-on-ZFS instances.

… the lazy way

I’m sure you have already noticed, as the virtuous programmer that you are, that this process is quite boring and error prone, and has to be repeated any time a new version of FreeBSD gets out.

So, here’s a script I made to automate all this process: gen_freebsd_zfs_ami.sh

Yeah, there is hardcoded stuff and you will probably need to tune it a little bit, but there are lots of things one could need to do depending on one’s specific case so… yeah.

That’s all for now, see you around ;)