Installing NixOS on a Raspberry Pi 3B

I have a Raspberry Pi 3B with an external hard drive at my parent’s house as an off-site backup server. Until recently I had Arch Linux ARM running on that, because when I originally set it up I was using Arch as my daily driver and it felt natural to use what I know already anyway.

Recently I decided to switch that Pi to NixOS, because NixOS is now my daily driver on my other machines. I like the prospect of having the Pi’s config from the same config repo I use for everything else, so I can reuse the config modules I already wrote there. It also seemed like a good exercise before switching my main server over to NixOS, which currently still runs Arch.

While I have it running now, the whole endeavour didn’t go quite as smoothly as I originally hoped. Here’s a write-up of what I did and what problems I ran into.

I scraped this information together from various sources linked during the text, and some pages from the NixOS wiki: here, here and here. This is all not terribly well documented, sadly. I plan to add some of the stuff explained in this post to nix.dev when I find the time.

Installing

NixOS SD Images

I started off with “Installing NixOS on a Raspberry Pi” on nix.dev. That tutorial is for the Pi 4B, but at that point I thought that my Pi was just that, so oh well.

Note that the ARM ISO images you find on the NixOS download page are normal live installation images just like the x86 ones, i.e. you would copy them on a USB drive or similar, boot from that and then install onto a different disk from that live system.

For a Pi you probably want an SD image that you can flash onto the SD card and use directly from the card. Most RPi distributions do it like that.

NixOS also provides such images, but not on the download page itself. You have to poke around in the Hydra CI builds. The most recent images can be found here: https://hydra.nixos.org/job/nixos/trunk-combined/nixos.sd_image.aarch64-linux (see the nix.dev guide).

Building the Config

I started off by doing a major refactor on my NixOS config repo so I could reuse my modules while easily excluding stuff that I don’t need on the Pi, which wasn’t trivial due to my config previously being structured in a rather convoluted way.

To start you off, the nix.dev guide has a config template for a Pi 4 that uses nixos-hardware.

For a Pi 3B like mine you don’t need extra stuff from nixos-hardware though. The config nixos-generate-config gives you works pretty much out of the box, although I stumbled a bit when configuring the bootloader, see below.

You can take a look at the config I ended up with here. It’s flake-based, integrated with the config modules I use for my other devices and includes some fun stuff like dynamic DNS through the DeSEC API.

Deploying to the Pi

Normally, to install my flake-based NixOS config on a new device I would just call nixos-install with --flake my-flake-url.

However, that didn’t work this time. It might for you, but my pre-existing flake-based config didn’t even evaluate in a reasonable timeframe on the Pi. I let it run over night, and it wasn’t finished in the morning, which was disappointing. Yeah, my config has a bunch of files, but it’s really not that big. I think. But well, Nix evaluation seems to be quite RAM-hungry, too hungry for the Pi 3B’s 1 GB.

So instead of building the config locally on the PI, I ended up building it on my PC. In order for this to work on a non-ARM machine, you have to add this to your NixOS config first though:

boot.binfmt.emulatedSystems = [ "aarch64-linux" ];

This allows you to do Nix builds for other architectures through QEMU without any setup other that this option, which is pretty neat.

You can then build the config from the config flake directory like so:

nix build .#nixosConfigurations.[the name of the config for the Pi].config.system.build.toplevel --print-out-paths

That prints out the store path to the built system, e.g. in my case /nix/store/p2qddwy3dl4shjxa487hsxdfcs4s4pan-nixos-system-obsidian-22.11.20230524.99fe1b8. We want to copy that path over to the Pi, including its dependencies (that’s called the closure). We can do that with nix copy.

For that, you need to be able to reach the Pi from your build machine via SSH. So, connect both devices to a network and add your SSH key to the root user’s authorized_keys on the Pi.

It’s important that you add an SSH key, and don’t just use a password, because at least by default nix copy doesn’t ask for a password and just hangs instead. You might be able to fix that by setting the env variable NIX_SSHOPTS="-t", but I haven’t tried that. It works for nixos-rebuild though, see below.

It’s also important to use the root user on the Pi, because for nix copy to work as expected you need be a trusted user on the target device. Otherwise you’ll get error messages about missing signatures. Even though the problem doesn’t have anything to do with signatures, just with you not being a trusted user. Wasn’t a fun error to debug.

Anyway, after establishing SSH access, you can then copy over the closure:

nix copy [the store path] --to ssh://root@[the IP of the Pi]

After copying, you can finally run nixos-install on the Pi:

nixos-install --root / --system [the store path] --no-root-password

By specifying --system we tell nixos-install to use that closure instead of attempting to build one, which is exactly what we want. Leave out --no-root-password if you want to set a root password during install.

Rebuilding Remotely

Thankfully, we don’t have to do all this annoying stuff for subsequent rebuilds of the system after the initial installation, because nixos-rebuild supports remote rebuilds directly.

NIX_SSHOPTS="-t -p [SSH port if not 22]" nixos-rebuild switch --flake .#[the name of the config for the Pi] --target-host foo@example.com --use-remote-sudo

--use-remote-sudo tells nixos-rebuild to obtain root access to the target host via sudo. -t as SSH option is required so a TTY is allocated, which is necessary for entering the remote sudo password. Instead, you could also directly SSH into the root user like we did in the installation process, but I prefer to disallow root SSH access on my machines.

Understanding the Bootloader and Firmware

The NixOS SD images use an MBR partition table and consist of two partitions: a small FAT firmware partition at the start that contains the firmware and bootloader, and an ext4 partition containing everything else.

What’s this firmware stuff? So. When the Pi starts up, the boot process begins with some on-board bootloader (curiously, that’s run by the GPU, not the CPU). That reads the SD card and looks into the first partition where it expects a bunch of firmware binaries with specific names and a config file, which are then loaded and continue the boot process. From there, a Linux kernel is loaded to finish up the boot. The firmware binaries are proprietary blobs, by the way, which is a little sad.

Because the boot process starts with an on-board bootloader that we can’t change (the Pi 4 has a writable EEPROM, so the on-board firmware can be updated, the older Pi’s don’t have that), and because that bootloader doesn’t support GPT partition tables, the SD card must use an MBR partition table.

You also can’t boot from USB or use a bootloader like GRUB without further steps, which is part of the reason you can’t just install a “normal” desktop linux distribution on a Pi.

NixOS and the Pi Boot Process

The NixOS SD images also have the Pi firmware in the firmware partition, but instead of letting the integrated bootloader load a kernel directly, the bootloader U-Boot is loaded and used as an intermediary.

U-Boot is more flexible and supports stuff like a boot menu, which is especially important for NixOS so you can select a generation to boot and so on.

U-Boot reads /boot/extlinux/extlinux.conf on the main ext4 partition. This contains the boot menu configuration. U-Boot then loads the selected kernel with the selected options from /boot/nixos.

So everything for that part of the boot prowess except the U-Boot binary itself is located on the main partition itself. Note that the firmware partition isn’t even mounted on NixOS by default, so it isn’t touched by system rebuilds.

The extlinux.conf file read by U-Boot is generated by NixOS if you enable it like so:

boot.loader.grub.enable = false;  # GRUB is used by default, turn that off
boot.loader.generic-extlinux-compatible.enable = true;

What confused me quite a bit at first: don’t be tempted by the options under boot.loader.raspberryPi and boot.loader.raspberryPi.uboot! These seem to try to install U-Boot itself and some Pi boot loader config stuff in /boot, which would only make sense if the firmware partition is mounted as /boot. But when using the SD images, the firmware partition isn’t mounted anywhere at all and pre-populated with U-Boot and the firmware stuff anyway.

U-Boot USB Troubles

On my first try I got the Pi working as I wanted, except for one thing: it hanged during U-Boot booting when a USB hard drive was attached. Which is somewhat inconvenient for a backup server that is meant to have the USB hard drive attached permanently that I want to be able to reboot remotely.

After a brief excursion into Pi UEFI firmware (see below), I was able to fix that by just using a newer SD image. I previously downloaded one from the release-22.11 branch from Hydra, but with a current one from trunk it worked. It has a newer U-Boot version (2023.1). I guess the problem was fixed in the meantime.

By the by, that’s a downside of the unmanaged-and-unmounted firmware partition approach of the NixOS images: U-Boot cannot be updated by NixOS. To update it you’d have to download and copy updated files manually.

A Brief Excursion into UEFI on the Pi

While trying to get around my problems with U-Boot (see above), I stumbled upon the Raspberry Pi UEFI firmwares. I linked the RPi 3 version, but it’s available for the Pi 4 as well.

It replaces the normal Raspberry Pi firmware you’d put in the firmware partition on the SD card and provides a fully-fledged UEFI interface, so you can boot UEFI stuff like you would on a PC. It includes a boot menu, an EFI shell and all that jazz.

That allows you to install vanilla Linux distributions on the Pi, which is pretty cool. If that interests you, take a look at this article: https://pete.akeo.ie/2019/07/.

After extracting the firmware into a FAT partition on an SD card as instructed, I was able to boot the Pi into the UEFI setup without any problems.

I tried to install a vanilla NixOS from an USB stick with the ARM install ISO, but sadly I didn’t end up using that. systemd-boot, my boot loader of choice, doesn’t support MBR disks. And even with the UEFI firmware you still have to format your SD card with an MBR due to the on-board bootloader not supporting GPT.

Using GRUB also didn’t work, it seems that NixOS currently has a bug with installing GRUB on ARM. So, when it turned out that the newer NixOS SD images work for me just fine (see above), I used that instead.

Still, I find the UEFI approach really nice, and I might try again with this firmware in the future. It seems you can do hybrid GPT/MBR disks and maybe that’ll allow me to use systemd-boot.