Sabbatical Diary: Weeks 28—29

Another delayed one—must try harder… :/

Spent some time doing more paper reviewing: EdgeSys’26 and ICDCS’26 this time, plus a couple of UKRI research funding proposals (none for 10 years then two turn up at once…).

Pursued some old contacts for collaboration possibilities with their current employers—here’s hoping the proposals are of interest!

Attended a couple of—largely theory—talks which was interesting. And led to an interesting follow-up conversation about one of my ideas with one of the speakers who also attended the UK Systems workshop which was, as ever, fun. And I got to edit some more webpages so that was nice :)

But most of the time was spent on two activities. First, finalising content for the final review meeting for the EDGELESS project and getting ready to travel to Munich, by train naturally. Second, recovering from a configuration screw-up on my part.¹ The content looks ok—more slides to tweak but a great demo video put together by my visitor, Theodoros Aslanidis.

The travel was a little more interesting to organise than I’d hoped due to the dreaded engineering works on the line. This meant that there were going to be three rail replacement buses between Cambridge and London² which would result in my not getting to St Pancras in time to get a sufficiently early Eurostar.³ Happily I realised that I could get a later train after the engineering works are scheduled to have stopped⁴ which would get me to St Pancras in time for the 11.04 to Amsterdam, from where I can get the overnight NightJet to Munich, scheduled to arrive at about 06.30, in time for the rehearsal day and then the review itself on Tuesday. Hopefully this will all go smoothly and it’ll be fine. Fingers crossed…

Recovering from the configuration screw up was a bit more stressful. It managed to make my laptop unbootable late at night, so I left it to the following morning to fix, and it took most of that morning indeed. The symptom was a kernel panic on entering the NixOS Stage 2 init, which seemed pretty bad. In the end I fixed it by:

• booting from a USB install media,
• getting a shell, mounting my partitions, and entering the resulting environment, and
• repairing the broken install, possibly by turning on systemd initrd.

I had, in my flake, to rename the system greyjay to nixos for this to work, and I tweaked a few config settings to make interaction via a console (i.e., no window manager) a bit less painful:

audit.enable = false # turn off auditing as it spams the console
consolemode = "auto" # force picking a sensible console font so I can see the entire entry
initrd.systemd.enable = true # some online sources claim it's more reliable?

Next, in the Grub menu selecting the entry and hitting e to edit the boot command line meant I could edit/add the following parameters for more information during boot:

loglevel=8 boot.debugtrace boot.trace systemd.log_level=debug systemd.log_target=console

…and I then pointed my iPad at the screen and started a high-speed (60fps) video recording to try to capture debug spew as it flew past. Had to type in the LUKS decryption passphrase once the boot scripts start looping asking for input. With all those debug boot parameters, this passphrase is echoed to the console as you type so be careful!

Finally, some commands and other options I found useful during this process, in case future-me manages to screw this up again:

sudo -i # to make a shell into a root shell

DEVICE=/dev/nvme0n1p4 echo -n $PASSPHRASE | cryptsetup luksOpen $DEVICE cryptroot --allow-discards --key-file=-

mount /dev/disk/by-label/nixos /mnt # mount root partition
mount /dev/disk/by-label/SYSTEM /mnt/boot # mount boot partition
mount /dev/disk/by-label/nixos-home /mnt/home # mount home partition
nixos-enter # do the `chroot` and all the rest to get into the bootstrapped system

…and then finally doing a full nixos-rebuild switch --sudo to recreate the environment. What fun! Like the good ol’ times when I used to break my computers on a regular basis!