Sabbatical Diary: Week 31

Quieter week with a flurry at the end. A good seminar, an excellent friend’s birthday meal, and of course, I voted.¹

Finally got round to setting up several meetings connected with a couple of research projects.

And also spent far too long resurrecting some backups. I like restic as backup software: it seems to have the right set of features, it’s fast, and so far it’s been reliable. But I discovered that backups on two of my three independent, geographically distributed, external USB hard disks on which had become corrupted. This was annoying to say the least.

I still haven’t worked out exactly how but I actually think that it’s Linux/systemd to blame: as far as I can see, the there appear to be two mounts of /dev/sda1 onto /run/media/system/backup-LOCATION created, one that mounts /dev/sda1 with type ext4 and a second with type autofs that mounts systemd-1 onto the same path (?!). This would be weird but, I guess, fine except that the latter doesn’t always seem to be cleanly unmounted when the disk is unplugged (that is, I disconnect my laptop from its hub and wander off).²

The net effect of this appears to have been that sometimes I got one backup disk mounted onto two different mount points. Unfortunately, possibly, restic then deems both backup repos to exist and appears to have tried to do things assuming the one disk was two different repos each with different encryption keys.

In any case, Bad Things^™️ happened and the backup repo on one disk appeared totally screwed while the other had a bad snapshot. For the first, there was nothing to be done but blow away the broken repo and start again:

sudo -u restic restic -r $RESTIC_REPOSITORY -p $RESTIC_PASSWORD_FILE init

Note the use of user/group restic/restic: I took the opportunity to finally fix this in my [NixOS]](https://github.com/mor1/rc-files/blob/main/nixos/systems/greyjay/default.nix#L309-L318) config so that backups were not managed as either my usual user account or root.

The second was, happily, fixable. First, check what’s up and repair what’s possible to repair:

sudo -u restic restic -r $RESTIC_REPOSITORY -p $RESTIC_PASSWORD_FILE cache --cleanup
sudo -u restic restic -r $RESTIC_REPOSITORY -p $RESTIC_PASSWORD_FILE check --read-data
sudo -u restic restic -r $RESTIC_REPOSITORY -p $RESTIC_PASSWORD_FILE repair index
sudo -u restic restic -r $RESTIC_REPOSITORY -p $RESTIC_PASSWORD_FILE repair snapshots --forget

This repaired most of it except for one snapshot that was still broken to the extent that it could not be forgotten or pruned via the restic tool (“ciphertext validation” failed), so I had to nuke it by hand:

sudo rm $RESTIC_REPOSITORY/snapshots/SNAPSHOT_HASH

And with that, all was once again well. Or at least, backed up.