« Last night the backup saved my life 🎵 »

Behind this weird title stolen from the disco hit « Last night the DJ saved my life » from the 80s by the group Indeep, another laptop issue that could have been very painful but ended up just being a bit of wasted time. Same as the song says « Last night the DJ saved my life from a broken heart », I could rephrase it to « Last night the backups saved my life from a broken Archlinux update ».

What happened? I’m not sure… I went through my usual Sunday evening routine where I backup my data to 2 different devices. A full copy except a few folders to an external encrypted SDD and a partial copy (mainly my ~/.ssh, ~/.gnupg, ~/.password-store and ~/.config folders) to an encrypted flash drive. I wrote about this already so no more details needed.

Then, I usually update my system. Using Archlinux (more specifically EndeavourOS, which is based on Archlinux), I don’t update it daily but weekly as I don’t want any surprises during the week. And 99% of the time, updates don’t bring surprises, but last night it went sideways… After the updates, I usually reboot my computer, a way to ensure I’m using the latest installed kernel version and software that were updated.

But the computer didn’t reboot… It failed to decrypt my internal hard drive where the system and my home data are… Leaving me with just a more or less broken prompt to try to fix things. The error was something like « Failed to start cryptography setup luks ». Not sure what went wrong, maybe a dracut related issue but I’m not sure. As it was almost 1AM, I decided to go to bed and see this morning.

This morning, after a couple of checks and tests, nothing fixed it… As I needed my computer to work, I had 2 options: either spend possibly hours trying to debug using the small screen of my phone to find a way to solve it… Or reinstall the system and use my external SSD with almost all my home to retrieve data.

Deciding to go the later road, I put the EndeavourOS USB key and started the computer. From the live install, I had access to the encrypted internal drive, so I even took the time to copy to my external drive the 2 folders that are not synced every Sunday (my ~/Téléchargements - aka downloads in French - and my ~/Videos folders). I didn’t mind loosing what was in there, but as I had still access to it, why not waste 20 minutes or so copying them to avoid loosing anything 🤷…

Once the copy was done, I went through the installation process, which thankfully is easy and fast using the EndeavourOS installer. More or less and an hour after booting my computer with the EndeavourOS usb key, I had again access to a working system (additional backups included).

All I needed was to copy all files back to my internal drive and reinstall some software with pacman / yay and they were configured just as before (even things like opened tabs in Firefox were still here). Just for future use, these 2 commands could really help speed things up:

sudo pacman -S vim neovim pass autorandr alacritty rofi rofi-pass zsh tmux mupdf htop dfc bat mpv ttf-firacode-nerd xss-lock xautolock emacs hunspell-fr hunspell-en_us rofimoji thunderbird nheko signal-desktop chromium libadwaita torbrowser-launcher hugo
yay -S zoom factorio streamcontroller vcmi

I’m sure I’m still missing things here, but with these 2 commands I have most of the tools needed to get back on track!

Also, a quick thanks to myself for some of my own blog posts to quickly find some things to do on top of just installing software. Things like the udev rule for my streamdeck or the configuration needed for hibernation and power management and so on!

As I said in a previous blog post where I weirdly thanked myself for blogging: keep on blogging! And keep on weekly backups! (or even pre-update backups, whenever that happens!).

A story that could have been way worse but ended up being a couple of wasted hours. I’m happy to have just lost those hours and no data though, so happy trade off :-).

TLDR; Make backups, they saved me yet again this morning!