Facing my digital worse nightmare

Hello everyone!

TLDR; I lost access to most of my data, including my GPG key (so don’t send me encrypted emails) and ssh keys (so no access to my servers). I’m going to start getting access back after publishing this post but will take time. I’ll keep this blog updated.

Some may have noticed that I “disapeared” from the internet for the past 2 months (some may not have noticed, which is fine too :)). I didn’t write on this blog or on my gemlog (which is not unusual from me), but also was away from mastodon, IRC, matrix channels and all other places I usually am… The reason for this silence is that I had to face my worse nightmare, digitally speaking: I lost access to my gpg key, my ssh keys, and a lot of other important (and some less important) data…

How could that happen? Well, a mix of bad luck and over confidence (or lack of paranoia) from me. Let’s recap…

On April 3rd or 4th, I connected as I often do my backup drive where I store a copy of some key files (GPG and SSH keys, my pass password store or a few handful of other files) in order to update the backups with newest version of those files (I did it twice a month). I’m not sure why but this not-so-old drive (5yo), which has been working flawlessly until that specific day, decided to die. Nothing I could do to read it, whatever the OS I tried. I didn’t save the dmesg logs but basically it was not even recognized as a drive and nothing I tried from various online documentation did anything.

I didn’t think too much of it. I mean it was my backup drive for a handful of files, the “real” backups of my system and all my important data are on NAS anyway, managed by borgbackup. So even though that drive was dead, I lost nothing I didn’t have elsewhere.

At this stage, a real paranoid person would have immediatly ordered a new drive to be delivered as fast as possible… But to avoid buying online and use a local store instead, I decided not too. Afterall, my laptop was working flawlessly so I could wait a few days. I had plan during the following weekend, so decided that I would go and buy one the weekend of the 13th.

But bad luck again, on the 9th of April, went to see a friend that was sick after work, leaving my laptop on as I sometimes do. Nothing unusual. I got back home a couple of hours later and when I went to my laptop, the screen was off (as it should), but moving the mouse did nothing and the laptop screen didn’t turn on… Tried a few things, like going to a TTY or using the “magic keys” to force reboot etc… Nothing worked. Well, even though it never happened with this laptop, I knew it could be a “simple” crash and that I was forced to hard reboot the laptop. So long pressed the power button until it turned off.

When pressing the power button again, nothing booted. Lights where indicating that it started, but the screen stayed dark… I’m not going to go though all the tests I’ve done (having a Frame.Work is really helpful to open it easily and remove / change things), but nothing worked.

At first, I thought it was the motherboard that died, so waited the next morning and went to a store to buy an external case for my NVMe drive to at least get my data back with an older laptop. Well… that didn’t work… A bit in despair, I remember that my old laptop that I transformed into my secondary proxmox server also had an NVMe drive. So I unmount the drive and put it in my frame.work laptop… And it booted…

Before that point, I was quite unhappy and sad that the frame.work laptop was “dead”, but I was still hopeful for my data. But when I finally realized what the real issue was, coupled with the death of my backup drive the week before… It was like I was hit by a truck (and having been hit by a car in my youth, I know a bit the feeling^^")…

So here I am, Wednesday late morning, still unable to work and with a very tough news to swallow… Because in less than a week, I lost the only thing that can give me access to all my encrypted data on my NAS (think about personal pictures, documents, etc), as well as the access to all my servers / VM … And all my passwords to all my services and such…

To make things worse, that week was already a very crazy week at work. I had to finish multiple things before the weekend and the deadlines were already tough. Loosing Wednesday morning was not helpful, and I couldn’t loose more time…

So I decided to focus on priorities and things I could do. I went a buy a new NVMe drive, installed archlinux (via endeavourOS) quickly with a very minimal workable configuration, retrieved my work related access and started to get to work to meet the deadlines. It kept me occupied late all week so I tried not to think about the consequences of what happened.

Until Friday night of course. At that point, I was so depressed and tired, I just decided not to touch my laptop or anything linked to “bacardi55”. Even though I have access to mastodon or my matrix client on my phone, I mentally banned them for the time being. The only service I continued using was my RSS reader (as it was configured on my phone and I changed the password from it to be able to use it from my laptop the week after). But even in my RSS reader, I ignored most bloggers and focused on news. Everything else reminded me too much about my situation.

Then, on Monday, I started a long traveling cycle for both work and personal reason… Went to Copenhagen and Paris for work during the week, went to the east of France during the weekend to visit the in-laws and then back to Paris for family reasons. Then I was supposed to go to Belgium for work before going back home in the south of France…

Busy couple of weeks… But it got worse! I believe my fatigue (long week and no vacation for almost 10 months) alongside my depressed mind probably made me weaker and I started getting sick during the weekend at the in-laws. Got some meds, but it just got worse day by day. The day I was supposed to go to Belgium, it became so bad I had to go to the doc again. I canceled the trip and the doc that gave me way more powerful stuff and 2 days of sick leave (was a Thursday anyway). I really don’t remember being that sick for my full adult life…

In the meantime, at the start of my travel, I reached out to the best (on paper at least) company to recover broken hard drives in France. I explained my problem and they estimated the cost to ±1.5k€ (only if they managed to retrieve the data, 0€ otherwise)… I was desperate so said yes and sent the drive to them quickly. I knew it would take time because I took the “no rushed” (to lower cost) option.

I had vacations planed for the first 15 days of May so decided to do nothing before I was back and keep my current “avoid the bacardi55 world” for now. Not taking a laptop or anything, deciding to clear my head from all this. During my vacation, I also decided to not do anything until the recovery results but was mentally preparing myself for bad news.

When I got back, I called the recovery company that said they were still working on it and still trying to recover the disk as what they tried so far didn’t work. I understood that with all the holidays in France at the start of May, it would take a bit longer than expected.

They got back to me this week on Monday (May 27th) to let me know that there was nothing they could do to save the data…

I was confident that they could, but was also preparing my mind that they couldn’t. As they say: « don’t be scared, be prepared for the worse » which is what I usually do. Most of the time, it helps taking very bad news and turning them in a positive manner. But as you read from this post, that didn’t work in April. As for this Monday, while I was still prepared for it, it still shocked me. I already had in mind what I would need to do if they couldn’t fix it and thought I was well prepared… But let’s say Monday night was not the greatest^^.

Anyway, that time I managed to bounce back a lot quicker as I was still kind of expecting those results… But didn’t find the energy to start executing my plan this week.

The first part of the plan is to actually write this post. Both to explain and reassure people that may wonder what happened to me, but also to act on it and start moving on…

Thanks to the deployment workflow I set up in March, I can deploy blog posts without access to my server, I only need to be able to push code on sourcehut. I reseted my password and added a new ssh key to push code there. So if you read this, it means that part worked… FWIW, I can’t deploy new gemlog right now as I don’t have the same automation in place for it, was part of April projects but as you understood, that didn’t happen…

After publishing this post, I’ll start trying to recover access to my servers and associated services. I know some will be easy, some won’t, and I’ll try to share my experience here later on. I should be back on mastodon and IRC/Matrix in the next days if everything goes according to the plan…

Anyway, now I’m feeling better and made my peace with that, even though there might be some data that I lost that will hunt me for the rest of my life, but what can I do about it… This blog post has been too long anyway… Just thought I explained the 2 months hiatus…

Now I also need to think about my backup strategy again. I will probably still use borg for the full backup, but it is clear now that some files like my gpg keys and other needs to be saved not on one another drive, but most probably 2 or 3… So that never happens to me again.

Well, « leave and learn » as they say…