Regaining access to Proxmox VMs and Containers

Introduction

Following my data loss crisis, I have to regain access to my servers, VMs and containers… I thought that writing about the process may help other or my future self… So today, let’s talk about how I got back my access to my proxmox hosts, VMs and Containers.

Proxmox hosts

My proxmox cluster is composed of 2 physical machines. The good news is that I remembered the root password of the primary server as for once I didn’t used a password generated by pass. But to change the host password if you have physical access to the machine would be the same as what I had to do to retrieve access to the VMs. So if that’s your case, just read the paragraph about retrieving access to proxmox VMs.

Well, I had the password, but no more ssh access. Proxmox makes it really easy by having a built-in web console access as long as you can connect to the web UI. Once logged-in to the web UI, I opened the shell that was logged-in as root.

All I did then was changing the ssh configuration to allow password authentication, then connect via ssh (as root) with the password, change the content of the ~/.ssh/authorized_keys files to add my public key. And then remove the ability to connect via ssh via password.

Proxmox automated the population of that ssh key to the other server of the cluster which was nice. But I still didn’t have the password for the second server, but as they are connected as a cluster, I also had a shell access as root via the web console, so it was as easy as clicking on that shell button, and run passwd to change the root password. No need to change the ssh config to add the key, the key was already there thanks to proxmox.

At this point, my access to proxmox hosts were fully retrieved. Now for the VMs and containers.

Proxmox VMs

To retrieve VMs access, it is like retrieving root access to a physical machine. First, reboot the VM via the proxmox web UI. Then go to the console of that VM directly. You should see the VM start and then display the grub start screen:

On this grub screen, type e to edit grub options, you should see a screen like this:

Change the line starting with linux /boot/… and replace ro quiet with rw init=/bin/bash as in the image:

Use ctrl+x to save and boot the VM. Changes are temporary for this start, so after a reboot you don’t have to change back to original values, it will start as before.

It should boot as root directly, and thanks to the rw added before the init, the drive is already mounted and can be edited. It means now all you need is to run passwd to change the root password and/or passwd <user> to change the password of a specific user.

Once done, stop the VM with reboot -f.

And voilà, you know have your user password again.

Now the next thing is to retrieve ssh access, for that I followed the same process as for the host via the web console, login in via the new password, changing the ssh configuration, pushing my public key and change the ssh config back. Easy peasy :).

Proxmox container

For Containers, it is even simpler thanks to proxmox. All you need to do is connect to the host (either via ssh or via web console) and run:

pct enter <container ID>

With <container ID> being the ID of the container visible on the left panel.

Then, as before, a simple passwd and/or passwd <user> will allow you to change the root/user password(s). And the same process as above to retrieve ssh access.

Conclusion

While I haven’t gone through each VMs / containers yet (I have many and didn’t have much time today), it is good to know that I can get access to all of them. I may rebuilt a few as some VMs should be containers so I may use this situation to change that.

Retrieving access to VMs and containers was simple enough, I know that retrieving access to some of the services running on these VMs/Containers will be harder… I do think that some may need to be deleted and redeployed, but that’s for another day.

I also retrieved access to 2 raspberry pi I have on my network, but that’s for another post as this was a bit different. I still need to check if I can retrieve access to my digital ocean droplets, but this will be a task for tomorrow. Then the real challenge will be retrieving access to all services…

But as I said in my previous post, one step at the time!