Bulk Delete Proxmox Backups Using Linux Commands

A client was tasked today with removing over 1400 backups from Proxmox, which were generated daily for two guests on a schedule. After a couple of years the backup destination was full and someone went looking for the cause. It is worth noting that the client also has a PBS backup system in place, also backing up the same guests along with all the others, so these specific backups were unnecessary.

Proxmox is great

Proxmox is a software framework that manages Linux QEMU/LXC virtualisation and handles things like cluster configuration synchronisation, automates configuration, provides a really nice Web GUI, a catalogue of command line tools, and a fully features REST API. We love Proxmox, we use it internally, we support customers using it, and we have maximum respect for the guys who develop it.

Having said all this, the WebGUI whilst really useful for monitoring, basic configuration, and simple tasks, is not the best place to handle bulk operations.

Our client had spent the best part of 4 hours manually deleting backups, one at a time using the Web GUI, when, realising his life was literally slipping away, he raised a ticket for help.

Proxmox Backups

Proxmox backup is very simple, and that makes it very flexible and reliable. Proxmox stores backups in the following locations

Backups are stored as 3 files, the actual compressed backup with a .tar.zst extension (if you used compression), a log file with .log and a notes file with .tar.zst.notes. The actual backup file is a genuine TAR archive file, and you can manually extract it if needed, and manually restore, but today we're simply dealing with the thousands of backups to be removed.

It is important to mention that PBS is not the same, it is similar but there is more metadata and features, if you want to manually restore from a PBS server then it won't be covered here.

Command Line

We're going to be using the Linux shell for this operation, and that's the perfect place to handle this quickly and easily. Since we know where the backups are and we know they are just files, removing them should be very easy.

First, we need to open a shell, this can be done in the Proxmox GUI by selecting any node (or the node for local backup) and then clicking "Shell" top right. You can of course use SSH with your favourite terminal tool, or just use Linux SSH.

Now we need to find the folder where the backups are stored. In our example case here, the backups were stored in a NFS share called PROXBACKUP. Therefore:

ls -1h /mnt/pve/PROXBACKUP/dump > ~/prune.txt

Will generate a list of filenames in that folder and put those into a file named prune.txt in our home directory. If we wanted to be more specific we could do something like...

ls -1h /mnt/pve/PROXBACKUP/dump/vmdump-lxc-107* > ~/prune.txt

To only select backups for ID 107. Now we should open that file and check it is what we expect it to be...

more ~/prune.txt

And we see from the output that indeed we have a list of files:

vzdump-lxc-107-2024_04_18-04_10_31.log
vzdump-lxc-107-2024_04_18-04_10_31.tar.zst
vzdump-lxc-107-2024_04_18-04_10_31.tar.zst.notes
vzdump-lxc-107-2024_04_19-04_17_19.log
vzdump-lxc-107-2024_04_19-04_17_19.tar.zst
vzdump-lxc-107-2024_04_19-04_17_19.tar.zst.notes

Nice. Ok, let's convert that into a script to remove these files, and we're going to use sed (Stream Editor) for this. We could use awk but sed is easier really.

sed 's/^/rm -f \/mnt\/pve\/PROXBACKUP\/dump\//' ~/prune.txt > ~/prune.sh

So here we're adding "rm -f /mnt/pve/PROXYBACKUP/dump/" to the front of each line. This will generate a file in prune.sh that looks like...

rm -f /mnt/pve/PROXBACKUP/dump/vzdump-lxc-107-2024_04_18-04_10_31.log
rm -f /mnt/pve/PROXBACKUP/dump/vzdump-lxc-107-2024_04_18-04_10_31.tar.zst
rm -f /mnt/pve/PROXBACKUP/dump/vzdump-lxc-107-2024_04_18-04_10_31.tar.zst.notes
rm -f /mnt/pve/PROXBACKUP/dump/vzdump-lxc-107-2024_04_19-04_17_19.log
rm -f /mnt/pve/PROXBACKUP/dump/vzdump-lxc-107-2024_04_19-04_17_19.tar.zst
rm -f /mnt/pve/PROXBACKUP/dump/vzdump-lxc-107-2024_04_19-04_17_19.tar.zst.notes

We should of course inspect this file in its entirety and make sure it is exactly what we want, there's no going back if we mess this up. "more" allows you to page through it by pressing space, or line by line pressing enter.

Once we're sure that's what we want, we need to run it and this takes two steps because by default in Linux, files are not executable.

chmod +x ~/prune.sh
bash -x ./prune.sh

This will execute our script, removing each file one by one. By using bash -x we're able to see each line executed, otherwise we'd have no output until the script was finished.

Alternatives

Like everything in Linux, there are alternative solutions, but I want to explain why we chose to export a list of files into a file, add commands to the front then run the script.

We could have simply run something like

rm -f /mnt/pve/PROXBACKUP/dump/vzdump*.tar*

And that would have indeed removed all the backups, without confirmation or check and you don't know what was deleted. We could use...

find /mnt/pve/PROXBACKUP/dump/ -type f -name 'vzdump*.tar*' -delete

Also very effective and would do the same job, but again no way to know really what it's deleting without running it first omitting -delete and then we should save that output somewhere for a record, etc.

In a production environment where your every action could cause a crisis and you need to have a clear record of what you did, when and why, it is important to give yourself as much protection as possible, and this is why we did it the way we did. We had multiple places to inspect the action

• We inspected the prune.txt file to make sure it included the files we wanted to remove.
• We sent a copy of prune.txt to the client and they confirmed these were the files they wanted removing.
• We inspected the bash script for accuracy before running it.
• We ran the bash script with bash -x to capture a log of its actions, which we the documented.

Summary

I hope someone finds that useful, and it saves them a stack of time. There are many ways to perform most tasks, some will be easier, some longer, some fraught with danger and others just monotonous. Choosing the right path is by far the most important decision.

It would be remiss of me not to mention that we have 15 years of Proxmox experience, and we're a pay-as-you-go provider, with no long contracts or rigid terms, just pay for the time you use - be that fully managed, part/hybrid or simply call us when it breaks.