Advanced Backups for VPOP3
Posted by Paul Smith on 05 November 2013 05:36 PM
As everyone knows, backups are very important, especially with an important piece of software such as a mail server.
With VPOP3 v5 and later, VPOP3 will automatically make a backup of its database every day. The database in v5 and later contains all the settings, users, messages, etc, so is essentially a full backup and can be used to restore a server in the same state as it was at the time of the backup.
The database backup settings in VPOP3 are set in Settings -> Database -> Backup.
VPOP3 uses the standard PostgreSQL backup program called pg_dump to create the backups. There are altnerative ways to backup a PostgreSQL database, but this is often the most appropriate and safest for people who aren’t database experts.
The default backup behaviour is to make a full backup into the VPOP3 installation directory every day, and cycle on a weekly basis. You will probably have noticed big files called ‘DBBACK-n.DMP’ in the VPOP3 folder. These are the backup files. It can be a good idea to tell VPOP3 to store these files elsewhere, so that (a) they don’t fill up the VPOP3 disk, and (b) if the VPOP3 disk fails, you don’t lose your backups as well. This article in our Wiki tells you how to store the backups elsewhere or change the backup rotation cycle.
The backups can be big, but because hard disk space is cheap nowadays, it is usually worth making these backups. Storing them on a USB hard drive is often a simple solution to disk space issues.
Offsite Backups & Compression
One important thing to note is that, by default, the backups are compressed. Locally this can be a good thing. However, if you are storing the backups offsite using some cloud-based backup solution, it will usually mean that the entire backup has to be uploaded every day, which may not be practical. This is because file compression will usually defeat any ‘delta backup’ system – a single byte change in the original data file may cause the majority of the compressed file to be different.
So, one workaround is to tell VPOP3 not to compress the backups. This is set in the Backup Command Options setting in the database backup settings. The ‘-F c‘ option tells the pg_dump program to create a ‘custom’ format output file which is compressed by default. If you add the options ‘-Z 0‘, this will tell pg_dump not to compress the output file. This will make the backup file a lot bigger (typically about 2-3 times the size), but, because it is not compressed, delta backup systems should be able to detect changes in it much easier, so offsite backup uploads may be much smaller.
If the backup file is too big to be handled in one chunk, you could try changing the backup options to ‘-F d -Z 0‘. This tells pg_dump to put the backup into a directory with one file per table, rather than a single file for the entire backup. This may be even easier for a delta backup system to handle.
Volume Snapshot Service (VSS)
If your third party backup program supports VSS, then you can just backup the entire VPOP3 directory (and all subdirectories) as it is, without needing to backup the DBBACK-n.DMP files. Note that if you do this, the backup acts as if the PC has been powered off without shutting down properly. PostgreSQL will automatically run a database recovery when it restarts after restoring the backup, but it is worth testing this before relying on it.
Note that if your backup program does NOT use VSS, then backing up the entire VPOP3 directory will NOT produce a usable backup. In that case you will need the DBBACK-n.DMP files to be able to restore a working installation.
An alternative backup system advanced users may want to try is to use RDIFF to generate local ‘delta’ backups. RDIFF is a Linux program, but you can get versions ported to Windows – eg from http://personal.hlfslinux.hu/hijaszu/rdiff.html
Change the backup command options to ‘-F c -Z 0′ to create an uncompressed backup
Making a base backup
Now, make an uncompressed base backup. Keep this backup safe as you will need it to be able to recreate later backups from the deltas. For the sake of the following instructions, we’ll assume you’ve called this base backup ‘database.dmp‘
Next, you need to make an RDIFF signature file
rdiff signature database.dmp database.sig
Depending on the size of the base backup, this may take a while. This command will create a file called database.sig which is used for generating deltas in the future (you don’t have to keep the base backup on the server to generate deltas).
Once this is finished, you can move the base backup (database.dmp) to a safe location, and leave the signature file (database.sig) on the server
Making deltas of daily backups
The next day, when you have created a new backup, called ‘database.new’, you need to create a delta file using the previous signature file
rdiff delta database.sig database.new database.delta
Now, you just need to keep the ‘database.delta’ file safe, and the database.new file can be deleted.
You can continue to do this every day. This is like a differential backup where you need the base backup and the latest delta file to be able to recreate the latest backup file. Over time the delta file will grow in size, and you may want to periodically create a new base backup & signature file
VPOP3 has an option to run a command after the backup has completed, so you could use a couple of batch files to do this delta process automatically (you need two batch files because the operation may take longer than the normal timeout, so you need one batch file to launch the second one as a background task)
Batch file 1 (launchdelta.cmd)
start cmd /c dodelta.cmd %1
Batch file 2 (dodelta.cmd)
rdiff delta database.sig %1 %1.delta
Set the ‘command to run after backup’ in VPOP3 to cmd /c launchdelta.cmd %f
Note that the above batch files assume that VPOP3 stores its backups into a location where the VPOP3 service has permission to read/write files – it will not work if the Target File Network Username/Password options are used.
To recreate the latest backup as database.new, take the base backup (database.dmp) and latest delta file (database.delta) and use the command
rdiff patch database.dmp database.delta database.new
(Note that the signature file is not needed to perform this command)
You can then use the database.new file in the normal VPOP3 database restore instructions
Note that the RDIFF delta and patch commands may take a long time depending on the size of the files. Using this option is a compromise to save disk space at the expense of time. For most people, they will want to recover the server as quickly as possible, so a bit of money spent on extra disk storage will be the preferred option.
Read more »