Advanced Backups for VPOP3
Posted by Paul Smith on 05 November 2013 05:36 PM
As everyone knows, backups are very important, especially with an important piece of software such as a mail server.
With VPOP3 v5 and later, VPOP3 will automatically make a backup of its database every day. The database in v5 and later contains all the settings, users, messages, etc, so is essentially a full backup and can be used to restore a server in the same state as it was at the time of the backup.
The database backup settings in VPOP3 are set in Settings -> Database -> Backup.
VPOP3 uses the standard PostgreSQL backup program called pg_dump to create the backups. There are altnerative ways to backup a PostgreSQL database, but this is often the most appropriate and safest for people who aren’t database experts.
The default backup behaviour is to make a full backup into the VPOP3 installation directory every day, and cycle on a weekly basis. You will probably have noticed big files called ‘DBBACK-n.DMP’ in the VPOP3 folder. These are the backup files. It can be a good idea to tell VPOP3 to store these files elsewhere, so that (a) they don’t fill up the VPOP3 disk, and (b) if the VPOP3 disk fails, you don’t lose your backups as well. This article in our Wiki tells you how to store the backups elsewhere or change the backup rotation cycle.
The backups can be big, but because hard disk space is cheap nowadays, it is usually worth making these backups. Storing them on a USB hard drive is often a simple solution to disk space issues.
Offsite Backups & Compression
One important thing to note is that, by default, the backups are compressed. Locally this can be a good thing. However, if you are storing the backups offsite using some cloud-based backup solution, it will usually mean that the entire backup has to be uploaded every day, which may not be practical. This is because file compression will usually defeat any ‘delta backup’ system – a single byte change in the original data file may cause the majority of the compressed file to be different.
So, one workaround is to tell VPOP3 not to compress the backups. This is set in the Backup Command Options setting in the database backup settings. The ‘-F c‘ option tells the pg_dump program to create a ‘custom’ format output file which is compressed by default. If you add the options ‘-Z 0‘, this will tell pg_dump not to compress the output file. This will make the backup file a lot bigger (typically about 2-3 times the size), but, because it is not compressed, delta backup systems should be able to detect changes in it much easier, so offsite backup uploads may be much smaller.
If the backup file is too big to be handled in one chunk, you could try changing the backup options to ‘-F d -Z 0‘. This tells pg_dump to put the backup into a directory with one file per table, rather than a single file for the entire backup. This may be even easier for a delta backup system to handle.
Volume Snapshot Service (VSS)
If your third party backup program supports VSS, then you can just backup the entire VPOP3 directory (and all subdirectories) as it is, without needing to backup the DBBACK-n.DMP files. Note that if you do this, the backup acts as if the PC has been powered off without shutting down properly. PostgreSQL will automatically run a database recovery when it restarts after restoring the backup, but it is worth testing this before relying on it.
Note that if your backup program does NOT use VSS, then backing up the entire VPOP3 directory will NOT produce a usable backup. In that case you will need the DBBACK-n.DMP files to be able to restore a working installation.
An alternative backup system advanced users may want to try is to use RDIFF to generate local ‘delta’ backups. RDIFF is a Linux program, but you can get versions ported to Windows – eg from http://personal.hlfslinux.hu/hijaszu/rdiff.html
Change the backup command options to ‘-F c -Z 0′ to create an uncompressed backup
Making a base backup
Now, make an uncompressed base backup. Keep this backup safe as you will need it to be able to recreate later backups from the deltas. For the sake of the following instructions, we’ll assume you’ve called this base backup ‘database.dmp‘
Next, you need to make an RDIFF signature file
rdiff signature database.dmp database.sig
Depending on the size of the base backup, this may take a while. This command will create a file called database.sig which is used for generating deltas in the future (you don’t have to keep the base backup on the server to generate deltas).
Once this is finished, you can move the base backup (database.dmp) to a safe location, and leave the signature file (database.sig) on the server
Making deltas of daily backups
The next day, when you have created a new backup, called ‘database.new’, you need to create a delta file using the previous signature file
rdiff delta database.sig database.new database.delta
Now, you just need to keep the ‘database.delta’ file safe, and the database.new file can be deleted.
You can continue to do this every day. This is like a differential backup where you need the base backup and the latest delta file to be able to recreate the latest backup file. Over time the delta file will grow in size, and you may want to periodically create a new base backup & signature file
VPOP3 has an option to run a command after the backup has completed, so you could use a couple of batch files to do this delta process automatically (you need two batch files because the operation may take longer than the normal timeout, so you need one batch file to launch the second one as a background task)
Batch file 1 (launchdelta.cmd)
start cmd /c dodelta.cmd %1
Batch file 2 (dodelta.cmd)
rdiff delta database.sig %1 %1.delta
Set the ‘command to run after backup’ in VPOP3 to cmd /c launchdelta.cmd %f
Note that the above batch files assume that VPOP3 stores its backups into a location where the VPOP3 service has permission to read/write files – it will not work if the Target File Network Username/Password options are used.
To recreate the latest backup as database.new, take the base backup (database.dmp) and latest delta file (database.delta) and use the command
rdiff patch database.dmp database.delta database.new
(Note that the signature file is not needed to perform this command)
You can then use the database.new file in the normal VPOP3 database restore instructions
Note that the RDIFF delta and patch commands may take a long time depending on the size of the files. Using this option is a compromise to save disk space at the expense of time. For most people, they will want to recover the server as quickly as possible, so a bit of money spent on extra disk storage will be the preferred option.
Read more »
Pruning old messages automatically
Posted by Paul Smith on 10 June 2013 01:58 PM
In VPOP3 Enterprise 6.2 and later you can tell VPOP3 to automatically delete old messages, based on specified criteria.
You can either do this globally, for all users, or just for specific users.
To edit the global criteria (rules), go to Settings -> Database -> Message Store, and look at the Global Prune Rules section. To edit the criteria for a specific user, edit the user, and go to the Prune Rules tab.
Both types of rules are configured similarly, so we will just describe the user rules here
How to add/edit a rule
To add or edit a “prune rule”, press the Add Rule button. This will add a new rule to the table, with a set of default settings. Don’t worry, this rule doesn’t take effect until you press the Submit button.
Now, you can simply alter the settings in the columns in the table, by clicking them. The settings define which messages should be deleted, not what should happen to messages.
The folder column defines which folder(s) the prune rule applies to. Initially this is set to Inbox, but you can click on the folder name to set it to any folder name you wish. Alternatively, you can set it to * to match all the folders for this user (note that you cannot use wildcards in conjunction with other characters (eg “Inbox/*” will not work as you expect).
The age column defines how old messages are before they will be deleted. This is set in days, and is based from when VPOP3 first saw the message. It is not based on when the message was written or sent. If the Age is set to 365, then VPOP3 will automatically delete any messages which were received by VPOP3 over 365 days ago.
The size column defines how big messages should be before they are deleted. This can be useful to only delete big messages, or to delete big messages sooner than smaller messages. The default is for no minimum size (all sized messages), but you can set it by clicking in that column.
The read column indicates whether this rule will apply to read messages, unread messages or either. This could be useful if you only want to delete messages that have been marked as read.
The flagged column indicates whether this rule will apply to flagged (starred) messages, unflagged messages, or either. This could be useful if you want users to be able to ‘star’ important messages so they don’t get deleted automatically.
If the deleted column is checked, then VPOP3 will only apply this rule to messages which have the IMAP4 ‘Deleted’ flag set. If this column is not checked, then the rule will apply to all messages. This can be useful if your users do not purge (or ‘expunge’) folders regularly
If the spam column is checked, then VPOP3 will apply this rule to messages marked as spam by VPOP3. This can be useful if you are not using the VPOP3 quarantine facility, but are, instead, having VPOP3 deliver the messages to the user, and are using VPOP3 Message Rules or email client rules to put spam messages into an IMAP4 folder.
Read more »