Background log writer
Posted by Paul Smith on 10 July 2014 12:50 PM
The diagnostic log files which VPOP3 generates can add a lot of extra disk load to a VPOP3 server. For this reason, VPOP3 doesn’t usually write data to the log files as soon as the event happens. The log file entries are held in memory until the “background log file writer” gets around to it, and writes them to the actual disk files. This generally works well, however if the disk is exceptionally slow (e.g. failing disks, or RAID5 SATA drives on a busy system) then it has been known to lead to out-of-memory errors.
The reason that this background writing system is necessary for VPOP3, where other servers may write log files immediately without serious consequences, is because VPOP3 can write to many different log files to make reading them easier. Depending on the settings, it can also write a lot of data to the log files – especially if session logs are enabled. This can cause a lot of disk seek activity, which slows disk access down across the whole PC.
In versions of VPOP3 prior to v6.7, the behaviour of the background log file writer was fixed in the software, but in v6.7, we added configuration options to customize the behaviour. This can be used to decrease disk load further, or to make the log files be refreshed more frequently.
You can customise and monitor the behaviour of the log file writer by going to Settings -> Diagnostics -> Log File Writer
If you turn off ‘Enable logging to file’, then VPOP3 will just discard all logging entries. This is not recommended for normal use, but may be useful as a diagnostic tool to see if performance problems are being caused by VPOP3’s logging.
The settings on this page tell VPOP3 how to change how often to write data to the log files, and how to automatically adjust its behaviour depending on the current load.
VPOP3 writes log file data in ‘bursts’, rather than as the log entries are generated. The main thing you can configure is how long to wait between these bursts. The first 4 settings let you configure this.
Min time between log writer bursts and Max time between log writer bursts set limits for the time between the bursts. In the screenshot above, this means that the delay between bursts can be between 0 and 10 seconds.
The If lines written < X then increase time between bursts and If lines written > y then decrease time between bursts settings tell VPOP3 how to adjust the time depending on load. They mean that if there is lots of data to log, VPOP3 will wait less time between writes, but if there isn’t much data then it will increase the time between the writes, to avoid writing lots of small chunks of data.
The Only write to disk if at least x lines waiting to be written setting prevents VPOP3 writing small bits of data. On a hard disk, each write will probably incur a delay – eg because of disk latency (waiting for the disk to rotate to the correct position) or seek time (moving the write head to the correct track on the disk), so writing small bits of data unnecessarily is not a good idea.
The Always write if oldest line is over x ms old setting prevents stale data sitting in memory for a long time. This overrides the previous setting. For instance, if VPOP3 is just logging errors, it may just generate one log entry every hour – if you had told VPOP3 not to write to disk if there are less than 10 lines to log, there could be a 10 hour delay before the first error is logged, which would not be very helpful, so you can limit the maximum time to delay.
The Current time between log writer bursts value shows how long VPOP3 is currently waiting between bursts of writing to disk.
The Log file stats section shows the current statistics. For instance, how many writes have taken place in the last 10 minutes, and how many lines & bytes were written in that time, and so on. This can help you see if certain log files may be causing a lot of disk access.
The pop3clt.log, smtpclt.log, smtpsvr.log, smtpclt.log, imapsvr.log, nntpsvr.log, nntpclt.log files are Session Logs which can be enabled/disabled on the Session Logs tab. If you have lots of writes to the VPOP3.log file, you may want to decrease the logging level on the General tab.
An advanced, but very effective way of decreasing disk load is to set up a RAM disk (e.g. http://en.wikipedia.org/wiki/List_of_RAM_drive_software#Microsoft_Windows) and have VPOP3 write its log files to that. Since you will rarely want to access diagnostic log files after a computer reboot, this works well. If you have a RAM disk, then you can change the background log file writer configuration so that it writes to the log files more frequently, as disk seek times are not an issue.
Using an SSD for log files is not recommended. It will work fine, but SSDs have limited write cycles, so using them for write-heavy tasks such as log files will reduce the lifetime of the SSD.
Read more »
VPOP3 v6.3 – GeoIP access restrictions
Posted by Paul Smith on 10 September 2013 01:32 PM
One of the new features in VPOP3 v6.3 was the support for ‘GeoIP’ in the Access Restrictions settings. This lets you specify that only IP addresses from certain countries can access VPOP3 services. Many people use this type of restriction to prevent access from countries commonly used by attackers.
In fact, the VPOP3 GeoIP facility isn’t limited to GeoIP data. Essentially it has a database which matches IP address ranges to a ‘tag’ and then you can check against the tags. A common use would be GeoIP – the tags would be country codes – but it could be used for other things as well. As the database is queried live, you could even update the database dynamically and have VPOP3 update its access restrictions automatically.
However, a common use is GeoIP, so that is what this article will describe.
Installing GeoIP data
The first thing is that VPOP3 does not include any GeoIP data. This is partly due to licensing restrictions of such data, and also because it often changes, and there are different sources.
So, to use the GeoIP facility, you need to obtain GeoIP data and import it into VPOP3.
One free GeoIP database source is [[http://dev.maxmind.com/geoip/legacy/geolite/|GeoLite from Maxmind]], so that is what this article will use. You need to decide what sort of data you will be checking against. For this example we will use the GeoLite Country database. You can also check against cities, or ASNs (essentially checking against Internet providers’ ranges).
For these steps you need the CSV database, so to download the GeoLite Country database, go to http://dev.maxmind.com/geoip/legacy/geolite/. At the bottom is a Downloads section, click on the Download link in the “CSV/zip” column for GeoLine Country. This file is about 1MB. Extract the CSV file from there to a temporary location (eg c:\temp).
If you want, you can open up this file to see what it contains. It will easily open in a text editor, or a spreadsheet program. Each line contains 6 columns. The first two are an IP address range – eg “126.96.36.199” to “188.8.131.52”. The second two columns are the same data but as a 32 bit integer rather than dotted IP addresses (it isn’t significant how these are calculated as we are going to ignore these). The fifth column contains the ISO country code for the country in question, eg ‘AU’ for Australia, or ‘GB’ for the United Kingdom. The sixth (last) column contains the name of the country/region, eg ‘Germany’ or ‘Russian Federation’. For our purposes we’ll be using the first two and fifth columns so we can link the IP address range to the country code.
What we need to do is import this data into the VPOP3 database.
We have a tool you can download to import the data. It’s called VPOP3GeoImport and it is run from a command prompt. To use it, download the file and extract into the VPOP3 installation directory. Then, from a command prompt run it as:
VPOP3GeoImport [options] importcsv <filename> 125
Normally you won’t need any options, but if you run “VPOP3GeoImport -?”, you will get a list of options available. (The ‘125’ at the end tells the program which columns to import).
This program will delete all existing data in the GeoIP database within VPOP3. Note that you do not need to stop VPOP3 while you perform this operation.
Using the GeoIP data
To use the GeoIP data, in the relevant VPOP3 Access Restrictions settings, select the GeoIP Lookup type in your access restriction, and enter the desired ‘tags’ in the Address box. In the case of the above country GeoIP data, the tags would be ISO country codes, eg US, GB, etc. To specify multiple tags to look for, you can separate them with commas or semicolons.
The GeoIP data is stored in a database table called geoipv4. This has 4 columns:
If multiple entries match the searched IP address, then each result will be checked against the specified access restrictions.
Read more »
New feature in v6.1 – Store and Forward
Posted by Paul Smith on 13 June 2013 08:28 AM
LAN forwarding has been a feature of VPOP3 for many years.
With LAN Forwarding, VPOP3 will forward messages on to another mail server using SMTP. This has two common uses:
Using VPOP3 in either of these situations can improve reliability as mail is still accepted even if the main mail server is down, however, the messages which VPOP3 has queued for forwarding to the main server are not available for viewing, so mail can still not be accessed until the main server is back working.
Store and Forward
In v6.1 we added a new feature to VPOP3 Enterprise called “Store and Forward”. In this case, VPOP3 will both store the mail in a local VPOP3 mailbox, and also forward it to another mail server.
If you use this feature, then users will be able to log into their VPOP3 account to view their mail, just as if this VPOP3 was their normal mail server, but if/when the main mail server comes back online, the mail will be forwarded onto that as with normal LAN forwarding.
However, it’s cleverer than simply copying messages to two places (which could easily be done with older versions of VPOP3):
Configuring Store and Forward
To configure Store and Forward, you first need to create all your users in your VPOP3 server, by going to the Users page, and pressing New or Bulk Add Users to add your users.
Next, edit the user by double-clicking on them, and go to the Advanced tab. You will find two boxes in there called Store and Fwd email target and Store and Fwd target server. In the email target box enter the full email address on the target mail server. In the target server box, enter the address of the target mail server (see notes below).
Finally, remove any previous LAN Forwarding configuration. The usual places this may be set is in Local Mail -> LAN Forwarding -> Configuration or in the Routing tab of the users’ settings (check for entries beginning with SMTP: in the Assistant or Forwarding settings).
When entering the Store and Fwd Target Server address:
Read more »
Pruning old messages automatically
Posted by Paul Smith on 10 June 2013 01:58 PM
In VPOP3 Enterprise 6.2 and later you can tell VPOP3 to automatically delete old messages, based on specified criteria.
You can either do this globally, for all users, or just for specific users.
To edit the global criteria (rules), go to Settings -> Database -> Message Store, and look at the Global Prune Rules section. To edit the criteria for a specific user, edit the user, and go to the Prune Rules tab.
Both types of rules are configured similarly, so we will just describe the user rules here
How to add/edit a rule
To add or edit a “prune rule”, press the Add Rule button. This will add a new rule to the table, with a set of default settings. Don’t worry, this rule doesn’t take effect until you press the Submit button.
Now, you can simply alter the settings in the columns in the table, by clicking them. The settings define which messages should be deleted, not what should happen to messages.
The folder column defines which folder(s) the prune rule applies to. Initially this is set to Inbox, but you can click on the folder name to set it to any folder name you wish. Alternatively, you can set it to * to match all the folders for this user (note that you cannot use wildcards in conjunction with other characters (eg “Inbox/*” will not work as you expect).
The age column defines how old messages are before they will be deleted. This is set in days, and is based from when VPOP3 first saw the message. It is not based on when the message was written or sent. If the Age is set to 365, then VPOP3 will automatically delete any messages which were received by VPOP3 over 365 days ago.
The size column defines how big messages should be before they are deleted. This can be useful to only delete big messages, or to delete big messages sooner than smaller messages. The default is for no minimum size (all sized messages), but you can set it by clicking in that column.
The read column indicates whether this rule will apply to read messages, unread messages or either. This could be useful if you only want to delete messages that have been marked as read.
The flagged column indicates whether this rule will apply to flagged (starred) messages, unflagged messages, or either. This could be useful if you want users to be able to ‘star’ important messages so they don’t get deleted automatically.
If the deleted column is checked, then VPOP3 will only apply this rule to messages which have the IMAP4 ‘Deleted’ flag set. If this column is not checked, then the rule will apply to all messages. This can be useful if your users do not purge (or ‘expunge’) folders regularly
If the spam column is checked, then VPOP3 will apply this rule to messages marked as spam by VPOP3. This can be useful if you are not using the VPOP3 quarantine facility, but are, instead, having VPOP3 deliver the messages to the user, and are using VPOP3 Message Rules or email client rules to put spam messages into an IMAP4 folder.
Read more »
VPOP3 v6.0 – Bandwidth control
Posted by Paul Smith on 03 April 2013 10:52 AM
One of the new features in VPOP3 Enterprise v6 is enhanced bandwidth control. This allows you to restrict the speed of VPOP3 connections for different users, services and IP addresses.
VPOP3 Basic has basic bandwidth control, so this article may still be useful, but VPOP3 Basic does not support ‘bandwidth pools’ or scripting.
Because of the flexibility it can get a bit complex, so this article will attempt to explain how the feature can be used.
If you go to the VPOP3 settings, and go to Services and select a service – we’ll choose the IMAP4 service, you’ll see a setting on the General tab called Bandwidth Throttling. The default setting for this is “No Limit”.
In VPOP3 Basic, you will only have two options: “No Limit” and “Custom”, in VPOP3 Enterprise you will have those two options, plus 1000 ‘Bandwidth Pool’ options.
Let’s look at the two basic settings first
The “No Limit” option, as the name suggests, means that there is no bandwidth limiting.
The “Custom” option lets you specify a maximum speed (in bytes/second) for each connection using this service. So, if you set the custom value for the IMAP4 service to 100000, then each connection using the IMAP4 service will be limited to 100,000 bytes per second (800,000 bits per second). If the server is handling 10 IMAP4 connections at once, then each connection is limited to 100,000 bytes per second, so the total throughput could be up to 1,000,000 bytes per second.
For many bandwidth control operations this has limited use, because you may want to throttle the IMAP4 service to only use so much bandwidth regardless of how many connections there are. This is where Bandwidth Pools come into play.
To have access to Bandwidth Pools, you need to have VPOP3 Enterprise.
Bandwidth pools are configured in Settings -> Misc Settings -> Bandwidth Pools. VPOP3 supports 1000 bandwidth pools. By default, they are all configured to allow unlimited bandwidth.
You can easily assign a bandwidth limit to a pool by double-clicking on the ‘Allowed Bandwidth’ column for the pool you want to edit, and typing in the new bandwidth limit (in bytes per second). You can also assign a name to the bandwidth pool for your future reference.
Now, you could go to the IMAP4 service settings, and choose ‘Bandwidth Pool 1′ to be the bandwidth limit for the IMAP4 service. When you have done that, then all the connections using that service will share the bandwidth from the chosen bandwidth pool.
If you wish, you could also choose ‘Bandwidth Pool 1′ to be the bandwidth limit for the POP3 service as well. Now, the bandwidth pool allowance is shared among all POP3 and IMAP4 connections.
If you wish, you can create ‘nested pools’. To do this, use negative numbers as the pool bandwidth limit where the negative number indicates the ‘parent’ pool. For instance, you could create settings as below. In this case, the “POP3″ and “IMAP4″ pools both share the “VPOP3″ pool limit. This is achieved by looking at the ID of the ‘VPOP3′ pool (3), and making it negative, and setting that as the limit for the ‘child’ pools.
The POP3 and IMAP4 pools are not copies of the VPOP3 pool, but share the pool, so all users of the POP3 and IMAP4 pools would be limited to 100,000 bytes per second in total.
In itself this does not give you any functionality you would not otherwise have (you could simply set both the POP3 and IMAP4 services to use the “VPOP3″ pool), but it does mean that in the future, you could change the limits from the Bandwidth pool settings without having to edit the services individually. This becomes even more useful when you come to use bandwidth scripting,
The real power of the bandwidth limits becomes apparent when you can use scripting (only in VPOP3 Enterprise) using the Lua programming language.
The reference guide for the bandwidth scripting is on our Wiki, but here are some examples and ideas. The reason we have used scripting rather than discrete settings is that scripting would allow the feature to be used in ways we may not have considered yet.
Every time a new connection starts to a VPOP3 service, or the ‘state’ changes (eg someone logs in), then a Lua function is called. This function is called ‘GetBandwidth’ and is stored in the ‘bandwidth.lua’ file in the VPOP3 directory (you can create/edit this file using a simple text editor such as Notepad). This function is passed several parameters, such as details of which service the connection is for, user details (if any), client IP address, and so on. Then, the function can return a value to indicate the bandwidth limit to be used:
As you can probably imagine by now, there are many ways this can be used. For instance:
External IP addresses are limited
-- pool '1' is set to the limited pool -- This function either returns unlimited for local users (192.168.x.y), or '-1' to assign external users to pool 1
if string.find(params["clientip"], "^192%.168%.") then return 0; -- unlimited else return -1; -- use pool 1 end; end;
Most users are limited, but the boss isn’t
-- pool '1' is set to the limited pool -- This function either returns unlimited for connections from 'theboss', or '-1' to assign other users to the limited pool
if params["username"] == "theboss" then return 0; -- unlimited else return -1; -- use pool 1 end; end;
External IP addresses are limited to different pools depending on time of day. The boss is always unlimited
-- pool '1' is set to the daytime limited pool -- pool '2' is the evening limited pool -- This function either returns unlimited for local users (192.168.x.y), or the boss, or '-1' or '-2' to assign external users to the relevant pool
if string.find(params["clientip"], "^192%.168%.") or params["username"] == "theboss" then return 0; -- unlimited else -- limited users t = os.date("*t"); if (t.hour >= 18) or (t.hour == 17 and t.min >= 30) or (t.hour < 8) or (t.wday == 6) or (t.wday == 0) then return -2; -- use pool 2 after 5:30pm or before 8am or at weekends else return -1; -- use pool 1 at other times of day end;
Read more »