We will shortly be doing an emergency restart on both Zimbra servers due to a bug that is causing performance and disk space issues. Expected outage for each server is about 2 minutes. Please accept our apologies for any inconvenience caused.

At approx 17:15 GMT today, we experienced a brief outage across our mail platforms. This was caused by an attempted SQL injection attack against our old control panel. The attack failed to access our systems, but did cause a very large SQL query to block the SQL servers and stop them responding to requests for a period of about 5 minutes. Needless to say, we’ve identified the point of attack, and fixed the code to prevent a repeat occurrence. Please accept our apologies for any inconvenience caused.

As some of you may have surmised, the patch that we applied on Thursday night did not fix the main problem we’ve been having with Zimbra. I have reported this to Zimbra, and they have come up with another patch for us, which I will apply to both servers at approx 11pm tonight. Once again, there will be a short outage while this patch is applied of the order of 5 minutes.

Zimbra have now provided us with a patch that should solve a lot of the problems that we have been recently experiencing. I am planning to install this tonight at 11pm, starting with zimbra001, and then zimbra002 just afterwards. This will cause a short period of downtime for each server – probably around 5 minutes.

Zimbra have now confirmed that both issues that we are being affected by are caused by the same bug, and they are expecting to have a minor release which fixes this next week. Once this is available, we will apply it as soon as possible. In the meantime, we will continue to monitor things very closely.

Since upgrading to Zimbra 7, we have been experiencing one or two bugs, which have affected the normal high reliability levels of the servers. We are currently waiting for the next minor release to appear, which will hopefully fix these bugs. In the meantime, we are doing what we can to mitigate the effects, and respond quickly when there are problems. The main two problems are:

1) Sometimes a mailbox becomes deadlocked, which prevents access to it. To fix this, we need to restart a process on the server. We currently have no way of detecting if this has happened, so we need to hear from you if it happens so we can deal with it.

2) A more serious problem has now struck zimbra002 three times. Twice on Saturday and once last night. This causes the server to queue messages internally rather than delivering them to mailboxes. Because to all intents an purposes the server is still working, this does not trigger our pagers, so we remain unaware of the problem until it is reported to us, which has resulted in some unacceptably long outages. To fix this, I’ve written a script that will specifically check for this condition, so we should get early pager alerts and hopefully be able to fix it before you or your customers notice.

In the meantime, please accept our apologies for any inconvenience caused.

We will be loading a minor upgrade onto our Zimbra servers that fixes a few bugs tomorrow (Thursday 14th) night at approx 11pm. We anticipate a 5-10 minute downtime on each server during the upgrade. Please accept our apologies for any inconvenience caused.

The upgrade to server zimbra001 has now been completed, and it is running Zimbra v7. Apologies for any inconvenience caused by the outage tonight.