Service Alerts & Newsletters
In the event of a serious problem, we send out advisories and updates using this Service Alert system. If you would like to be made aware of either upcoming planned maintenance or serious issues, you can subscribe to our alerts list here.
We occasionally send out a newsletter covering company news, new products and features and case studies. If you would like to receive this, please add your email address below.
As part of our commitment to the General Data Protection Regulation (GDPR) you need to explicitly register to receive notifications from us.
Recent Alerts
| 09/11/18 15:04 | zimbra002 Explanation | Closed |
| I believe an explanation is in order about the zimbra002 downtime this morning, so please bear with me while I try and explain. The Zimbra servers have a backup mechanism that can only back up to a mounted drive, so either a local drive or an NFS mount. Every night they do an incremental backup and once a week they do a full backup. The NFS server to which zimbra002 and zimbra003 (which never seems to be affected in the same way) are backing up to crashed in the night. It crashed in such a way that it was still responding to pings, but not NFS calls, so this wasn't picked up by our monitoring systems. When an NFS mount goes away, a Linux server treats this as a very bad thing, and basically waits around for it to come back again. This tends to affect the process that is attempting to access the mounted drive, and other processes normally continue unaffected. Because Zimbra is a monolithic piece of Java code, the backup process causes it to misbehave. However, it still responds on all the ports that we check with our monitoring systems - the problems occur after logging in, so once again not picked up by our monitoring systems. While we are still uncertain of the causes of the backup server outage, we're going to look at two approaches to avoid this problem recurring. 1) Start backing up zimbra002 to a different server. 2) Mount the disk, do the backup, then unmount it straight afterwards. That way, if the server has a problem, hopefully the mount will fail, and we'll skip the backup rather than breaking the server. Once again, apologies for all affected by this today, and hopefully this goes someway towards explaining things! By Adrian |
||
| 09/11/18 10:19 | Zimbra002 | Open |
| We are currently experiencing issues with the zimbra002 server - engineers are working to resolve this and we will update when we have an update. In the mean-time, please accept our apologies. By Iain |
||
| 09/11/18 09:53 | zimbra002 Outage | Ongoing |
| We're currently having problems with zimbra002 caused by a backup server crashing, which breaks the NFS mount. We're requesting a reboot of the backup server, and have initiated a reboot of zimbra002 remotely, so hopefully all will be back to normal shortly. In the meantime, apologies for any inconvenience caused, and we're off to try and figure out a better way of doing things so this situation can't happen again! By Adrian |
||
| 25/10/18 11:18 | Network Instability | Ongoing |
| We're currently experiencing some network instability caused by one of our routers running a little short on memory. I've just initiated a reboot, so things should settle down shortly. Apologies to those who were affected by this - should be mostly overseas routes rather than UK ones. By Adrian |
||
| 24/09/18 12:10 | zimbra004 Reboot | Open |
| The Zimbra server, zimbra004 has just crashed and is being rebooted. Please bear with us while we get it back into action and accept our apologies for any inconvenience caused by this. By Adrian |
||
| 03/07/18 10:49 | Zimbra Server Maintenance | Open |
| We will be performing minor maintenance of the following servers on Thursday the 5th July, starting at 22:00: zimbra002, zimbra003 and zimbra004 Downtime should be minimal. By Iain |
||
| 02/04/18 09:21 | mail001 Resolved | Resolved |
| I'm pleased to say we've been able to figure out the problem with mail001 server. Turned out to be a software race condition in some housekeeping code. A recursive function was getting stuck in a loop due to a user having an extremely long mail folder name that happened to cause a directory to be precisely 256 characters in length. This was causing it to eventually take all available memory, causing other services to slow down and eventually fail. By Adrian |
||
| 02/04/18 08:10 | mail001 Server Problem | Open |
| mail001 has crashed again, and is being rebooted. Obviously, some more investigation is required and possibly some emergency maintenance. Meanwhile, apologies for all those affected. By Adrian |
||
| 31/03/18 07:08 | mail001 Rebooted Successfully | Closed |
| One of our back-end mail storage servers had become unresponsive, and has now been rebooted. All services should now be functioning as normal. Apologies to all affected by this. We will try to investigate the cause of the crash, but odds are, we won't be able to find any clues, but at first glance it looks like an OS software glitch. By Adrian |
||
| 31/03/18 06:50 | mail001 Reboot | Open |
| One of our back-end mail storage servers has become unresponsive, and is currently being rebooted. By Adrian |
||
| 21/03/18 23:38 | Network Issues | Open |
| We've had some reports of connectivity issues tonight to certain providers, and have identified a problem with one of our upstreams that may be causing this. We have just switched off this link so that traffic should re-route through other providers, and have raised the issue with the provider of this connection. Please accept our apologies for any inconvenience caused tonight while we were figuring this out. By Adrian |
||
| 13/03/18 12:18 | Happy 10th Birthday syslog001! | Open |
| We thought it shouldn't go without mentioning that one of our servers, syslog001, has now passed it's 10th birthday. That's not just 10 years in service, but 10 years uptime, with no reboot etc. This is a testament to the reliability of the Dell hardware, the power supply in Telehouse and the FreeBSD software that runs on this server. Sadly, it's days are numbered - it has been due a replacement for some time, but as it is a non-essential piece of kit and was nearing it's 10th birthday, we wanted to give it a chance to make it all the way. So, happy birthday syslog001, and thanks for all the logs! By Adrian |
||
| 05/03/18 20:20 | SSL Upgrades | Open |
| Tonight we are performing some upgrades on our Proxy servers to ensure we're running the latest and greatest versions of SSL. This work should go largely unnoticed, but it's possible some users running really old systems will have compatibility issues as a result. Please let us know if you come across any issues, and we'll do our best to resolve - but in many cases it may just be time for those users to finally upgrade their own systems. By Adrian |
||
| 06/02/18 14:24 | Service Alert: Server Upgrade - zimbra002 - tomorrow (Wed 7th Feb) | Open |
| REMINDER: We will be upgrading our zimbra002 server to Zimbra 8.7 tomorrow night, with work commencing at 10pm. We will endeavour to minimise any disruption to service, and any inbound email received whilst each server is offline will be queued until the relevant server is live again. By Iain |
||
| 30/01/18 14:11 | Server Upgrade: zimbra003 - tomorrow (Wed 31st Jan) | Open |
| REMINDER: After being delayed for a week we will be upgrading our zimbra003 server to Zimbra 8.7 tomorrow night, with work commencing at 10pm. We will endeavour to minimise any disruption to service, and any inbound email received whilst each server is offline will be queued until the relevant server is live again. By Iain |
||
| 16/01/18 16:58 | Server Upgrade: zimbra004 - tomorrow (Wed 17th) | Open |
| REMINDER: We will be upgrading our zimbra004 server to Zimbra 8.7 tomorrow night, with work commencing at 10pm. We will endeavour to minimise any disruption to service, and any inbound email received whilst each server is offline will be queued until the relevant server is live again. By Iain |
||
| 19/12/17 14:40 | Zimbra Upgrades in New Year | Open |
| Important - Zimbra Upgrades As part of our ongoing programme of service enhancements, we will be performing upgrade work on our Zimbra servers during the month of January. This work will be undertaken outside of normal working hours (in the evening) and will involve a period of downtime for each server. We will endeavour to minimise any disruption to service and any inbound email received whilst each server is offline will be queued until the relevant server is live again. The dates will be as follows - we will be sending reminders beforehand. Wednesday 17th Jan: Zimbra004 - upgrade to Zimbra 8.7 Wednesday 24th Jan: Zimbra003 - upgrade to Zimbra 8.7 Wednesday 31st Jan: Zimbra002 - upgrade to Zimbra 8.7 Whilst Zimbra 8.8 has just been released, we want to monitor it over the coming months to ensure stability. Once satisfied that it will be reliable we will upgrade to that version. We haven't forgotten Zimbra001 and will contact customers on this server about moving users in the New Year. If you have any questions please let us know. By Iain |
||
| 25/10/17 14:13 | Major Proxy Server Issues - Update | Ongoing |
| Latest update on this. Everything should now be back up and running properly. We've managed to patch some software to control the connections coming in on POP3 over SSL so that another attack on that port shouldn't overload the servers. We will, of course, continue to watch closely! By Adrian |
||
| 25/10/17 13:16 | Major Proxy Server Issues | Ongoing |
| It looks like we've suffered from some sort of attack against our proxy servers, which has unfortunately resulted in the need to reboot them. This process is underway at the moment, and we hope to have services restored shortly. We are currently blocking some POP3S services, as this is the source of the attack, we believe, and we will look to restore these as soon as we can. Please accept our apologies and be assured we're working hard to restore service and figure out how we can prevent a recurrence. By Adrian |
||
| 24/10/17 19:21 | Proxy Server Issues | Ongoing |
| We're having some sporadic performance issues with one of our proxy servers. This is causing logins for IMAP/POP3/Webmail etc to fail for some people from time to time - effectively a series of brown-outs that have been occurring throughout the day and lasting 1-5 minutes at a time. After much investigation, we've managed to track it down to someone trying to brute force crack some users passwords via a POP3S connection. Unfortunately they're doing this with thousands of connections from approx 800 different IP addresses in the space of a minute, which is what is causing the server to overload. We're currently trying to assess the best way to deal with it, but please be assured we are looking into it and hope to have some kind of protection in place shortly to prevent it recurring. By Adrian |
||