Service Alert: zimbra003 Hardware Migration

Next Thursday 27th August we will be performing a hardware upgrade for zimbra003 to take it onto newer faster hardware and full SSD disk system.

This work will start at 10pm BST, and will hopefully complete within an hour or less, but it is possible that it will take several hours if things don’t go perfectly to plan.

Please accept our apologies for any inconvenience caused by this work.

Service Alert: zimbra002 issues earlier today

From Adrian:

An explanation is in order about the zimbra002 downtime this morning, so please bear with me while I try and explain.

The Zimbra servers have a backup mechanism that can only back up to a mounted drive, so either a local drive or an NFS mount.

Every night they do an incremental backup and once a week they do a full backup. The NFS server to which zimbra002 and zimbra003 (which never seems to be affected in the same way) are backing up to crashed in the night. It crashed in such a way that it was still responding to pings, but not NFS calls, so this wasn’t picked up by our monitoring systems.

When an NFS mount goes away, a Linux server treats this as a very bad thing, and basically waits around for it to come back again.

This tends to affect the process that is attempting to access the mounted drive, and other processes normally continue unaffected. Because Zimbra is a monolithic piece of Java code, the backup process causes it to misbehave.

However, it still responds on all the ports that we check with our monitoring systems – the problems occur after logging in, so once again not picked up by our monitoring systems.

While we are still uncertain of the causes of the backup server outage, we’re going to look at two approaches to avoid this problem recurring.

1) Start backing up zimbra002 to a different server.

2) Mount the disk, do the backup, then unmount it straight afterwards. That way, if the server has a problem, hopefully the mount will fail, and we’ll skip the backup rather than breaking the server.

Once again, apologies for all affected by this today, and hopefully this goes someway towards explaining things!

Service Alert: zimbra004 Reboot

The Zimbra server, zimbra004 has just crashed and is being rebooted. Please bear with us while we get it back into action and accept our apologies for any inconvenience caused by this.

UPDATE: Up and running again – may be slow for a few minutes as it catches up.

Service Alert: Zimbra Server Maintenance

We will be performing minor maintenance of the following servers on Thursday the 5th July, starting at 22:00:

zimbra002, zimbra003 and zimbra004

Downtime should be minimal.

Service Alert: Server Upgrade – zimbra002 – tomorrow (Wed 7th Feb)

Reminder

We will be upgrading our zimbra002 server to Zimbra 8.7 tomorrow night, with work commencing at 10pm.

We will endeavour to minimise any disruption to service, and any inbound email received whilst each server is offline will be queued until the relevant server is live again.

Service Alert: Server Upgrade: zimbra003 – tomorrow (Wed 31st Jan)

Reminder

After being delayed for a week we will be upgrading our zimbra003 server to Zimbra 8.7 tomorrow night, with work commencing at 10pm.

We will endeavour to minimise any disruption to service, and any inbound email received whilst each server is offline will be queued until the relevant server is live again.

Service Alert: Server upgrade – zimbra004 – tomorrow (Wed 17th)

Reminder

We will be upgrading our zimbra004 server to Zimbra 8.7 tomorrow night, with work commencing at 10pm.

We will endeavour to minimise any disruption to service, and any inbound email received whilst each server is offline will be queued until the relevant server is live again.

Service Alert: Zimbra Upgrades

Important – Zimbra Upgrades

As part of our ongoing programme of service enhancements, we will be performing upgrade work on our Zimbra servers during the month of January.

This work will be undertaken outside of normal working hours (in the evening) and will involve a period of downtime for each server.

We will endeavour to minimise any disruption to service and any inbound email received whilst each server is offline will be queued until the relevant server is live again.

The dates will be as follows – we will be sending reminders beforehand.

Wednesday 17th Jan: Zimbra004 – upgrade to Zimbra 8.7
Wednesday 31st Jan: Zimbra003 – upgrade to Zimbra 8.7
Wednesday 7th Feb: Zimbra002 – upgrade to Zimbra 8.7

Whilst Zimbra 8.8 has just been released, we want to monitor it over the coming months to ensure stability. Once satisfied that it will be reliable we will upgrade to that version.We haven’t forgotten Zimbra001 and will contact customers on this server about moving users in the New Year.

If you have any questions please let us know.

Service Alert: Zimbra004 Server – PSU Change

From approximately 7pm this evening (or shortly thereafter) we will be swapping out a faulty Power Supply Unit (PSU) on our Zimbra004 server.

We do not expect any impact, or outage, to the service.

Service Alert: Password Security

Password Security

We are seeing a significant rise in customers’ email accounts being used to distribute spam, having had their account credentials compromised.

In many cases this has been purely down to simple, fairly obvious, passwords being used. Would you believe that we have 276 mailboxes that use ‘password’ as their password? Or 73 with the password 123456?

It’s important for all our customers that we minimise the potential of our mail servers becoming blacklisted, so where patterns of outbound sending indicate a compromised mailbox and the distribution of spam, we will block the account from sending out any further email until the password is changed and a virus scan on the end users equipment performed if required.

This helps mitigate against blacklisting, but isn’t perfect by any means as it’s reactive in nature.

Whilst we can’t improve on the way we identify compromised mailboxes, we can improve the tools we give Partners to re-enable outbound SMTP immediately following a block.

Currently we send out an alert when an account is locked and rely on you contacting us to re-enable. Add to that, any blocking we’ve done has been at account level, rather than individual mailboxes, so one compromised mailbox can lead to outbound emails being blocked for the whole account.

As of Tuesday 27th June we are implementing a new process for dealing with compromised accounts:

  • Blocks can now be applied at individual mailbox level, rather than account level. This means that only the affected mailbox will be restricted from sending, rather than all users on that account.
  • Our systems will now automatically unlock any affected mailboxes once the user, or administrator, has changed the password.
  • Next Tuesday all mailboxes with passwords we deem to be easily compromised (for example, using part of the email address) will be blocked from sending outbound email until their password has been changed. This will only affect a relatively small proportion of our overall customer base, but needs to be implemented as the issue of compromised email boxes is on the rise.

We very much hope you’ll welcome the changes we’ve made which, as well as giving more control to customers in the event their email credentials are compromised, it also encourages everyone to think a little more seriously about the security of their email!