Using MessageBunker with Gmail

Gmail archiving

A requirement of MessageBunker that we store the user’s credentials, encrypted of course, so that we can access their mailbox for archiving. Wherever possible, we look for alternative solutions that allow for improved security and privacy. Gmail offers one such solution.

MessageBunker can be granted a token by Gmail with the end-user’s permission, that needs refreshing hourly. This token can be used to log in. The process is known as ‘oAuth’ as is fairly common on the web, but this is the only implementation for IMAP that we are aware of.

When a user creates a new archive on MessageBunker, we send them to Google to give us permission to access their Gmail. We then get a confirmation back and a token we can use to log in.

The user does not need to reveal their password and can revoke our access to their Gmail account at any time, without the need to change their password, or alter any other systems that are accessing the account.

Under the hood: Spam Wars

In the ongoing war against the spammers, we have put a lot of effort over the last year or two in looking at the effectiveness of various methods, and thought it might be helpful to give a bit of a behind-the-scenes look at some of the lists and methodologies we use and their relative effectiveness. What these stats don’t show is the amount of false positives or the amount of spams that we miss as with our diverse user base it is impossible to measure these things accurately.

We try to be quite aggressive at detecting spams as the majority of our users make use of what we call auto-whitelisting, where anyone they send an email to automatically gets added to their whitelist and doesn’t get checked for spam in the future (well not at stage 2 anyway – see below).

The first stage of our spam blocking is the most aggressive, and most sensitive. If we have false positives here, we tend to find out about it because we reject the connections based on the IP address that is trying to connect to us.

I’ve included links below so you can investigate and find out more about any particular list.

Firstly, let’s look at connections to our servers. Taking a sample day, of Wednesday December 14th 2016, we received a total of 6,680,134 inbound SMTP connections. Here’s what we did with them.

Spamhaus Zen 5,403,727
Invaluement ivmSIP 236,211
Accepted 1,040,196

Of those 1,040,196 accepted connections, we received 1,008,901 individual emails. These were then broken up as follows:

Whitelisted 138,055
Blacklisted 3,513
Too Large to Scan 6,960
Not scanned (user not enabled anti-spam) 131,480
Scanned 728,893

So we now have a grand total of 728,893 emails to feed into our anti-spam servers. These run a piece of software called Spamassassin that looks for patterns in emails that mean they are probably spam and score them accordingly. Unfortunately, the spammers have access to this, and the good ones are very clever at making their spams not look like spam to a computer (though still obviously spam to a human), so we rely quite heavily on various blacklists to identify spam for us.

In the last couple of years, the spammers have become even more sophisticated and found ways to send out millions of spams before the blacklists are able to list them. The blacklists are fighting back, however, with new lists such as InstantRBL and faster listings (particularly good at URIBL).

Taking our sample day with 728,893 spams to be scanned, here is how many are caught by each different method/list employed. These stats show unique hits (so, for example, if something is caught by two lists, or one list and other Spamassassin rules, it won’t show up).

Spamhaus Zen 2,062
URIBL 8,137
Invaluement ivmSIP24 2,029
Invaluement ivmURI 10,194
Barracuda 9,109
InstantRBL 8,442
Protected Sky 5,468
Spamassassin other rules 32,007
Total caught 113,439

It’s hard to draw a pretty chart from all of this. However, here are the headline figures. 6,680,134 inbound connections, 891,949 emails delivered to inboxes, which represents 13% of the total. 131,480 of those didn’t get a chance to be scanned because our end user didn’t have the feature enabled.

So there you have it, it’s an ongoing battle, and the battleground keeps shifting. The spammers have access to all of the same tools that we do – that’s the nature of the internet, so they will keep trying to find new ways to beat the system, and we will keep trying to find new ways to stop them.

Old MessageBunker is dead, long live the new one!

MessageBunker

Following on from our announcement at the end of September regarding the new MessageBunker interface and initial running of old and new versions in parallel, we can announce that, as of today, the old site has now gone – to be completely replaced by the new one.

Take a look at messagebunker.com