The Black Box.

Filtering out spam may not seem such a big deal – after all, even a kid knows the difference between a Viagra advert and a normal message! In the security world things are much more complicated as we have to create something akin to artificial intelligence that is capable of doing the job automatically, on the fly.

That’s no easy task and entails all sorts of demands in terms of efficiency, reliability, compatibility and so on. And you no doubt know where things stand with AI – there are plenty who claim to have got it figured, but there’s nothing really to show for it (or if there is, they’re doing a good job of keeping it a secret).

Anti-spam security is no easier a task than anti-malware protection. And may even be more difficult (or maybe I just understand more about viruses…). The spam industry is a multi-billion dollar business and tens of thousands of skilled bloodsuckers are behind the huge variety of junk that is sent out. And these parasites show great ingenuity when it comes to linguistics and other stuff to make spam reach your inbox.

On the face of it, a spammer’s work looks fairly easy – write a spam message, test it against several of the most popular anti-spam filters and spawn via a botnet.  But few customers realize that a spam message’s lifecycle is just half an hour to an hour long. 90% of a mass mailing will never reach its intended recipients – spam filters, activated with an update or triggered by statistics, will intercept it.

And it’s that black box – the thing that withstands the worst things that email traffic throws at it and keeps your inbox clean – that I want to discuss here.

First of all, a bit of background. Since 2002 our anti-spam solution (KAS) has got through four generations of engine and we’re now developing a fifth. A single blog post would hardly suffice to recount everything. Basically, KAS has acquired lots of bits and bobs over the last 10 years. It boasts over 10 methods of spam analysis alone. That’s why I’ll start with our new ‘Möbius‘ technology – just in time for its debut in the latest version of KAS for Exchange Server.

One of the biggest problems associated with filtering spam is the speed at which an update can be delivered. And we release up to 200 updates a day. It is pretty clear that the faster our customers get updates, the less spam finds its way into their inboxes. Besides, spam messages spread over a network faster and more often than malware.

Before Möbius, an average update delivery took an outrageous 22 minutes! That, by the way, is a very good result for the industry. True, that included literally everything – multilevel testing, uploading and downloading to and from public servers and all sorts of possible technical delays. As a result, there was a 2-3% discrepancy between the detection level in our anti-spam lab and that on the customer side. After analyzing the situation we decided the best solution was a special client-server connection with push updates. That’s why we chose the name Möbius – to symbolize the inseparable connection between the anti-spam lab and our users in the form of database synchronization :)

How Möbius works

It’s probably best to compare it with traditional updates of anti-spam databases that involve a server and a client. Basically, the client regularly sends requests to the server and, if there’s something new, it downloads an update. You wouldn’t think there’s much that could be optimized here besides making the regular requests as frequent as possible.

Möbius, however, consists of 3 parts: a back end, a front end and a client. The back end gets an update from the anti-spam, performs automatic testing and transfers it to the front end. Moreover, it doesn’t work with all updates, just the most urgent ones – text and image signatures. We therefore speed up the process, avoiding “bulky” updates and downtime linked to business logic. The front end maintains a continuous connection to the customer and as soon as a new update is available, downloads it to the product where the Möbius client receives the update, unpacks, compiles and enables it. The subsequent updates are performed practically in real time – the interval between their creation in the anti-spam lab and going live in the product is now less than 1 minute!

All in all, it is pretty similar to the ‘cloud’, but its features are optimized for a specific task. However, Möbius has several advantages over the traditional cloud:

• Delivers updates faster (it pushes them to the product without waiting for a scheduled update)
• Works faster (spam filtering is performed at the customer side – there is no need to transfer hash checksums of unclassified messages to the cloud and wait for a response)
• Provides better security (the anti-spam module receives data in its original format, not hash checksums – meaning there’s more scope for analysis)
• More efficient and reliable (depends less on the customer’s traffic and the traditional update delivery system)

I should note here that we also use typical cloud technology in our anti-spam – yet another area in which we have achieved true hybrid security. For example, there is an Urgent Detection System (UDS – remote categorization of a message according to specific properties), UDS2 (UDS with specific fuzzy-hash implementation for more efficient spam detection) and Content Reputation (in addition to UDS2 – automatic evaluation of spam reputation for various content characteristics). All this has the makings of a doctorate degree, but more on that later.

Let’s proceed to the facts.

Möbius is part of the fifth KAS engine. It has now taken over the watch in our product for Microsoft Exchange Servers. Further plans will see it integrated into other gateway and server solutions: Mail Gateway, Hosted Security and KAS SDK. There are as yet no plans to include it in our products for home users. First of all, a home user has little need for a database that updates at such a rate and, secondly, there aren’t that many computers that would meet Möbius’s requirements for maintaining a dedicated connection to the front end. And frankly speaking, the front end is not ready for it either!