Notes, comment and buzz from Eugene Kaspersky – Official Blog

February 23, 2012
  • 0
  • 0
  • 0

Features You’d Normally Never Hear About – Part Four.

Hi all,

Once again, the subject is spam.

Depending on the “stars” and the time of year, the proportion of spam can range from anywhere between 70 and 90% of all email traffic.

Sounds like a lot, eh? But when you take all Internet traffic into consideration, it’s not actually that much – email traffic accounts for around just 1%. On the other hand, you can’t just forget about spam. Here is a bit more about spam’s role in the cybercrime ecosystem. Combating this particular evil is part of the massive war we are waging on cybercriminals. It’s no exaggeration to say that if we fail on this front, the rest of our efforts will amount to nothing.

In other words, we love anti-spam technologies and promote them as much as possible. There is, however, a subtle difference from anti-malware technologies. More precisely, there are different criteria for evaluating the quality of protection for anti-spam and anti-malware technologies. For malware it’s fairly easy: the higher the detection level, the better. For spam it’s more important to have no false positives. This is quite reasonable: it’s much better for the user to take a couple of seconds to delete a spam message that sneaks through the filter than miss important business correspondence. So, protection against spam is, in a way, a more complicated task, literally trying to kill two birds with one stone. In this difficult task, cloud technologies are a great help.

As I wrote earlier, we’ve been using cloud technologies for a while, and with considerable success. But one interesting detail has amazingly been overlooked, and unfairly so. In the cloud-based Kaspersky Security Network (KSN), (video, details) there’s a rather impressive anti-spam cloud. It started from the Urgent Detection System (UDS). The link to similar anti-malware technology is no coincidence: both are based on similar principles.

This is how the traditional anti-spam technology works.

Let’s say an email arrives at a computer. It is immediately assailed by various anti-spam technologies, both local and cloud-based, which test the message and give verdicts. Based on these, the system decides whether this message lives or dies.

And this is what happens in the UDS.

The system takes a micro-signature from the email message and sends it to the cloud to check it against a dedicated spam database. Earlier we used 16-byte hashes; in 2011 we started the UDS2 (UDS 2nd generation) procedure involving 4-byte fuzzy hashes, which are more effective against obfuscated texts and are therefore better at filtering out spam. Importantly, these hashes do not create extra work for the analyst, since the system creates them automatically based on collected spam samples.

The two main issues for anti-spam are: (i) how fast updates are developed (analysts are only human and their resources are not unlimited); and (ii) how fast those updates are delivered to the user. For that reason, most developments in this field are now geared towards creating and refining a variety of dedicated technologies, like Möbius, and replacing human expertise as much as possible. This is where UDS2 has enormous potential.

You see, UDS2 has a feature called clustering. The first generation of UDS returned simple replies to users’ queries about whether a certain signature existed in the database or not. UDS2, however, groups similar signatures into clusters and calculates their spam reputations to boot! This in turn helps process spam automatically in the cloud.

The clustering and automatic spam processing is performed by Content Reputation technology. In 2012, we plan to release a number of new features based on it. The first is Rescan, which allows the user to impose a short (20-30 minutes) optional delay in checking only suspicious emails. In real life, such messages make up no more than 1% of the entire mail traffic, so the amount of quarantined mail will not exceed 100 MB even in a large organization. This period of time will be sufficient for us to figure out what is spam and what isn’t, and add a signature.

The second feature is Auto-ban, or automatic blocking of emails which were grouped into high spam reputation clusters. Naturally, they will then be checked manually by the analyst, and the blocking can be instantly rolled back if necessary. In this approach, the system prompts the human to do the analysis rather the human asking the system, which makes things faster and more effective.

Both features are now being put through their paces in internal tests and are delivering very encouraging results. For instance, Rescan reduces the amount of unrecognized spam by a factor of ten (yes, ten!) and improves the general detection rate from 99.50% to 99.95% (that’s right!) This is in fact among the best results across the entire industry, while still keeping the false positive rate at zero.

We also have some other very ambitious plans. First of all, we intend to develop a multidimensional ‘clusterizer’ to group and calculate reputations for a number of other attributes on top of fuzzy hashes. There are oodles of such attributes in KSN.

I wonder if we’re going to make it into the elite 100/0 club (100% detection / 0 false positives) in testing this year? Wish us luck!

comments 1 Leave a note

Mário Madrigrano Jaber

Here in Brazil we have a cybercriminal acting on Facebook pages.
He sets up a fake domain and with the false “http” claiming to be from Facebook Tean, it blocks your first page and is sending him emails saying they will only “unlock” your page if you will send digital copies of personal documents such as identity card, document collection of federal tax, birth certificate and also a copy of credit cards. Believe it if you want!
The worst of it is that it can “lock” it.
I tried all possible ways, changing password, etc., And failed.
Perhaps this “hacker” has even the complicity of some official Facebook itself, which is not impossible.
Should there be a way to track this type of cybercriminal and apply it the harsh penalties of the law
While there is interest in my country of local authorities, this type of stroke will continue to exist.
From what I understand this system does not require any personal data protection, but Facebook’s own administrator, Mr. “Mark Zuckerberg” do something real to prevent hackers to seize the pages of its users.
While he himself as the owner of the site not interested, we are lost.

May God protect us!

Thank you Eugene.

Mário Madrigrano Jaber

Reply to conversation
Trackbacks 1

Kaspersky (Server) Anti-Spam: No Longer the Underdog; More Top Dog. | Nota Bene

Leave a note
March 27, 2015

A quick guide to Utah arches.

Why this national park is called the Arches is a rhetorical question. But if you haven’t been following this mini-on-the-road series from Utah, then read this. Yes, you want huge natural rock arches – you need to come here. There’s just so much awesome archness here. Wikipedia says there are as many as 2000 here, […]

March 25, 2015

U tire of Utah’s canyons? Not possible!

The red rocks of Utah – a simply captivating sight; from the outside, that is, which is what we checked out yesterday. Today it was time we had a look at all this from the inside. So off we headed to the Arches National Park‘s Fiery Furnace rocky massif. This is what we saw…

March 24, 2015

The Utah Saints: Crimson columns and massive mushrooms.

Since the previous day my brain had been variously boggling and boiling. This was eased a little by steam being emitted from said boggling and boiling brain out via my camera, but that alleviation process then went too far, leading poor brain into a state of half dehydration. The diagnosis sounds like this: I’ve (finally) […]

March 20, 2015

A practical guide to making up a sensation.

There are many ways to make up something sensationalist in the media. One of the practical ways is to speculate and create conspiracy theories. Unfortunately, there’s a demand for such stories and they have a very good chance of making a splash. So how can a global company with Russian roots play a part in […]

March 19, 2015

Bonny Bonneville. Salty Utah.

Still in the US toing and froing between cities, we decided to make a quick stop in a town I’d never heard of: Wendover, Utah State. The town houses the nearest airport to the famous Bonneville Salt Flats. A saliently saline splendid spot!

March 18, 2015

Massachusetts snow woe, and a US-AV decade of champions.

I’ve been in the USA countless times. Usually it’s just for short stays with a few different places to visit, but there’s normally plenty of interesting tales to tell afterwards. Not this time! This time it was business, business, and again business. In this post, alas, there’ll be nothing too riveting for you, dear reader […]