September 30, 2011
Benchmarking Without Weightings: Like a Burger Without a Bun.
Hi everyone!
With the help of my colleagues I’ve been slowly but surely getting up and running a series of posts (here and here) about key technologies – to introduce them to the public, judge the reaction, and then gather ideas. But besides singing the praises here, I’d also like to give you my opinions on comparative tests – those that inform the public how efficient these technologies are. Alas, there are not that many tests I trust and can recommend.
There are just too many shortcomings in today’s testing methodologies, meaning the tests provide only a snapshot of the tested products and miss the whole picture. But it precisely the whole picture that is what customers need. Unfortunately, the majority of tests still employ old testing practices (like on-demand testing with outdated malware collections), which don’t reflect current real-life user scenarios.
And so now let me say a few words about PassMark. This is a very respected organization and I really admire the job it does. However, its recent anti-virus performance test has at least one significant flaw, which could mislead readers and cause them to make purchases based on faulty comparisons.