Dude buys software
Dude installs software
Dude forgets about software until it is time to renew

Dude is your average end user.

Simply put, most average people will simply install a product and assume it is set right. They are not uber-admins like us who tinker and dick around until we have things tight.

And no test is truly unbiased. It just isn’t. It’s like statistics, you pretty much always end up with the result you were looking for by excluding things that you don’t want to muck up the works.

Whilst I’m sure the tests were well intentioned, they are not particularly scientific and as the author admits, Sophos results were significantly improved if the tester turned on the relevant options, suggesting the product documentation was not consulted.

Having retrieved the samples (the author having posted the malware samples on a public website!!!) it appears a few extra settings were required, in particular, one of the samples was a email stored as MIME but the ‘decode MIME’ option wasn’t turned on. Another sample was in fact a potentially unwanted application and again, the option to enable detection for PUAs wasn’t used.

While Sophos performed better than many of its competitors, the sample set was far too small, the methodology was confused and the author obviously isn’t well versed in handling malware. All of this goes to show that testing anti-virus products is a lot more complex than grabbing a few samples and scanning them.

I would recommend anyone wishing to compare products to look for recognised testing organisations and publications such as West Coast Labs and Virus Bulletin.

So take particular care in using this biased testing.

Do you just setup your AV and not configure the options?

The ESET and David Harley papers in particular go into details as to why Antivirus Fight Club is probably not something worth paying much attention to.