Anti-Malware Comparison Testing

[mikey]

When I see all the many different so called tests of anti-malware products, all I see is a bunch of advertising. In addition to the fact that no two of these so called tests find the same results, I have some other concerns;

How does one who is testing take into account the fact that all of the variables involved as well as the tools themselves are in a perpetual state of flux? Any real testing takes a bit of time. Even before any test is published, the results will already be invalid. While BrandX is still working on a particular definition, BrandY has already done so. Yet the BrandX definition will be published the next day and possibly better developed than that of BrandY. 

Any real benchmark of these tools must include the study of it's removal routines for each type of nasty currently in the wild. That alone is a daunting study. But without it, there is no value to the testing. 

Most F/Ps (false positives) don't occur on a freshly installed system. Removing items falsely can and very often does cripple innocent components. How do you measure the probability of F/Ps? 

A true benchmark of the detections must include a validating sampling of targets. Limiting the sampling does not represent a true test at all. I can make any tool look good by simply limiting the sampling for the test. In all of the testing that has been published by online magazines and other so called professionals, this limiting has caused every single one to have different results. This has also been used as a marketing strategy. 

In addition to studying the detections and removals, what other features are offered by the tools? What proactive features exist and do they work as pitched? Almost every scanner advertises that it protects the system. Do they really?...to what degree?...how much of it is just bloat? 

Should any testing done by those affiliated with a particular tool be considered viable? How does anyone reading a test result know if a test was done by someone who is affiliated or has interest in a particular tool? 

I have yet to see a real viable comparison test/benchmark. IMO, the methodology to perform a real comparison does not exist. Also, I believe that 99% of the so called tests published to date are simply advertising ploys and have absolutely no truth to them. I believe the other 1% are just done by well meaning folk who just simply don't have the understanding or expertise required in order to perform such testing.

Ask yourself; Why doesn't any two published comparison testings report the same results?

[Corrine]

Facing reality, we know that even if a so-called comparison is not sponsored by an anti-malware company, that company most likely provides advertising revenue to the organization conducting the tests.  When reading the results of various testings conducted, the tests are always in a controlled environment and not necessarily following recommended removal methods by the vendor for the particular malware (i.e. scanning in safe mode). 

Anyone who has spent any amount of time "on the forums" knows that no software, whether it be Anti-virus, -malware, or -trojan will work if the file(s) being removed are not in detection. 

IMO, the real testing is done "in the field".  I think we tend to see what works best for the types of infections encountered.  When users write, "I've scanned with __________, _________ and __________ but still have a problem" and we repeatedly see the same entries in the logs, that "testing" is likely more illustrative of how well those software applications work in real world environments.

[mikey]

I just wish I could get folks to see through the hype and see the realities. I'm so tired of hearing; "That can't be right because 'so and so' said brandx is the best."

I try to teach my users real sec so that they don't need any brand. But most folk think they absolutely need some brand or even two or three scanner products on board when the truth is that they are just buying a false sense of security.

I think it is sad that so many still periodically rip the guts out of their system while usually using multiple scanners and without even a clue to what items they are removing. Then these same folks wonder why their system and applications fail to function properly after they have been ripped apart by various scanners.

In our sys;

We don't want our system to be bogged down with massive bloatware that uses half the available resources including tech support.

We don't want to maintain/update multiple signature DBs that are ALWAYS going to be behind the infection instead of preventing it. If we were to get a newer malware on board somehow, none of the conventional scanners are likely to even detect it much less remove it properly anyway. 

We don't want to spend the human resources required for clean up after an infection that could have just as easily been avoided.

What we do want is 'control' and that is what we strive for. IMO all resident signature based scanners are just dinosaurs.

The industry continues to suck money from users with annual subscriptions to lease (not sell) their wares. Of course they aren't going to innovate to techniques and processes that really protect...that would decrease their revenue...a revenue & market share now sought after by every scam artist out there. Some vendors have even been caught playing on both sides of the street.

So, I really feel sorry for those folks who are still being exploited by both the malware propagators and the anti-malware industry simultaneously.

You know, I just don't believe that the majority of folks are morons. The pri/sec community doesn't even try to teach REAL security and the only reason I've been able to see over the years is because it is less profitable for the bogus anti-malware industry.

Yes, some low level techniques like blocklist content filters are suggested but only a handful of folks try to teach real security. Thus the majority of users are led down the path to 'continual exploitation' by both the propagators and the so called pri/sec community.

Even one of the well established anti-malware developers (DiamondCS) has now denounced the bogus and obsolete methods that have thus far been predominant (scam) and they completely retooled their organization. More and more honest developers will surely follow suit. If they don't, I hope they get left behind. There have also been a rash of new products offered lately that do offer security for the average user...HIPS. IMO It's time for this long siege to end.

I also think that the majority of folks looking for help and guidance want to learn how to avoid infections. It would be nice to see that actually happen throughout the pri/sec community instead of just telling them that brandx scanner will fix everything.

So many times in the pri/sec community I hear the leaders saying "The average users are too lazy or too stupid to learn anything new."

My answer to that is; HOGWASH