The rise of customized malware is forcing security software vendors to change their tactics quickly and begin using customers' machines as their initial line of threat detection intelligence, according to a new report from Yankee Group.

Echoing recent comments made by industry leaders like Symantec -- which is considering white-listing techniques, among many other emerging plans, to thwart the trend toward so-called server-side polymorphism -- Yankee Group Analyst Andrew Jaquith writes in a new research note that "herd intelligence" will be one of the most effective ways for vendors to detect and address increasingly customized threats.

By turning their customers' endpoint devices into malware collectors that can funnel information about new attacks back into their global networks of threat sensors and scanning technologies, Jaquith said, security applications vendors may make faster progress in stemming the tide of lower profile, smaller volume threats.

As malware authors have begun enlisting more malware toolkits and other technological means to create a greater number of attack variants than can be found and processed by honeypots and signature-based security software tools, the analyst said, it will become vital for vendors to aggregate threat data using customers' computers.

Some smaller vendors, including ESET, Panda Security, Prevx, and Sana Security, have already begun working in such a fashion, turning their deployed endpoints into collectors.

Larger vendors looking to add that type of technology to their pallets could seek to acquire one of those vendors, Sana in particular, to advance their plans more rapidly, Jaquith contends.

The idea is simple, according to the analyst. If attackers are going to attempt to create different attacks for nearly every individual user, then security software vendors must use their customers' machines as their eyes and ears for discovering and addressing those variants.

On the flip side, as the vendors amass information about new attacks, they can simultaneously help other customers determine whether new applications or Web sites are dangerous or safe to use, the analyst said.

"When an unknown binary attempts to execute, the client-side agent sends detailed telemetry information to a remote centralized server and asks whether it is good, bad, or unknown," said Jaquith. "The server makes a disposition decision based on all the collective history accumulated by the herd. By pooling information about all executing programs across its installed base, the herd makes smarter decisions and can confer immunity faster to new variants."

As part of the effort, security vendors may also need to begin sharing more of that information with their rivals to create a larger network effect for thwarting malware on a global basis, according to the expert.

It may be hard to convince rival vendors to work together because of the perception that it could lessen differentiation between their respective products and services, but if the process clearly aids on the process of quelling the rising tide of new malware strains, the software makers may have little choice other than to partner, he said.

"By turning every endpoint into a malware collector, the herd network effectively turns into a giant honeypot that can see more than existing monitoring networks," said Jaquith. "Scale enables the herd to counter malware authors' strategy of spraying huge volumes of unique malware samples with, in essence, an Internet-sized sensor network."