Life in IT is full of onerous tasks. Along with making good backups and maintaining a solid patching regimen, you must ensure that multiple levels of antimalware software are properly deployed. Unfortunately, in heavily virtualized environments, antivirus can go beyond being a pain to manage and actually become a threat in and of itself. As the saying goes, sometimes the cure is worse than the disease.
That antivirus software can slow down a machine probably comes as no surprise to anyone. Any software that watches each and every disk I/O and inspects it for threats adds overhead that didn't previously exist. In most cases, this manifests itself through marginally higher disk latency and greater CPU load. But with careful use of scanning exclusions (for heavily used databases and the like), it's usually not enough to bring a system to its knees.
Recently, however, I've been presented with two excellent examples of how antivirus run amok can have enormous sitewide impact -- and how it can be difficult to detect the cause unless you know to look for it and have the monitoring data necessary to do so.