Last week (January 8th and 9th) we received a dozen reports of messages that simply vanished in the ExchangeDefender system. Upon investigation it turned out that one of the antivirus engines was picking up false positives: marking messages with certain PDF attachments as infected when in fact there was no infection there. The actual infection was simply a detection of an exploit, one that can easily and inadvertently be created by older versions of Acrobat.

We have removed the antivirus engine from the rotation (don’t worry, everything is still being scanned by several other scanners). While the problem in the definition files was already addressed (Exploit.PDF-9669) and widely blogged and discussed, we need a way to deal with false positives. Prior to this we have never had an instance of a reported false positive with an antivirus engine but as more antivirus vendors get into the business of not just detecting viruses and worms but also exploits and other dangerous content, our reporting will have to get better as well.

The bigger question here is: Why was I not notified? If this happened here, it would also explain why I am never received any of the other messages. Allow me to address that in two ways:

1) Almost all of our “missing messages” tickets are related to the messages being quarantined as SPAM and not coming into LiveArchive. At the present time there is no way to get a SPAM message into LiveArchive, even after it’s released from the Quarantine. Because our replication is done at the scan time, we have to move the copying protocol elsewhere to enable post-release and SPAM content.

Followup question: But Vlad, I need to be able to view my SPAM and respond to it while my server is down!! And you can, right from admin.exchangedefender.com! All of our new enhancements are coming to that portal which is completely partner branded and next month we’ll even have training you can just point your clients to.

2) We have never before seen a false positive from an antivirus engine. We’ve seen it crash, we’ve seen it fail to detect a real infection, we’ve seen it bring the scanning node to a crawl and just about everything you’d expect from a piece of security software: just never a false reading. Consequently, we never wrote a process to monitor for the false positives and we never bothered to present the infection logs because so many contained meaningless junk. Several years ago, after countless alerts for Sober and Nimda and so on, we disabled end user reports for antivirus and it was eventually dropped from the product completely.

At this time,  we are sketching a way to put back a configurable alert system for infections should this happen again. We are also creating a system by which you’ll be able (administrators only) to access the infected quarantine items from the web UI).

IMPORTANT: While these infections appeared to be lost forever, we do have them stored on our servers. Reported messages are being released (by hand) by our support teams so if you know the message sender/recipient/subject and date the message was sent, we can retrieve the message and deliver it.

-Vlad