When [SPAM] and [SureSPAM] slide through

Lately we have been fielding a lot of questions about why [SPAM] and [SURESPAM] messages keep on sliding through to the end users. We have also seen a lot of activity with users complaining about SPAM making it to them uninterrupted when it comes from an email address within their domain. Here is the problem:

In nearly all cases that we investigated, the user actually whitelisted their own domain or their own domains email address.

Why would this happen? Well, users tend to scan messages and look for familiar names and subjects. When they encounter something they recognize, like an email address from their colleague or from themselves, they trust the sender. When they trust the spoofed address, all future mail comes through, causing frustration for everyone involved.

Advise your users not to trust their own email address space when it shows up in ExchangeDefender SPAM reports. ExchangeDefender only intercepts messages going in and out of the organization, it does not filter internal messages. Any mail with the domains address space caught by ExchangeDefender is highly likely to be spoofed.

Of course, usage and configuration of ExchangeDefender is up to you, we make the product flexible enough to allow you to set your own policies. But blindly trusting entire domains and mirrored trust sets (from exchangedefender.com to exchangedefender.com for example) will only let dangerous items through. Consider tightening up the ship if you are seeing ExchangeDefender starting to slip, our metrics show that our detection rate keeps on going up as both volume and percentage.

As always, thanks for letting us clean your mail.

New ExchangeDefender outbound servers

With ExchangeDefender 4.x infrastructure already in place on the inbound servers, we are moving our focus to the implementation of ExchangeDefender 4.x on the outbound servers. The new systems will go into production this weekend, April 5-6, and will feature new IP address relay servers:

outbound1.exchangedefender.com 65.99.255.236
outbound2.exchangedefender.com 65.99.255.232

This is the same IP range that is currently in use so you should not have to make any modifications or changes to your systems. Everything will just work transparently.

If you are currently using SPF, which we do not recommend, you will have to adjust it to include the new IP addresses. For illustration purposes, here is a look at the possible SPF record:

domain.com = “v=spf1 ip4:65.99.255.236 ip4:65.99.255.232 ip4:65.99.192.8 ip4:65.99.192.91 a:outbound.exchangedefender.com -all”

This record will restrict relaying for domain.com to ExchangeDefender outbound servers only (naturally you should include your own IP as well for non-smarthosted deliveries) but the above should get you started.

We don’t expect any issues as the new system has been in beta testing for a few months with no significant problems. Performance, logging, reporting and enforcement functionality that it delivers are far beyond comparison with the current service so we are really looking forward to it!

The New Soul of ExchangeDefender

At times in software development it becomes easier to make little optimizations to the process so long as the processing power becomes more affordable and available. Then one day you look up from what you’ve built on your monitor and see a wall of lights, indicators, pie charts and trends that use pretty pictures and colors to paint the portrait of the complexity that manage all the “little” issues you didn’t fix along the way. We are happy to report that after working around the clock for weeks we have regained a lot of control and a wall!

The story of ExchangeDefender is pretty simple. In the 90’s, we could not keep our Exchange servers up, fighting with relaying, viruses, hackers and platform instability became a daily frustration. So we figured, “Let’s put a box in front of it to just scan the junk and pass on valid stuff to Exchange” and it worked. The little AMD K5-100 workstation with an IDE hard drive and McAfee virus scanner did enough to keep our Exchange infrastructure running. Fast forward 12 years, the rise of SPAM, the rise of Malware, the pdf spam, the composite jpg spam, the rise of botnets and we have the ExchangeDefender of today, a 2,800 server network spread over 26 data centers and a full shift dedicated to just racking the new hardware and provisioning new nodes as our company and our load grow.

With data center space starting to shrink, power demands growing, power cost growing and the trend of SPAM increases turning vertical, we had a choice: change the way we manage SPAM flow or start breeding hamsters to generate power. Since nobody volunteered to clean the cage, we changed the way content is passed through and into our network.

The Old Way

Messages were accepted and scanned in the order they arrived with no bias shown for RBLs, origin or content. We took the message, checked it, and if it was clean we delivered it. With the SPAM loads constantly over the 95% range, legitimate messages had a 95% chance of being scanned with less urgency than their SPAM counterparts. To further complicate the issue, we scanned all messages equally: SPAM checks, heuristics, viruses, RBL presence, message integrity, the works. This created an immense demand for processing power, storage and network capacity.

Simply put, imagine a window of one second during which 10 messages were received and scanned sequentially from first to last. If the clean message was message #1, no worries. But if it was #10 in the scanning order it would have to wait for us to check for every minor chance that the nine messages before it had content anywhere from a pharmaceutical ad to banking fraud.

The New Way

We first changed the way we rely on our extensive knowledge of SPAM sources and senders and we created a system in which a bias is placed on messages coming from reputable senders. RBLs and statistical models are not perfect, that is why we will never bounce or delete an incoming messages without 100% certainty. Legitimate organizations can end up on RBLs from time to time, etc. But if we received over a million SPAM messages from a host in China with no reverse DNS, why should we treat the message #1,000,0001 from the same IP address with the same priority as the messages from hosts that constantly deliver legitimate mail that our users want to read? The good news is, we no longer do because messages are prioritized well before they hit the beef of ExchangeDefender.

Over the past month we have implemented a new SPAM filtering traffic shaping process that takes messages from known SPAM sources and puts them into a low priority queue to be scanned with far less of our arsenal than the messages from sources that are not as proven of SPAM heavens. So if you’ve mismanaged your infrastructure enough to end up in SpamCop, SpamHaus, SORBS, have an open relay, no reverse DNS the only thing we need to know is if the recipient trusts you. If they don’t, store/drop/delete, if they do, we scan for viruses, complex checks, custom filters and more.

This system allows us to enhance our scrubbing of unknown mail and keep you safe from more innovative threats while keeping SPAM stored just in case. Messages will still be accessible, still scanned. This new process shifts the burden of mail server management on the sender to keep their infrastructure in check while improving the security of our recipient customer. The era of trust on the Internet is long gone, mail servers that are mismanaged to the extent that they spew millions of SPAM messages, viruses and malware cannot be trusted not to have their logs scanned and malware routes replicated through the social means of replaying legitimate mail traffic to bypass filtering systems.

The Story So Far

So far, we have been able to improve nearly all tracked metrics of ExchangeDefender.

The level of SPAM detected is way up. The level of false positives (messages ExchangeDefender thought were SPAM but were released from quarantines by end users) is way, way down. The latency in delivery, scanning and processing are way down. Nearly everything we do has significantly improved over the past few weeks and while it took an enormous amount of hard work we feel it leaves the Internet a safer place for our clients and a far less profitable or even viable business for spammers.

Results

We have done our way to make sure the messages you receive get to you faster, safer while reducing the time needed to manage whitelists, review junk folders or worry about non-receipts. Our support has reflected this as well, despite February being the strongest sales month on record for ExchangeDefender, ever, we have had less support requests, ever!

Now, after a little deserved break, we turn to ExchangeDefender 4.x and think of ways we can use the new savings to make our service work for you even better.

As always, thank you for your business and thank you for trusting us to keep your inbox clean.

Update on ExchangeDefender Maintenance

Generally we keep ExchangeDefender maintenance on the NOC site but due to the extent of the maintenance done we felt it was important to place it here as well. Over the past 12 hours we have conducted urgent maintenance on the following systems:

1. ExchangeDefender reporting servers (ongoing, 8-10 hours remaining)
2. ExchangeDefender antispam engine (ongoing, up to 99.2% efficiency)
3. ExchangeDefender outbound network (complete)
4. ExchangeDefender global load balancing (complete)
5. ExchangeDefender cluster upgrades (complete)

It will take another 2-3 hours for the latency across the network to go down. The email reports will resume later today. The new outbound network hardware is online and active. The new reporting servers will go online this weekend along with the global network upgrade to Microsoft Exchange 2007 Service Pack 1.

We apologize for the inconvenience the additional mail latency has caused you and your clients, this will be the last major network upgrade for quite some time and the last major maintenance before the start of the ExchangeDefender 4.0. Most of the engine underneath of ExchangeDefender is running 4.x code already.

For updates on the ExchangeDefender maintenance please keep an eye on our NOC site at www.ownwebnow.com/noc

Enhanced SPAM filtering engine now online

Over the weekend we deployed a few enhancements to the antispam engine and hopefully starting tomorrow you should see far less junk in your Inbox. We have been testing this engine in parallel and have not seen a significant increase in false positives so we are positive this will be an improvement in overall performance and filtering rates.

As usual, if anything slips please forward it to spam@ownwebnow.com

Note on the forwards:

In the event that an obvious SPAM message gets through ExchangeDefender and into your mailbox, you can forward it to us at spam@ownwebnow.com for further inspection. There are times that the SPAM you received was from a source that is not known for spamming or that the message just did not contain enough obvious junk content to be filtered through. We also ask that you forward inline SMTP headers (in Outlook right click on the message in the listing, select Message Options and copy Internet headers) to us so we can verify that the message indeed passed through our network and that it was not released by one of your coworkers that it was cc’ed to (this happens all the time, someone releases the message from a familiar looking sender/subject and its a SPAM that was either cc’ed or bcc’ed to everyone in the organization). For 4.0, we are working on an Outlook addin.

If you are one of our partners and would like to hide the ownwebnow.com domain name, you can have messages sent to spam@exchangedefender.com. If you would rather give your customers your own address just create a mail enabled contact in your organization and forward it to either alias. Be advised that we may sometimes reply from that address in which case they will see our company name and our contact information – however, this is very infrequent! (mostly when people forward a few dozen identical items to which they have obviously subscribed to, such as popular newsletters, shopping sites, etc)