Sat Sun Mon Tue Wed
ExchangeDefender Inbound
ExchangeDefender Outbound
ExchangeDefender Applications
Exchange 2010 Hosting
Exchange 2007 Hosting
Offsite Backups
Web Hosting
Blackberry Enterprise Server
SharePoint 2007
SharePoint 2010
April – 2008 – ExchangeDefender Network Operations

April 29, 2008

Network event in Dallas

Filed under: Network Operations — admin @ 7:54 am

We are currently investigating a network event in our Dallas region network centers, since approximately 5 AM EST we have been receiving complaints about network connectivity and availability. There are currently no outages and there have been no outages but certain customers are unable to reach the services on our network.

If you are experiencing an issue, please open a support request and include a traceroute to the service you are trying to reach (ex: 65.99.255.50).

We will update this ticket as soon as we have further information. 

Update (9:05 AM EST, 14:05 GMT): We believe the network issue some of our customers have experienced has been resolved. Particularly affected were some of our UK customers (not BT) and local customers with Level 3 connectivity. Although we see the traffic back up at usual levels, it may take about an hour or so until all the mail catches up and gets delivered.


April 23, 2008

Investigating problems with roadrunner network

Filed under: ExchangeDefender, Virtual Hosting — admin @ 1:40 pm

We are currently working with RoadRunner (formerly Time Warner, AOL) service provider in United States, they are experiencing issues with their SMTP servers and randomly rejecting SMTP traffic. Currently mail is flowing through but some is bouncing back from them due to a reason they are still trying to narrow down. We will update when we have further information or a resolution.

This issue affects our entire global network, and some external sites we have tested.

Update: 6:34 PM EST: Even though we have not been officially updated, the problems with RoadRunner appear to have been resolved. 


April 6, 2008

Exchange 2007 SP1 upgrade underway

Filed under: Exchange Hosting — admin @ 8:01 am

We will be starting the SP1 upgrade on our systems in roughly two hours (10 AM EST, -5 GMT) and expect to have all operations completed by noon. To minimize the surprises and potential conflicts, the entire cluster and all its members will be patched at once. That unfortunately does mean a bit of total outage, but it does minimize the chance of anything breaking in the process. Hosted Exchange has been one of our most solid products and we look forward to keeping it that way.

Update: Exchange 2007 Service Pack 1 is now up and running, systems have been imaged and we are all done.


April 2, 2008

Bug in ExchangeDefender Mail Report Counts

Filed under: ExchangeDefender — admin @ 10:28 am

Earlier today we identified a major bug in the system that was used to generate statistics for SPAM email daily and intraday reports for some users. Although the issue affected only a few thousand people, I have chosen to pull it out of the production systems to avoid further confusion and lack of email report integrity. As soon as the bug fix is tested thoroughly, we will be placing it back into production. In the meantime, you will not see “Non SPAM Mail” total under statistics anymore.

Problem Details

ExchangeDefender daily and intraday reports are built using SQL queries against the mail log database. There are three queries executed for each report, one to obtain the SPAM messages, one to obtain SureSPAM messages and one to obtain the total number of rows in the table, both SPAM, SureSPAM and messages let through. Each SPAM query is executed within a check that verifies if the user settings are to store/quarantine junk mail because otherwise we have nothing to report if the messages are delivered and/or deleted. Totals for SPAM and SureSPAM are calculated within the respective settings check blocks. For example:

if (User Quarantines SPAM)
{
    Get SPAM Total
    Print Report
}

if (UserQuarantinesSureSPAM)
{
    Get SureSPAM Total
    Print Report
}

Get Total Messages Received

Not SPAM = Total Messages – SPAM Total – SureSPAM Total

The problem with the Not SPAM count came in if the user did not store/quarantine their SPAM or SureSPAM which would mean the blocks of code that calculate the totals for the group would not get executed. The Non SPAM total would not get the correct amount of SPAM or SureSPAM subtracted from it and it would appear to the user as if they were missing messages because they surely were not receiving the amount that the report had indicated.

Stupidity Details

We figured we could save a few cycles by not running an extra query and total if the users did not store/quarantine SPAM or SureSPAM. Unfortunately, the equation for Not SPAM did not take that check into account and instead of subtracting the correct totals for SPAM and SureSPAM which are still logged but never reported, we were subtracting a zero thereby inflating the Not SPAM total for certain users.

The good news is that it was simple enough to fix, sorry for all the frustration that has come out of this as both my support, my partners and my clients were seeing different results across the network. Considering I am responsible for the above I apologize for all the problems this has caused for you.

Vlad Mazek


Powered by WordPress