At 9:40 we experienced a power event in the Dallas DC affecting ExchangeDefender internal infrastructure. This affected performance and availability of admin.exchangedefender.com, this web site, our support portals and Exchange hosting infrastructure.
By 9:50 everything appears to be back to normal but as these are highly utilized servers the performance may be affected for another 10-15 minutes as we run our tests to make sure everything is back to normal.
We apologize for the inconvenience and will update this ticket shortly.
Tonight (2/22/10) around 6PM Eastern we will begin to implement split GALs on the Australia Exchange 2007 server. Between 6PM and 8PM Eastern there is expected to be brief interruptions in service as the new security policies are implemented.
The Ahsay service on backup74 has been temporarily stopped as we migrate users to the newer volumes. Service is expected to be restored by 4pm Eastern.
The Dewey BES service has expierenced a few random issues since Microsoft patching on the 10th. In an attempt to revert the changes, we’re in the process of removing the updates before restoring from a backup. Service on DEWEY BES will not be interrupted throughout the extended maintenance of the BES server.
Update 10:50 AM Eastern: The BES service for Dewey has been restored 100%. We found an issue with two security updates, including KB977914
The Ahsay OBS service on backup74 is currently stopped as we migrate users to larger volumes. Service is expected to be restored by our peak backup time of 12 am Eastern.
At 8 PM Eastern tonight we will be taking backup74 offline to reattempt storage installation. The installation and provisioning is expected to take up an hour and a half and service will be automatically restored after the upgrade is complete.
Update 10:07 PM Eastern: The new controller install failed again. We’ve hooked up a temporary drive to alleviate the space issue while we order a new controller. Service is expected to be restored by 11 PM Eastern.
Update 10:58 PM Eastern: Service has been restored.
We’ve had reports from a few clients that they are having issues accessing our Exchange 2007 server DEWEY. We are currently investigating into this issue.
Update 11:20 AM: Service on DEWEY has been running 100% over an hour. It seems close to 10 PM Eastern on 2/4/2010, IIS had issues compiling the metadatabase..which ultimately did not throw any errors in windows logs or in any of our monitoring software. We ran a quick repair on the metadatabase soon after being notified of service outage and restarted IIS which resolved the issue.
Backup74 will be going offline from 6pm Eastern till 10pm Eastern today (2/2/10) for extended maintenance. We will be installing 4TB of additional storage to the server.
Update 8:20 PM Eastern: After installing the new controller the server was unable to boot property. We’ve removed the new controller and ordered a new replacement.
We have received reports from a few partners that ExchangeDefender users are not receiving mail released from Quarantine. We are investigating into this issue and assure that no mail has been lost. If you experience issues in releasing mail from quarantine please open a support ticket at support.ownwebnow.com including the senders email address , recipients email address, and time stamp.
Update – Noon EST – We have resolved the issue that caused the release mechanism to fail, the problem has been fixed and as of about 11:30 the full functionality has been restored. Now a word from Vlad Mazek, CEO:
As we have widely blogged, we have been migrating from the old systems and old software with the release of ExchangeDefender 5. The final step, which was taken this weekend, involved removal of the old infrastructure at the core of the command center and as is usually the case with complex software, some middleware got in the way.
Just release the messages again and they will flow in. Furthermore, if you’re not familiar with https://admin.exchangedefender.com this would have been a perfect time to check it out, in ExchangeDefender 5 we do offer full access to SPAM so you can view it live without releasing – you can even reply directly from the site!
Again, my apologies about the inconvenience this has caused.
Cell: (407) 536-VLAD