Our Engineers have informed us that they will be performing maintenance work on our Backup73 server. They have decided to perform this maintenance during business hours as most of the scheduled jobs are set begin before or after regular business hours. We thank you for your patience with us will the maintenance is performed and apologize for any inconvenience this issue may cause.

We are currently receiving alerts from our monitoring software that tracks average mail delivery time in ExchangeDefender that nodes in Dallas are experiencing larger than normal queues.

We’ve identified the issue with our logging server that receives the SMTP transaction logs from each message as it transverses the network. We are currently working to alleviate the pressure and rebalance the message load across all nodes.

Users should expect to see up to a 15 minute delay on messages until 1:50 PM Eastern when delivery services are back to speed.

Update 1:44 PM: The logs server has been reactivated and we are watching traffic pour in. Just to note, the admin.exchangedefender.com site has been offlined while the log server is finishing the start up check.

Update 2:00 PM: Mail delivery is catching back up, however, the mail logs tables for domains are being checked for consistency before being brought back online. We expect the admin.exchangedefender.com site to be up within 30 minutes

Update 2:20 PM: Access to admin.exchangedefender.com has been restored. The inbound servers are still actively processing the queue loads

Update 4:35 PM: All inbound / outbound nodes have flushed their queues and are operating at normal processing time. We will continue to work on resolving the pressure issues seen by the logs server throughout the weekend.

We’ve stopped the OBS service on backup73 as we rebalance users across the drives. Maintenance is expected to last for 45 minutes and service is expected to resume at 1:30 PM Eastern.

We’re performing maintenance on Backup73. We expect it to last 3 hours. We’re performing the maintenance during business hours as our usage records show most backup jobs are scheduled for before and after business hours.

99% of all of our services have been restored. Unfortunately, we have 3 servers that are still affected by the outage. Remembering that it was a power outage  and Windows Server’ capability of handling such outages is not on par with Linux, we’ve had some servers take longer to come back online. Services currently affected:

Dewey MBOX2 – It’s a secondary mailbox server on the DEWEY cluster, as such its a small user count but if your clients are on it, their mailbox is inaccessible.

Daisy – Legacy Exchange 2003 server

VS4 – One of our Virtual servers

These services affect a small but equally important portion of the client base, as such we’re all hands on deck on restoring services on these servers ASAP. For the Hosted Exchange users still affected please remember LiveArchive is online, if you’re having any issues authenticating to it or otherwise please open a support request and we’ll get you back online on LiveArchive as quickly as possible.

Final Update:

All services have been restored, there is still spooled mail being delivered but new mail is being delivered in real time. With that said our CEO Vlad Mazek held a GoToWebinar outlining all the important facts about this event. We invite you to listen to the points covered as well as use the provided information the in the PPTx. And once again we’d like to apologize for the inconvenienced caused to you and your customers.

Webinar Powerpoint

Webinar Video (Requires GoToWebinar codec)

Previous:

We’re employing an emergency recovery plan, essential services will be coming online but they’re not running on their full infrastructure. Please expect services to come back online slowly as services are running on our emergency failover (and the datacenter has NOT fully restored the outage). Latency on the servers, that are online, is expected as customers try to reconnect to the servers. We’ll provide more details as they come available. We will be keeping folks available on the phone to give the information below live. Again this is a continuation of the information found on our twitter feed at twitter.com/XDNOC

We’re currently polling all the data from the twitter feed to present it all in one view.

Below you will find a redacted version of the twitter feed so updates can be read without cross conversations from partners during the outage, please note (like headers these chronologically from the bottom up):

Our failover systems are kicking in and service is restored to support portals, web sites, outbound ExchangeDefender, louie, & rockerduck.

Please note:THESE ARE EMERGENCY FAILOVER systems, not the real thing. Services will be restored by the utility/power/electricians/etc​.

ExchangeDefender outbound service has been re-established as well as Exchange 2010 LOUIE and ROCKERDUCK

DC/Electrical teams have established a provisional return of services for 6:30 PM EST. We will update this advisory at that time.

We are working with the DC to move around equipment for a temporary solution. OWN sites and ED outbound will be up soon.

DC Update “There has been an issue affecting one of our 6 service entrances. The actual ATS is having an issue and all vendors are on site.”

The datacenter staff has confirmed an outage with the power plant and has individuals on staff attempting to redirect power around the core

Service is still affected and the latest from the DC reports that the backup EPO overloaded and tripped. The issue is still being addressed

The issue has been identified as power related in the DC. Services are slowly coming online. We will update when service is fully restored.

Routing issues in Dallas at the moment. If you’re having issues accessing and have Level3 in your way its going to take some patience today.

In addition we’re polling updates from our DC’s status to ensure that we’re providing as much detail as possible on the outage itself (times are CST):

Current Update

Our team and electricians are working diligently to get the temporary ATS installed, wired and tested to allow power to be restored. As the ATS involves high-voltage power, we are following the necessary steps to ensure the safety of our personnel and your equipment housed in our facility.

Based on current progress the electricians expect to start powering the equipment on between 6:15 – 7:00pm Central. This is our best estimated time currently. We have thoroughly tested and don’t anticipate any issues in powering up, but there is always the potential for unforeseen issues that could affect the ETA so we will keep you posted as we get progress reports. Our UPS vendor has checked every UPS, and the HVAC has checked every unit and found no issues. Our electrical contractor has also checked everything.

We realize how challenging and frustrating that it has been to not have an ETA for you or your customers, but we wanted to ensure we shared accurate and realistic information. We are working as fast as possible to get our customers back online and to ensure it is done safely and accurately. We will provide an update again within the hour.

While the team is working on the fix, I’ve answered some of the questions or comments that have been raised:

1. ATSs are pieces of equipment and can fail as equipment sometimes does, which is why we do 2N power in the facility in case the worst case scenario happens.

2. There is no problem with the electrical grid in Dallas or the heat in Dallas that caused the issue.

3. Our website and one switch were connected to two PDUs, but ultimately the same service entrance. This was a mistake that has been corrected.

As you could imagine, this is the top priority for everyone in our facility. We will provide an update as quickly as possible.

Previous:

14:53

Thank you for your patience as we work to address the ATS issue with our #2 service entrance. We apologize for the situation and are working as quickly as possible to restore service.

We have determined that the repairs for the ATS will take more time than anticipated, so we are putting into service a backup ATS that we have on-site as part of our emergency recovery plan. We are working with our power team to safely bring the replacement ATS into operation. We will update you as soon as we have an estimated time that the replacement ATS will be online.

Later, once we have repaired the main ATS, we will schedule an update window to transition from the temporary power solution. We will provide advance notice and timelines to minimize any disruption to your business.

Again, we apologize for the loss of connectivity and impact to your business. We are working diligently to get things back online for our customers. Please expect another update within the hour.

13:34

It has been determined that the ATS will need repairs that will take time to perform. Fortunately Colo4 has another ATS that is on-site that can be used as a spare. Contractors are working on a solution right now that will allow us to safely bring that ATS in and use it as a spare while that repair is happening.

That plan is being developed now and we should have an update soon as to the time frame to restore temporary power. We will need to schedule another window when the temp ATS is brought offline and replaced by the repaired ATS.

13:05

There has been an issue affecting one of our 6 service entrances. The actual ATS (Automatic Transfer Switch) is having an issue and all vendors are on site. Unfortunately, this is affecting service entrance 2 in the 3000 Irving facility so it is affecting a lot of the customers that have been here the longest.

The other entrance in 3000 is still up and working fine as well as the 4 entrances in 3004. Customers utilizing A/B should have access to their secondary link. It does appear that some customers were affected by a switch that had a failure in 3000. That has been addressed and should be up now.
This is not related to the PDU maintenance we had in 3004 last night. Separate building, service entrance, UPS, PDU, etc.

We will be updating customers as we get information from our vendors so that they know the estimated time for the outage. Once this has been resolved we also distribute a detailed RFO to those affected.

Our electrical contractors, UPS maintenance team and generator contractor are all on-site and working to determine what the best course of action is to get this back up.

One of our pop/imap servers used for WebHosting freebie mailboxes primarily appears to have been attacked. We’re resolving the issue as we speak there may be some delays found in mail flow from today. Measures have been taken to ensure the root cause does not repeat itself.

We have received reports of a 5 minute window where there was packet loss to our Datacenter in Dallas. It appears this issue was on the ISP level of the Network and has been resolved in its entirety.

Additional Info from our DC:

The network issues experienced today began at approximately 12:22pm CST were caused by an issue within Level3’s network. This issue affected Level3 customers nationwide, and is not isolated to Colo4.

We’re currently investigating an issue with one of the cas nodes on LOUIE. Currently all metrics appear in good health but we’re investigating end user reports of erratic connectivity.

[2:10 PM] We’ve tracked the issue down to slugglish DC lookups. We’re working to address these as quickly as possible.

[2:41 PM] We’re still working on address the Domain Controller lookup issue for the Outlook clients. This is a courtesy update to let our partners know we’re still working on the issue urgently.

[2:58 PM]We’ve traced the issue to a problem with the HUB role interfacing with the DOMAIN Controller, we’re putting a change through and should have more information in the next 10 minutes.

Please remember that LiveArchive is accessible.

[3:06 PM]There are reports of mail delivery delays which, unfortunately will naturally happen after clients begin to reconnect and mail is pushed from local clients. We are also going to restart the information store on LOUIEMBOX2 as that seems to be the only queues which are in excess.

[3:17 PM] We will also restart the load balancer to rebalance connections before the information store comes back up.

[3:27 PM] We are remounting the databases for LOUIEMBOX2 and will shortly resume mail flow.

[3:38 PM] We have resumed mail flow for users on LOUIEMBOX1. The database files are replaying the log files from the past half hour to ensure all data is present before we resume service.

[3:48 PM] Service has been fully restored and we are monitoring the traffic as the queues get processed

We’ve concluded our investigation and the root cause of the latency on LOUIE, which prompted us to stop service was a running backup job on MBOX2 from 12:30 AM. The jobs are configured to terminate if they do not complete by 7AM Eastern, which seems to have not occurred. We will be working with the backup vendor to resolve the issue with backup jobs not terminating. Since the original report from partners and our monitoring was related to RPC / OWA / Outlook performance, we started with the CAS servers and worked backwards. We will reevaluate our monitoring checks in an attempt to avoid mistaking CAS related latency and overall RPC and network call latency.

Today we will be conducting fail over scenarios for our PBX system in response to the outage from last week. Throughout the test we will be disabling our main PBX to allow calls to fall over to our failover numbers. Clients should not experience any issues in reaching parties on our end during the PBX testing.

We’re experiencing packet loss in our primary DC. It appears that services are beginning to come online but we have not yet considered the issue resolved.

[4:57 PM] All services have been restored with the Exception of LiveArchive. We should have this service online shortly.

We’ve received alerts about queues on Rockerduck backing up to 100 messages. Upon investigation, the Edge server was stuck in a reboot phase after Windows updates. We’ve restarted the server and mail is once again flowing on Rockerduck.