The outages in the northeast are slowing down deliveries to some customers but have been improving throughout the morning as more customers restore power. We’re increasing capacity as we speak to adjust for the increased rejection/reattempt rates.
August 29, 2011
Our Engineers have informed us that they will be performing maintenance work on our Backup73 server. They have decided to perform this maintenance during business hours as most of the scheduled jobs are set begin before or after regular business hours. We thank you for your patience with us will the maintenance is performed and apologize for any inconvenience this issue may cause.
August 26, 2011
We are currently receiving alerts from our monitoring software that tracks average mail delivery time in ExchangeDefender that nodes in Dallas are experiencing larger than normal queues.
We’ve identified the issue with our logging server that receives the SMTP transaction logs from each message as it transverses the network. We are currently working to alleviate the pressure and rebalance the message load across all nodes.
Users should expect to see up to a 15 minute delay on messages until 1:50 PM Eastern when delivery services are back to speed.
Update 1:44 PM: The logs server has been reactivated and we are watching traffic pour in. Just to note, the admin.exchangedefender.com site has been offlined while the log server is finishing the start up check.
Update 2:00 PM: Mail delivery is catching back up, however, the mail logs tables for domains are being checked for consistency before being brought back online. We expect the admin.exchangedefender.com site to be up within 30 minutes
Update 2:20 PM: Access to admin.exchangedefender.com has been restored. The inbound servers are still actively processing the queue loads
Update 4:35 PM: All inbound / outbound nodes have flushed their queues and are operating at normal processing time. We will continue to work on resolving the pressure issues seen by the logs server throughout the weekend.
We’ve stopped the OBS service on backup73 as we rebalance users across the drives. Maintenance is expected to last for 45 minutes and service is expected to resume at 1:30 PM Eastern.
August 19, 2011
We’re performing maintenance on Backup73. We expect it to last 3 hours. We’re performing the maintenance during business hours as our usage records show most backup jobs are scheduled for before and after business hours.
August 11, 2011
99% of all of our services have been restored. Unfortunately, we have 3 servers that are still affected by the outage. Remembering that it was a power outage and Windows Server’ capability of handling such outages is not on par with Linux, we’ve had some servers take longer to come back online. Services currently affected:
Dewey MBOX2 – It’s a secondary mailbox server on the DEWEY cluster, as such its a small user count but if your clients are on it, their mailbox is inaccessible.
Daisy – Legacy Exchange 2003 server
VS4 – One of our Virtual servers
These services affect a small but equally important portion of the client base, as such we’re all hands on deck on restoring services on these servers ASAP. For the Hosted Exchange users still affected please remember LiveArchive is online, if you’re having any issues authenticating to it or otherwise please open a support request and we’ll get you back online on LiveArchive as quickly as possible.
August 10, 2011
All services have been restored, there is still spooled mail being delivered but new mail is being delivered in real time. With that said our CEO Vlad Mazek held a GoToWebinar outlining all the important facts about this event. We invite you to listen to the points covered as well as use the provided information the in the PPTx. And once again we’d like to apologize for the inconvenienced caused to you and your customers.
We’re employing an emergency recovery plan, essential services will be coming online but they’re not running on their full infrastructure. Please expect services to come back online slowly as services are running on our emergency failover (and the datacenter has NOT fully restored the outage). Latency on the servers, that are online, is expected as customers try to reconnect to the servers. We’ll provide more details as they come available. We will be keeping folks available on the phone to give the information below live. Again this is a continuation of the information found on our twitter feed at twitter.com/XDNOC
We’re currently polling all the data from the twitter feed to present it all in one view.
Below you will find a redacted version of the twitter feed so updates can be read without cross conversations from partners during the outage, please note (like headers these chronologically from the bottom up):
Services have been restored to our Dallas DC. We are going through checks now to resume normal operation.
Our failover systems are kicking in and service is restored to support portals, web sites, outbound ExchangeDefender, louie, & rockerduck.
Please note:THESE ARE EMERGENCY FAILOVER systems, not the real thing. Services will be restored by the utility/power/electricians/etc.
ExchangeDefender outbound service has been re-established as well as Exchange 2010 LOUIE and ROCKERDUCK
DC/Electrical teams have established a provisional return of services for 6:30 PM EST. We will update this advisory at that time.
We are working with the DC to move around equipment for a temporary solution. OWN sites and ED outbound will be up soon.
DC Update “There has been an issue affecting one of our 6 service entrances. The actual ATS is having an issue and all vendors are on site.”
The datacenter staff has confirmed an outage with the power plant and has individuals on staff attempting to redirect power around the core
Service is still affected and the latest from the DC reports that the backup EPO overloaded and tripped. The issue is still being addressed
The issue has been identified as power related in the DC. Services are slowly coming online. We will update when service is fully restored.
Routing issues in Dallas at the moment. If you’re having issues accessing and have Level3 in your way its going to take some patience today.
In addition we’re polling updates from our DC’s status to ensure that we’re providing as much detail as possible on the outage itself (times are CST):
Our team and electricians are working diligently to get the temporary ATS installed, wired and tested to allow power to be restored. As the ATS involves high-voltage power, we are following the necessary steps to ensure the safety of our personnel and your equipment housed in our facility.
Based on current progress the electricians expect to start powering the equipment on between 6:15 – 7:00pm Central. This is our best estimated time currently. We have thoroughly tested and don’t anticipate any issues in powering up, but there is always the potential for unforeseen issues that could affect the ETA so we will keep you posted as we get progress reports. Our UPS vendor has checked every UPS, and the HVAC has checked every unit and found no issues. Our electrical contractor has also checked everything.
We realize how challenging and frustrating that it has been to not have an ETA for you or your customers, but we wanted to ensure we shared accurate and realistic information. We are working as fast as possible to get our customers back online and to ensure it is done safely and accurately. We will provide an update again within the hour.
While the team is working on the fix, I’ve answered some of the questions or comments that have been raised:
1. ATSs are pieces of equipment and can fail as equipment sometimes does, which is why we do 2N power in the facility in case the worst case scenario happens.
2. There is no problem with the electrical grid in Dallas or the heat in Dallas that caused the issue.
3. Our website and one switch were connected to two PDUs, but ultimately the same service entrance. This was a mistake that has been corrected.
4. Bypassing an ATS is not a simple fix, like putting on jumper cables. It is detailed and hard work. Given the size and power of the ATS, the safety of our people and our contractors must remain the highest priority.
5. Our guys are working hard. While we all prepare for emergencies, it is still quite difficult when one is in effect. We could have done a better job keeping you informed. We know our customers are also stressed.
6. The ATS could be repaired, but we have already made the decision to order a replacement. This is certainly not the cheapest route to take, but is the best solution for the long-term stability.
7. While the solution we have implemented is technically a temporary fix, we are taking great care and wiring as if it were permanent.
8. Colo4 does have A/B power for our routing gear. We identified one switch that was connected to A only which was a mistake. It was quickly corrected earlier today but did affect service for a few customers.
9. Some customers with A/B power had overloaded their circuits, which is a separate and individual versus network issue. (For example, if we offer A/B 20 amp feeds and the customer has 12 amps on each, if one trips, the other will not be able to handle the load.)
As you could imagine, this is the top priority for everyone in our facility. We will provide an update as quickly as possible.
Thank you for your patience as we work to address the ATS issue with our #2 service entrance. We apologize for the situation and are working as quickly as possible to restore service.
We have determined that the repairs for the ATS will take more time than anticipated, so we are putting into service a backup ATS that we have on-site as part of our emergency recovery plan. We are working with our power team to safely bring the replacement ATS into operation. We will update you as soon as we have an estimated time that the replacement ATS will be online.
Later, once we have repaired the main ATS, we will schedule an update window to transition from the temporary power solution. We will provide advance notice and timelines to minimize any disruption to your business.
Again, we apologize for the loss of connectivity and impact to your business. We are working diligently to get things back online for our customers. Please expect another update within the hour.
It has been determined that the ATS will need repairs that will take time to perform. Fortunately Colo4 has another ATS that is on-site that can be used as a spare. Contractors are working on a solution right now that will allow us to safely bring that ATS in and use it as a spare while that repair is happening.
That plan is being developed now and we should have an update soon as to the time frame to restore temporary power. We will need to schedule another window when the temp ATS is brought offline and replaced by the repaired ATS.
There has been an issue affecting one of our 6 service entrances. The actual ATS (Automatic Transfer Switch) is having an issue and all vendors are on site. Unfortunately, this is affecting service entrance 2 in the 3000 Irving facility so it is affecting a lot of the customers that have been here the longest.
The other entrance in 3000 is still up and working fine as well as the 4 entrances in 3004. Customers utilizing A/B should have access to their secondary link. It does appear that some customers were affected by a switch that had a failure in 3000. That has been addressed and should be up now.
We will be updating customers as we get information from our vendors so that they know the estimated time for the outage. Once this has been resolved we also distribute a detailed RFO to those affected.
Our electrical contractors, UPS maintenance team and generator contractor are all on-site and working to determine what the best course of action is to get this back up.
August 1, 2011
One of our pop/imap servers used for WebHosting freebie mailboxes primarily appears to have been attacked. We’re resolving the issue as we speak there may be some delays found in mail flow from today. Measures have been taken to ensure the root cause does not repeat itself.
We have received reports of a 5 minute window where there was packet loss to our Datacenter in Dallas. It appears this issue was on the ISP level of the Network and has been resolved in its entirety.
Additional Info from our DC:
The network issues experienced today began at approximately 12:22pm CST were caused by an issue within Level3’s network. This issue affected Level3 customers nationwide, and is not isolated to Colo4.
July 26, 2011
We’re currently investigating an issue with one of the cas nodes on LOUIE. Currently all metrics appear in good health but we’re investigating end user reports of erratic connectivity.
[2:10 PM] We’ve tracked the issue down to slugglish DC lookups. We’re working to address these as quickly as possible.
[2:41 PM] We’re still working on address the Domain Controller lookup issue for the Outlook clients. This is a courtesy update to let our partners know we’re still working on the issue urgently.
[2:58 PM]We’ve traced the issue to a problem with the HUB role interfacing with the DOMAIN Controller, we’re putting a change through and should have more information in the next 10 minutes.
Please remember that LiveArchive is accessible.
[3:06 PM]There are reports of mail delivery delays which, unfortunately will naturally happen after clients begin to reconnect and mail is pushed from local clients. We are also going to restart the information store on LOUIEMBOX2 as that seems to be the only queues which are in excess.
[3:17 PM] We will also restart the load balancer to rebalance connections before the information store comes back up.
[3:27 PM] We are remounting the databases for LOUIEMBOX2 and will shortly resume mail flow.
[3:38 PM] We have resumed mail flow for users on LOUIEMBOX1. The database files are replaying the log files from the past half hour to ensure all data is present before we resume service.
[3:48 PM] Service has been fully restored and we are monitoring the traffic as the queues get processed
We’ve concluded our investigation and the root cause of the latency on LOUIE, which prompted us to stop service was a running backup job on MBOX2 from 12:30 AM. The jobs are configured to terminate if they do not complete by 7AM Eastern, which seems to have not occurred. We will be working with the backup vendor to resolve the issue with backup jobs not terminating. Since the original report from partners and our monitoring was related to RPC / OWA / Outlook performance, we started with the CAS servers and worked backwards. We will reevaluate our monitoring checks in an attempt to avoid mistaking CAS related latency and overall RPC and network call latency.
July 25, 2011
Today we will be conducting fail over scenarios for our PBX system in response to the outage from last week. Throughout the test we will be disabling our main PBX to allow calls to fall over to our failover numbers. Clients should not experience any issues in reaching parties on our end during the PBX testing.
July 20, 2011
We’re experiencing packet loss in our primary DC. It appears that services are beginning to come online but we have not yet considered the issue resolved.
[4:57 PM] All services have been restored with the Exception of LiveArchive. We should have this service online shortly.
July 18, 2011
We’ve received alerts about queues on Rockerduck backing up to 100 messages. Upon investigation, the Edge server was stuck in a reboot phase after Windows updates. We’ve restarted the server and mail is once again flowing on Rockerduck.
We’ve been working on Matilda throughout the night. It appears to be an ISP level issue as we’re awaiting word from our AUS DC on further details. Please remember to use LiveArchive during these times. It is available at https://livearchive.exchangedefender.com
07:14 am. This is a courtesy update, unfortunately we still have not received a resolution to the issue from our DC. Please continue to use LiveArchive.
This issue has been resolved.
June 22, 2011
12:28 PM (EST) on 6-22 Our engineers discovered an issue on a couple of individual nodes within the ExchangeDefender network that may have caused some temporary delay to both Inbound and Outbound messages. As of this moment our ExchangeDefender Engineers are working diligently on resolving this issue. From all of us here at OWN Web Now we would like to offer all of our partners our sincerest apologize for this unforeseen issue we experienced today but please rest assured the issue will be resolved.
June 2, 2011
We’re going to perform an emergency reboot on DEWEY. While we’re unable to replicate the reported issues connecting to profiles on DEWEY the reports are enough that we need to ensure that all of our clients can access their profiles. We’ll be doing this in 10 minutes. We should be back online within 15 minutes of that timeframe.
May 27, 2011
At 11:00 PM Eastern we will be rebooting the LOUIEMBOX1 server to repair the windows server backup feature. During the reboot, mailboxes hosted on LOUIEMBOX1 will experience a brief service interruption as the server reboots. We understand that this is short notice, however in the interest of maintaining a healthy backup history, we must reboot the server before the next scheduled backup.
Update 11:30 PM Eastern: We are now rebooting the LOUIEMBOX1 server.
Update 11:45 PM Eastern: Service has been restored to users on LOUIEMBOX1
May 26, 2011
We are performing Active Directory maintenance on the LIVEARCHIVE network in order to resolve issues with database latency. During maintenance, users may receive warning about their mailbox being unavailable. We appreciate your patience as we make the necessary modifications in order to restore service to 100% operation.
May 23, 2011
We’re currently investigating some issues with a specific database on DEWEY there may be a quick service interruption while we research this.
[9:57 am]It appears that the system will require a restart after the reports of slowness have increased please stand by. We’ll be doing so in 10 minutes. Please remember LiveArchive is available during these windows.
[10:21 am]Mail is now delivering as expected, please allow time for the queued mail to deliver. Thank you for your patience.
May 20, 2011
Huey is experiencing some issues with some client connection services. We’ve gone through all of the services carefully to avoid this. However, it is necessary to complete this in the next 5 minutes.
Work on Huey is continuing. Please remember LiveArchive is available during these windows of time.
« Newer Posts — Older Posts »
Powered by WordPress