At 10:30 AM Eastern we will be shutting off the Ahsay OBS service on Backup74 to install the latest updates to the OBS platform. Service is expected to be down for less than 15 minutes.
Update 10:35 AM Eastern: The update has been completed. Service has been restored to Backup74.
Update 10:31 AM Eastern: The service has been shut off for the update.
Over the weekend, the Tomcat web service on backup74 stopped answering new requests. We’ve opened up a ticket with Ahsay and was provided with a hotfix to install. We are going to try to make the changes to the Tomcat config files to restore service before installing the hotfix patch. Service is expected to be functional within the hour.
Update 10:50pm Eastern: Service to backup74 has been restored without installing the hotfix. We are planning to install a stable patch upgrade to the server later in the week.
The Ahsay OBS service has been stopped on Backup74 while we move users around on the volumes.
HUEY and DEWEY are currently offline. While working with Microsoft to troubleshoot an authentication issue, the servers went lights out. We’ve requested a reboot at the DC an service is expected to be restored within 15 minutes.
Update 4:57 PM: Service has been restored.
Update 4:36 PM: The server took a bit longer than expected to recover from the reboot, however the information store timed out when mounting. We are rebooting the server again to address the issue.
We’ve received complaints from users about inaccessibility to access backup74. We will be rebooting backup74 shortly to resolve the issue.
We will be rebooting two of our US Exchange servers at 8:00 PM Eastern. HUEY and DEWEY will be unavailable for 15 minutes during the reboot.
Update 8:20 PM: Service on Dewey has been restored.
Update 8:13 PM: Huey’s services have been restored. Dewey hasn’t rebooted yet.
Update 8:05 PM: The reboot process has begun. Service will be restored momentarily.
Our Australian Exchange 2007 server is currently down hard. The datacenter is experiencing a networking issue and we are working to resolve the issue.
Update 6:15 PM Eastern: The server is back online. The issue was caused by the network ISP change at the data center and the downtime was unexpected. We will be changing the IP address of Australia to match the new network configuration. This change will not require any modification by our partners.
Our shared POP3 mailbox server mail1 is currently under an extreme load and slowly processing messages. We are working to alleviate the queue load and restore mail routing.
Update 1:11 PM: The load on the mail1 server has been reduced and new mail is being processed successfully. We anticipate issues for some users connecting as all users are concurrently connecting to download mail.
We will be rebooting our US Exchange Server Scrooge today to relieve resource usage on the server. The reboot time has not been established yet however this post will be update when a time has been set.
Update 12:25 AM: Scrooge has been rebooted and is back online.
Update 5:22 PM: We have delayed the scheduled reboot until Midnight tonight due to concerns of unavailability by our West coast customers.
Update 2:15 PM: We have delayed the scheduled reboot until 9pm tonight in order to run extra maintenance on the server.
Update 10:50 AM: We have scheduled the reboot time to occur at 5pm Eastern today.
Backup74 will be temporarily offline from 11:40am – 1:00 pm as we move accounts to new volumes. This is estimated to take an hour.
Update 2:30 PM Eastern: Service has been restored.
Over the weekend our replication servers experienced a larger than normal lag in replicating statistical data from our Dallas Colo. This caused quarantine reports to show up as empty as there was no new data replicated when the reports were generated. Reporting is expected to be fully functional by the next scheduled cycle.
The Australian exchange server is currently unavailable. There is an issue with the datacenter and it has gone beyond the normal 1 hour SLA. We are doing everything in our power to restore service immediately.
Update 1:34 PM Eastern: The Australian server is back online. The issue was caused by a failure in the core network of the datacenter. Service is now 100% operational.
On Friday, September 4th 2009 at 10:00 AM Eastern we will be changing the IP address for our Australia Exchange server due to changes in the network layout in the data center. Before the change we will lower the TTL time on the DNS record (exchange.australia.exchangedefender.com) to minimize the chance of downtime/unavailability.
Users on the Australia server should not have to make any changes after the IP address change for service to continue working.
Update 12:10 PM Eastern: The IP change has been delayed until later today. We will update this post before the change occurs.