In a few moments we will begin the upgrade of our network to ExchangeDefender 7. The process will be seamless to our users and we don’t expect any issues. We are prepared for the unexpected issues and here are some suggestions:
1. Sign up for the Network Operations Blog updates (http://www.ownwebnow.com/noc)
2. If you don’t like web / RSS feeds, check out our Twitter feed (http://www.twitter.com/XDNOC)
3. Make sure you have distributed the User Guides to your end users so they can navigate the new User Interface. Link to end user collateral is here.
During The Upgrade
Check here first. We will be posting updates here and to the Twitter feed as soon as we identify and document a bug.
If you see an issue that has not been addressed, open a support request at https://support.ownwebnow.com
Call us at +1 (407) 209-3276 if the issue is urgent. Note: This number will be discontinued after the launch on June 1, 2011.
Updates
May 30, 11 PM EST: All services have been upgraded to ExchangeDefender 7. All tests have confirmed that sites and services are completely operational.
Comments Off
At 11:00 PM Eastern we will be rebooting the LOUIEMBOX1 server to repair the windows server backup feature. During the reboot, mailboxes hosted on LOUIEMBOX1 will experience a brief service interruption as the server reboots. We understand that this is short notice, however in the interest of maintaining a healthy backup history, we must reboot the server before the next scheduled backup.
Update 11:30 PM Eastern: We are now rebooting the LOUIEMBOX1 server.
Update 11:45 PM Eastern: Service has been restored to users on LOUIEMBOX1
Comments Off
We are performing Active Directory maintenance on the LIVEARCHIVE network in order to resolve issues with database latency. During maintenance, users may receive warning about their mailbox being unavailable. We appreciate your patience as we make the necessary modifications in order to restore service to 100% operation.
Comments Off
We will be installing Exchange 2010 SP1 RU3 on the LOUIE cluster tonight beginning at 9PM Eastern.
During the upgrade on the CAS servers, users may see a brief disconnection while individual CAS nodes are upgraded, however, they should automatically connect to another available CAS server.
During the upgrade on the Mailbox servers, users may see a brief disconnection from public folders while individual Mailbox codes are upgraded, however, they should automatically connect to the next available Public Folder Replica.
During the upgrade on the HUB servers, users may see a brief delay for incoming mail, however, outgoing mail should not be affected.
This upgrade will allow us to seamlessly move mailboxes from LOUIE to our newest Exchange Cluster if partners would like to move.
Update 9:10 PM Eastern: We are beginning to upgrade the CAS servers in LOUIE
Update 10:15 PM Eastern: We have completed the upgrade to the CAS servers and will now begin on the HUB and Mailbox servers.
Update 11:04 PM Eastern: Service has been restored to the LOUIE network and we’ve confirmed access to our newest Exchange cluster.
Comments Off
We’re currently investigating some issues with a specific database on DEWEY there may be a quick service interruption while we research this.
[9:57 am]It appears that the system will require a restart after the reports of slowness have increased please stand by. We’ll be doing so in 10 minutes. Please remember LiveArchive is available during these windows.
[10:21 am]Mail is now delivering as expected, please allow time for the queued mail to deliver. Thank you for your patience.
Comments Off
Huey is experiencing some issues with some client connection services. We’ve gone through all of the services carefully to avoid this. However, it is necessary to complete this in the next 5 minutes.
Work on Huey is continuing. Please remember LiveArchive is available during these windows of time.
Comments Off
Per our previous NOC posting about LOUIE, tonight we will be performing maintenance as we physically move the servers around. Unfortunately we are running behind schedule and work was not able to begin at the scheduled 10PM mark. We estimate that we will be able to begin around 3AM Eastern and will update this NOC posting once work has begun.
Update 3:30 AM Eastern: We’ve received the green light to begin the move process. At 4:00 AM Eastern we will be powering off the LOUIE network to physically move the servers to our newest cage in our Dallas data center.
Update 4:05 AM Eastern: We’re beginning the move for the LOUIE servers. Customers on the LOUIE network will be unable to access their mailboxes for the next 20 minutes as the move occurs.
Update 5:05 AM Eastern: The move completed about 15 minutes ago – we are running our performance tests to ensure that the post move process completed successfully
Comments Off
We’re currently working on LiveArchive as some Mail databases are not accepting delivery of new mail. Up to this point access to the Mailbox had not been affected. We’re in the process of performing emergency maintenance. Please stand by for an update as to when the work is complete.
Update: 70% of the Databases are remounted and beginning to receive mail. We’ll update shortly when the work has completed.
Update: 100% of the Databases have remounted with the exception of DB8 which was already under maintenance.
Comments Off
On 5/19/11 at 10 PM Eastern we will be performing maintenance on the LOUIE Exchange 2010 network as we physically move the servers into a new cabinet. During the maintenance period, users will briefly loose connection and access to their mailboxes, however, service is expected to be restored shortly thereafter.
Prior to maintenance we will update our NOC blog with a more refined timeline.
Comments Off
Please note we’ve received a note about a possible outage for some upstream connections to our Australian DC. They’re saying only certain routes are affected. There is no current ETA.
Comments Off
We do apologize for any inconvenience this issue with Louie may have caused your client. Our engineers discovered that the backup jobs on the server went to truncate the log files and it forced a dismount of the database. Our monitoring software attempted to remount the database, but the backup job would continue to dismount it. As of this moment the engineers have confirmed that the backup has been completed and all services have been fully restored and are currently active.
Comments Off
We’re currently researching a network outage that seems to have stricken our Level3 Network into our data center.
Update 11:50: This is has been fully mitigated. Everything is back up and running! We apologize for any inconvenience to our Level3 clients, this shouldn’t happen again. We’ll be posting additional details as we complete our investigation.
Comments Off
Our monitoring software has alerted us to an abnormally high page count on LOUIEMBOX2. We were able to dehydrate a few processes to alleviate the pressure, however, we will need to restart the storage device for users on DB3.
At 10:00 PM Eastern we will be restarting the storage device for DB3 on LOUIEMBOX2. During the restart, users will be unable to access their mailboxes for up to 10 minutes. We expect service to be completely restored by 10:15 PM Eastern.
Update 10:00 PM Eastern: We are beginning the process of dismounting DB3 and restarting the storage array.
Update 10:10 PM Eastern: Service has been restored to users on DB3
Comments Off
