We’re in the process of conducting a disk intensive metric test on the LOUIE mailbox and CAS servers to ensure that issues discovered earlier this week are no longer present.

During the next 10 minutes clients with large mailboxes or with multiple instances of their mailbox opened (Outlook and activesync, etc) may notice brief timeouts or sporadic error messages as we pool all resources to completing the test as soon as possible.

About 10 minutes ago we started to see larger amounts of packet loss on the RPC channel for LOUIE. To restore service immediately, we’ve disabled the CAS array and bound the IP to a specific node. Users may see a brief 2-5 minute disconnect as the ARP routing flushes from the CAS array to the specific node.

Update 11:29 AM Eastern: The ARP flush was completed and full CAS service has been restored to LOUIE. We will continue to monitor channel for any signs of RPC latency or packet loss

We’ve received report from our monitoring servers about massive packet loss on the Exchange hosting network. Upon investigation, it appears to be a routing issue with one of the main providers to the core router. We’ve initialized fail over mode and traffic is now routing through our alternative provider. Customers may have experienced a brief disconnection as Outlook lost connection through the old routes and discovered the new routing for the Exchange network.

Tonight we will be rebooting the DEWEYMbox2 server as it is no longer allowing users to resolve profile names on new mailbox setups and is randomly disconnecting users from the directory service. The reboot is scheduled for 10:00 PM Eastern and is anticipated to only take 15 minutes to complete.

On Saturday, March 19th at 10:00 PM Eastern we will begin work on LOUIE blackberry to improve email delivery speed from Exchange to the handset. During maintenance, we will be off-loading BES users to new BES servers in the network. Between the hours of 10:00 PM and 1:00 AM Eastern, users may see off and on delays for email delivery speed.

About 30 minutes ago Autodiscover started pushing updates of the node array list that included a previously removed node.. The IP that was previously used by the removed node is now in use by another server and when the update was pushed, some outlook clients would have tried to connect to louiecas3 and received SSL warnings from outbound-jr

If any users received this pop up, you can safely tell the users to click “No” on the SSL prompt as the connection is not needed. After the user closes the dialog box, they may continue to work as normal.

We will be performing service restarts (mainly, information store) on LOUIE mailbox servers to force AD permission replication. Users may experience a momentary disconnect from Outlook or Mailbox Unavailable errors in OWA. Service is expected to be fully restored by 9:45 PM Eastern

Update 9:27 PM Eastern: Service has been restored to customers on LOUIE. Users may continue to see temporary errors as all users reconnect, however, as traffic reconnects, service will go back up to speed.

Tonight starting at 11:00 PM Eastern we will be performing maintenance on the LOUIE blackberry server to improve overall service delivery speed. During the maintenance, users should expect to see a delay in inbound and outbound email or operation time outs. Service is expected to be restored by 12:30 AM Eastern.

We are in the process of installing a mandatory update for Blackberry Enterprise Server which is set to resolve recent activation issues with new handsets. Users on BES may notice a slight delay in items synching with the server until 10:45 AM Eastern.

Update 10:46 AM Eastern: The update has been successfully installed and service for LOUIE BES is coming back online.