http://exchangedefender.com/noc/feed/ Exchange Defender Network Operations RSS Feed en-us Thu, 17 May 2012 11:23:00 AM
Earlier today we started to send out alerts for clients that enabled SMTP monitoring in ExchangeDefender for connection failures.]]> ../ Mon, 14 May 2012 12:38:00 PM
All users should be fully online, a more detailed report will follow on the NOC page.]]> ../ Mon, 14 May 2012 12:27:00 PM
We are in the process of recovering from this issue. A portion of the affected population is back online now. The rest should be up shortly.]]> ../ Mon, 14 May 2012 12:26:00 PM
We've found the issue and working to resolve it. Remember LiveArchive is always available to ensure your clients remain online.]]> ../ Mon, 14 May 2012 12:13:00 PM
Researching reports of connectivity issues to RD. Please note that LiveArchive is available if needed.]]> ../ Thu, 10 May 2012 10:51:00 AM
Mail delivery issues on RD appear to be resolved after readjusting resource distribution to domain controllers. ]]> ../ Thu, 10 May 2012 10:40:00 AM
Minor mail delivery delays (up to 10m) on ROCKERDUCK. We are working to identify the issue.]]> ../ Thu, 10 May 2012 08:15:00 AM
Service on backup90 has been resumed and all user homes are available with previous data]]> ../ Wed, 09 May 2012 04:45:00 PM
We are finalizing the copy from disk 2/4 and still restoring copy from disk 3/4 on backup90]]> ../ Wed, 09 May 2012 08:31:00 AM
We are now restoring data on drive 3/4 on backup90]]> ../ Tue, 08 May 2012 08:54:00 PM
We've copied 2/4 of the drives to the iSCSI cluster for backup90 -- service should resume tomorrow afternoon]]> ../ Tue, 08 May 2012 04:35:00 AM
User data is currently being copied to the iSCSI device]]> ../ Mon, 07 May 2012 10:50:00 PM
User profiles are currently being copied over to the temp iSCSI cluster]]> ../ Mon, 07 May 2012 04:16:00 PM
We are close to being able to recreate users on the iSCSI cluster. Integrity check is 80% completed.]]> ../ Mon, 07 May 2012 08:45:00 AM
The index verification job has failed. We are going to move all data to a temporary iSCSI server as we replace the faulty controller on b90]]> ../ Sat, 05 May 2012 01:34:00 PM
We are starting an index rebuild on the data array for backup90. This is anticipated to last until Sunday.]]> ../ Sat, 05 May 2012 10:06:00 AM
We are currently running an integrity check on the data volume that holds backup data on backup90]]> ../ Sat, 05 May 2012 07:31:00 AM
Work is still on going on this server.]]> ../ Thu, 03 May 2012 09:19:00 PM
we will restore access to all items before the beginning of business.]]> ../ Thu, 03 May 2012 09:18:00 PM
We currently have one public folder replica offline for maintenance. Customers may see missing items and folders in their public folders]]> ../ Thu, 03 May 2012 04:06:00 PM
Currently experiencing delays with PFs on LOUIE. Please note no mail will be lost, only delayed.]]> ../ Mon, 30 Apr 2012 03:44:00 PM
We are investigating login issues with back90]]> ../ Thu, 26 Apr 2012 11:35:00 AM
Users may have experienced slight delays from 11:13-11:21 EST. The issue has been mitigated and normal mail flow has resumed]]> ../ Tue, 24 Apr 2012 05:40:00 AM
delay notices may have been received last night that were generated by test. Unfortunately these were unavoidable for us to accurately test]]> ../ Tue, 24 Apr 2012 05:39:00 AM
We've finally resolved the issue by redesigning our transport agent and then increasing the delay notification timeout per Microsoft recom.]]> ../ Fri, 20 Apr 2012 08:29:00 AM
We are currently gathering details from clients to narrow down the specific node that sent out the messages.]]> ../ Fri, 20 Apr 2012 08:27:00 AM
Livearchive servers falsely sent out delay notification messages for messages to xd286### ]]> ../ Wed, 18 Apr 2012 11:43:00 AM
Connectivity issues have been restored.]]> ../ Wed, 18 Apr 2012 11:01:00 AM
Researching connectivity issues to EUROPE]]> ../ Wed, 11 Apr 2012 11:21:00 AM
Our Exchange Engineers were able to rectify the issue with LiveArchive. You should no longer see delay messages from the LiveArchive server]]> ../ Wed, 11 Apr 2012 09:44:00 AM
We are currently working on the delay notifications from postmaster@la.livearchive3.exchangedefender.com. Regular mail flow is unaffected.]]> ../ Tue, 03 Apr 2012 11:31:00 AM
We're currently investigating a couple of reports of BES messages not sending and receiving.]]> ../ Sun, 18 Mar 2012 08:02:00 PM
We're currently investigating an issue with the ahsay software that's causing the web portal to be unavailable.]]> ../ Fri, 16 Mar 2012 08:52:00 AM
We are resuming move requests from the problematic database to the newer DAG protected databases for Della.]]> ../ Fri, 16 Mar 2012 08:45:00 AM
Issue appeared to be related to a single database on Della which has been problematic in the past. Service has been restored]]> ../ Fri, 16 Mar 2012 08:26:00 AM
We are researching an connectivity issue report to Della at this time. We will provide more details as they become available.]]> ../ Wed, 07 Mar 2012 04:38:00 PM
Disk replacement was held off until today as the previously shipped drive tested bad. ]]> ../ Wed, 07 Mar 2012 01:02:00 PM
It appears the issue is limited to cox.net recipients and was due to a invalid cox.net entry to ExchangeDefender. Issue has been cleared out]]> ../ Wed, 07 Mar 2012 12:41:00 PM
Mail is only failing delivery for cox.net. IPs do not appear to be blacklisted -- we are continuing to investigate]]> ../ Wed, 07 Mar 2012 12:09:00 PM
There was a period between 11:45 - 12:00 where messages may have seen up to a 10 minute delay due to configuration reloads across the board.]]> ../ Tue, 06 Mar 2012 01:37:00 PM
Connectivity to backup74 has been restored.]]> ../ Tue, 06 Mar 2012 12:14:00 PM
We are currently looking into connectivity issues with backup74]]> ../ Mon, 05 Mar 2012 03:17:00 PM
We are in the process of replacing a failed RAID disk for the DATA RAID array. ]]> ../ Mon, 05 Mar 2012 03:17:00 PM
We will be renewing SSL certs for exchangedefender.com you may be see website related services interrupted briefly throughout the night.]]> ../ Mon, 05 Mar 2012 06:28:00 AM
Service on backup90 has been restored. Server upgraded to new hardware over the weekend.]]> ../ Fri, 02 Mar 2012 10:31:00 AM
We are looking into connectivity issues with backup90]]> ../ Wed, 29 Feb 2012 03:20:00 PM
Connectivity has been restored to backup74]]> ../ Wed, 29 Feb 2012 02:47:00 PM
DDoS attack on inbound Dallas]]> ../ Wed, 29 Feb 2012 12:57:00 PM
Checking connectivity issue with backup74]]> ../ Wed, 29 Feb 2012 10:46:00 AM
Upgrades have completed and service has been restored to backup90. We would like to thank everyone for their patience during the upgrades.]]> ../ Tue, 28 Feb 2012 07:11:00 PM
We're doing another round of upgrades to backup90]]> ../ Mon, 27 Feb 2012 04:14:00 PM
All work has been completed on Della]]> ../ Mon, 27 Feb 2012 04:01:00 PM
We will be shutting down the mailbox servers in DELLA to upgrade the network card.]]> ../ Mon, 27 Feb 2012 07:46:00 AM
Issue was resolved by 7:20 AM Eastern by moving the mail processing role to MBOX2. We will investigate the root cause during off hours.]]> ../ Mon, 27 Feb 2012 07:45:00 AM
Della users experienced outbound delay notifications for users on MBOX1. ]]> ../ Sat, 18 Feb 2012 03:25:00 PM
Maintenance completed time lapsed 6 minutes.]]> ../ Sat, 18 Feb 2012 02:57:00 PM
We're moving back the window to start roughly 3:30 est, and to last approximately 30 mins (this is a conservative ETA)]]> ../ Sat, 18 Feb 2012 02:57:00 PM
Clarification the portion of Encryption affected is the retrieval only.]]> ../ Sat, 18 Feb 2012 02:26:00 PM
We're continuing work on upgrading the server housing Encryption and Web File Sharing. The service will be offline for roughly 20 @ 3pm est]]> ../ Thu, 16 Feb 2012 10:06:00 PM
We had to do a quick maintenance on our relay tables for exchangedefender, this may have caused a small interval where outbound mail bounced]]> ../ Thu, 16 Feb 2012 10:06:00 PM
We had to do a quick maintenance on our relay tables, this may have caused a small interval where relay tables were not inserted correctly.]]> ../ Thu, 16 Feb 2012 10:30:00 AM
We're currently working on our balancing routines to mitigated increased volume.]]> ../ Tue, 14 Feb 2012 11:39:00 AM
We're currently being DDoS'ed on our inbound network. This may affect some delivery times.]]> ../ Mon, 13 Feb 2012 03:45:00 PM
BB connectivity has been restored.]]> ../ Mon, 13 Feb 2012 01:08:00 PM
Researching connectivity complaints to RDBES]]> ../ Sat, 11 Feb 2012 11:59:00 AM
We will be performing one last upgrade to these services, we should be back shortly.]]> ../ Sat, 11 Feb 2012 11:59:00 AM
This work has been completed.]]> ../ Sat, 11 Feb 2012 11:38:00 AM
We'll be performing a hardware upgrade to Encryption and File Sharing. Please stand by.]]> ../ Sat, 11 Feb 2012 11:29:00 AM
We'll do one more upgrade shortly to this service.]]> ../ Sat, 11 Feb 2012 11:28:00 AM
This maintenance has been completed.]]> ../ Sat, 11 Feb 2012 11:14:00 AM
We're now begining the hardware upgrade on backup90. We anticipate 20 minutes of downtime.]]> ../ Sat, 11 Feb 2012 11:06:00 AM
The issue has been resolved and mail has flushed on RD]]> ../ Sat, 11 Feb 2012 11:06:00 AM
The issue appears to be related to a corrupted SP2 install on the EDGE network. We are reapplying the update to correct the issue. ]]> ../ Sat, 11 Feb 2012 10:54:00 AM
The routing issues causing this problem have been mitigated and this issue has been resolved.]]> ../ Sat, 11 Feb 2012 10:54:00 AM
We're working on an internal routing issue with RD currently. No mail has been lost while this issue has been ongoing.]]> ../ Sat, 11 Feb 2012 09:44:00 AM
Hueys connectivity issue, was caused due to hardware failure on the server itself. We have resolved the issue. The services are coming onlin]]> ../ Sat, 11 Feb 2012 09:43:00 AM
We're investigating some routing issues for mail delivery on LOUIE.]]> ../ Sat, 11 Feb 2012 09:42:00 AM
Currently investigating a DC routing issue affecting Huey.]]> ../ Thu, 09 Feb 2012 05:27:00 PM
All connections to RD should be restored, this includes OWA and Activesync.]]> ../ Thu, 09 Feb 2012 01:39:00 PM
Researching connection issues to RD where users are being directed to cas.la.rockerduck.exchangedefender.com]]> ../ Wed, 08 Feb 2012 09:58:00 PM
At 8:54 PM Eastern RDMBOX2 dismounted databases and could not fail over due to automatic recovery being disabled. Resolved at 9:05 PM.]]> ../ Wed, 08 Feb 2012 07:46:00 AM
All service issues with RD have been resolved. We will postpone continuing work until later tonight]]> ../ Wed, 08 Feb 2012 07:38:00 AM
Mailbox databases have been remounted on RD and we do have clients connected however some still report issues. Still investigating. ]]> ../ Wed, 08 Feb 2012 06:55:00 AM
We've received alerts from our monitoring about network related issues with RD. We've confirmed the DAG has gone offline]]> ../ Wed, 01 Feb 2012 05:50:00 PM
Everyone should be connected now.]]> ../ Wed, 01 Feb 2012 01:07:00 PM
We are investigating reports with BB devices on LOUIE not receiving mail.]]> ../ Tue, 31 Jan 2012 12:17:00 PM
The threat has been discipated, we're working on flushing the queues. ]]> ../ Tue, 31 Jan 2012 11:56:00 AM
Mail has been coming under DDos Pressure we're currently working to dissipate the issue.]]> ../ Mon, 30 Jan 2012 11:35:00 PM
Issue was resolved by power cycling the server]]> ../ Mon, 30 Jan 2012 11:31:00 PM
Between the hours of 6:45 PM - 10:30 PM the MAIL1 server was not responding to some user requests.]]> ../ Sun, 29 Jan 2012 06:50:00 PM
Work has been completed service has been completely restored.]]> ../ Sun, 29 Jan 2012 06:49:00 PM
The process is at 33% percent, please remember LiveArchive for users impacted during these late hours in the UK. ]]> ../ Sun, 29 Jan 2012 06:31:00 PM
ETA on the DB3 move is currently at 28% percent. ]]> ../ Sun, 29 Jan 2012 06:21:00 PM
We're currently working on an issue with DellaDB3. The issue is only affecting users on this DB.]]> ../ Fri, 27 Jan 2012 11:39:00 AM
All databases for RDMBOX1 have been remounted on RDMBOX1]]> ../ Fri, 27 Jan 2012 11:28:00 AM
Half of the databases have been moved back to RDMBOX1]]> ../ Fri, 27 Jan 2012 11:03:00 AM
RDMBOX1 is coming back online and we will be resuming service on it after health checks pass]]> ../ Fri, 27 Jan 2012 10:40:00 AM
Just to clarify, one Dallas mbox server for RD has gone down and failed to Cali. We are working to move service back to Dal]]> ../ Fri, 27 Jan 2012 10:40:00 AM
The issue was isolated to OWA users only and has been resolved.]]> ../ Fri, 27 Jan 2012 10:35:00 AM
Mbox1 had an issue and has failed over to Mbox5.]]> ../ Thu, 26 Jan 2012 06:50:00 AM
Issue has been linked to a specific domain controller and has been resolved.]]> ../ Wed, 25 Jan 2012 09:16:00 PM
We are taking down all CAS nodes in LOUIE except for LOUIECAS4]]> ../ Wed, 25 Jan 2012 04:42:00 PM
Multiple customers have reported issues with setting up new profiles on LOUIE. We will be disabling all but one CAS node to diagnose]]> ../ Mon, 23 Jan 2012 08:06:00 AM
Access has been restored after the network adapter was reset. We will continue to thoroughly monitor this over the next three hours]]> ../ Mon, 23 Jan 2012 07:49:00 AM
Clients on RDMBOX2 may experience issues accessing their mailbox as we sort out a networking issue. Failure to DAG node is not requred]]> ../ Sat, 21 Jan 2012 10:04:00 AM
Completed without issue.]]> ../ Sat, 21 Jan 2012 09:36:00 AM
We're doing a quick reboot due to an important Windows Update on Della. Expected to last no more than 5 minutes.]]> ../ Thu, 12 Jan 2012 02:49:00 PM
Queues have flushed and normal service is expected. This NOC case will remain open as we investigate further to achieve a permanent solution]]> ../ Thu, 12 Jan 2012 01:51:00 PM
We're still working this issue with every resources available. ]]> ../ Thu, 12 Jan 2012 09:53:00 AM
We are currently receiving alerts about extended delivery times on inbound messages.]]> ../ Tue, 10 Jan 2012 02:07:00 PM
This issue has been fully dissipated.]]> ../ Tue, 10 Jan 2012 12:24:00 PM
We are currently thwarting off a DDOS attack. During the attack there may be a delay in receiving messages.]]> ../ Mon, 09 Jan 2012 11:58:00 PM
Current BES service has been restored on the original server. From what we can tell in the logs devices are syncing. We are monitoring BES]]> ../ Mon, 09 Jan 2012 01:55:00 PM
We are going to install BES for LOUIE on a new server. We do not anticipate full service restoration by 5PM Eastern.]]> ../ Mon, 09 Jan 2012 12:46:00 PM
The backup job for LOUIE BLACKBERRY has completed. We are proceeding with the reinstall of BES]]> ../ Mon, 09 Jan 2012 12:45:00 PM
We are currently at a 'stand still' with LOUIE BLACKBERRY as we are performing a full backup before reinstalling BES]]> ../ Mon, 09 Jan 2012 12:20:00 PM
At the current moment we estimate service to be restored on LOUIE BLACKBERRY by 2:00 PM Eastern at the latest.]]> ../ Mon, 09 Jan 2012 12:19:00 PM
We are reinstalling the blackberry enterprise software on LOUIE. At this point users will not have to reactivate once service resumes]]> ../ Mon, 09 Jan 2012 10:56:00 AM
Token authentication between LOUIEBLACKBERRY and the Domain Controller began to fail all requests which is preventing the service to loading]]> ../ Mon, 09 Jan 2012 10:11:00 AM
We're investigating reports of sync issues. We'll provide more details as they become available.]]> ../ Wed, 04 Jan 2012 03:07:00 AM
Tomorrow @ 430AM We will be relocating the sites on WWW2 (Windows web hosting) to a new physical host]]> ../ Tue, 03 Jan 2012 08:39:00 AM
A server reset has resolved the issue. We will investigate the cause of the site outage in the off hours tonight.]]> ../ Tue, 03 Jan 2012 08:34:00 AM
We received alerts about LOUIESHAREPOINT4 sites being unresponsive]]> ../ Thu, 29 Dec 2011 08:51:00 AM
Fail Over issue has been resolved. Engineers believe issue only affected OWA access, shouldn't have affected Outlook. Mail Flow not affected]]> ../ Thu, 29 Dec 2011 05:40:00 AM
It appears that this issue only affected OWA access which should not have affected the Outlook client or mail flow. ]]> ../ Thu, 29 Dec 2011 04:21:00 AM
The Engineers have become aware of an issue with RD failing over to the LA cluster and are currently working on restoring service to Dallas]]> ../ Wed, 28 Dec 2011 08:04:00 PM
All checks completed services have been restored.]]> ../ Wed, 28 Dec 2011 03:59:00 PM
Resources have been allocated and queues are flushed.]]> ../ Wed, 28 Dec 2011 03:45:00 PM
User may experience mail delays on RD from 3:35-3:45. We are going to suspend mail flow from 3:50-3:55 to allocate resources to edge server]]> ../ Wed, 28 Dec 2011 03:45:00 PM
We're researching connectivity issues with backup73 currently. We'll provide details as they become available.]]> ../ Thu, 22 Dec 2011 12:14:00 AM
The patch has been applied to the load balancer and service was not impacted.]]> ../ Wed, 21 Dec 2011 08:07:00 PM
Work completed service has been fully restored.]]> ../ Wed, 21 Dec 2011 07:57:00 PM
We'll be performing a quick hardware upgrade to the Windows Hosting platform. Expected maint time 10 minutes.]]> ../ Wed, 21 Dec 2011 01:39:00 PM
We have thwarted a DDOS attack, however, some messages may be delayed.]]> ../ Wed, 21 Dec 2011 11:37:00 AM
In the mean time we've implemented a small patch to restart the load balancer services if connections are dropped]]> ../ Wed, 21 Dec 2011 11:30:00 AM
Tonight starting at 11:30 PM Eastern we will be updating our load balancer for Rockerduck to address a bug ]]> ../ Tue, 20 Dec 2011 11:55:00 AM
Clients on Rockerduck may have experienced a brief disconnection between 11:45 - 11:50 AM EST]]> ../ Tue, 20 Dec 2011 10:41:00 AM
The ED Engineers have become aware of a slight delivery delay on the RD cluster and currently working on the issue.]]> ../ Mon, 19 Dec 2011 11:59:00 AM
Maintenance procedures have been completed.]]> ../ Mon, 19 Dec 2011 10:30:00 AM
The ED Engineers will be performing maintenance on RDMBOX1. To avoid any latency issues the DB will be temporarily moved to RDMBOX2]]> ../ Mon, 19 Dec 2011 09:09:00 AM
The issue has been resolved. The Engineers had diagnosed that the issue was cause by a DDOS attack. Ther server should now be accessible.]]> ../ Mon, 19 Dec 2011 08:51:00 AM
The Web Hosting Engineers have become aware of a connectivity issue with the Mail1 server. They are currently researching the issue.]]> ../ Thu, 15 Dec 2011 09:23:00 PM
A DDOS attack slowed LiveArchive delivery for the LA nodes. The attack has been suprised there should be no impact on live mail.]]> ../ Thu, 15 Dec 2011 06:45:00 PM
Connectivity to MATILDA via Outlook has been confirmed. Outlook may need to be restarted to connect.]]> ../ Thu, 15 Dec 2011 06:37:00 PM
We are performing an emergency reboot on the mailbox server. We estimate services to be restored in 15 minutes.]]> ../ Thu, 15 Dec 2011 06:15:00 PM
Researching reports of users not able to connect to MATILDA via Outlook.]]> ../ Thu, 15 Dec 2011 08:18:00 AM
Regrettably we had Mail1 land on a couple of RBL's due to a partner's mail account being compromised. We have requested de-listment of Mail1]]> ../ Wed, 14 Dec 2011 04:18:00 PM
Databases have failed over to RDMBOX2. We cannot see any logged 'downtime' but because communication was lost we moved the active manager]]> ../ Wed, 14 Dec 2011 04:18:00 PM
We are failing over databases from RDMBOX1 to RDMBOX2]]> ../ Wed, 14 Dec 2011 04:16:00 PM
Communication with RDMBOX1 was lost. We are working to determine the status before failing DBs from MBOX1 to passive nodes]]> ../ Mon, 12 Dec 2011 12:24:00 PM
The Issue was tied to a message log file issues and it has been fully resolved. ]]> ../ Mon, 12 Dec 2011 12:14:00 PM
The Exchange Engineers are currently investigating a connectivity issue with the Louie server. ]]> ../ Sat, 10 Dec 2011 09:50:00 AM
Service has been fully restored on the APC change.]]> ../ Thu, 08 Dec 2011 11:06:00 AM
The ED Engineers have switched LiveArchive traffic to the LA cluster due to a connectivity issue with the Dallas cluster.]]> ../ Wed, 07 Dec 2011 02:53:00 PM
The issues on della have been resolved. Active directory was restored to a timestamp 30 minutes before the outage occurred.]]> ../ Wed, 07 Dec 2011 01:31:00 PM
There appears to be an active directory issue in DELLA. We've stopped all services on della as we investigate.]]> ../ Wed, 07 Dec 2011 12:53:00 PM
Researching reports of unable to load a Outlook profiles]]> ../ Wed, 07 Dec 2011 11:11:00 AM
We've replaced one problematic CAS server and added another to RD. Please report any issues or slow response to support.ownwebnow.com]]> ../ Tue, 06 Dec 2011 02:00:00 PM
The backup on RDMBOX2 has completed. We are now moving active databases off of MBOX2 to MBOX1]]> ../ Tue, 06 Dec 2011 01:48:00 PM
We've received alerts about issues with RDCAS1 in ROCKERDUCK in terms of new profile setup and OWA functions. We are disabling from the LB]]> ../ Tue, 06 Dec 2011 01:30:00 PM
We are waiting for the backup job on MBOX2 to complete before moving the active databases to MBOX1]]> ../ Tue, 06 Dec 2011 01:13:00 PM
We've received alerts from monitoring about a RAID controller battery issue. We are moving all active databases off MBOX2]]> ../ Mon, 05 Dec 2011 09:08:00 PM
Work on DB7 on Rockerduck has been postponed to a later date.]]> ../ Mon, 05 Dec 2011 04:56:00 PM
Tonight we will be rearranging the DB logging layout on RD. Users on RDDB7 will be disconnected from their mailbox for about 15 mins @ 10EST]]> ../ Thu, 01 Dec 2011 09:16:00 AM
Our Engineers have been able to diagnose that this issue is only affecting one particular SP site and should not affect the entire farm.]]> ../ Thu, 01 Dec 2011 09:00:00 AM
The Engineers have been able to diagnose an issue with the back-end DB for the SharePoint server. They are currently working on the DB.]]> ../ Thu, 01 Dec 2011 08:44:00 AM
The SharePoint Engineer have become aware of a connectivity issue with one of the SharePoint servers. We are currently researching the issue]]> ../ Wed, 30 Nov 2011 12:54:00 PM
We've activated the Los Angeles version of LiveArchive while we attempt to resolve issues with Dallas Databases]]> ../ Tue, 29 Nov 2011 09:27:00 PM
We will be dismounting DB2 and DB3 from all DAG members to clear log files as replication has fallen behind]]> ../ Sun, 27 Nov 2011 06:41:00 PM
Client access restored. Signs of a brute force attack on the RDP service which prevented client access. Mail Flow was not affected.]]> ../ Sun, 27 Nov 2011 05:48:00 PM
We are currently Investigating a connectivity issue on the Australia Hosted Service.]]> ../ Wed, 23 Nov 2011 08:58:00 PM
Upgrade failed again. To restore services, we've brought the sites online. Have limited admin, ie no indexer rsts etc. Dtails to come]]> ../ Wed, 23 Nov 2011 04:24:00 PM
Progress for the sharepoint configuration wizard is at 44%]]> ../ Wed, 23 Nov 2011 04:24:00 PM
The upgrade process has failed for SP4. We are rerunning the upgrade with the force parameter.]]> ../ Wed, 23 Nov 2011 03:32:00 PM
We are now at 40%.]]> ../ Wed, 23 Nov 2011 02:58:00 PM
Progress is currently at 38%]]> ../ Wed, 23 Nov 2011 02:17:00 PM
Progress for the sharepoint configuration wizard is at 36%]]> ../ Wed, 23 Nov 2011 02:06:00 PM
Progress for the sharepoint configuration wizard is at 34%]]> ../ Wed, 23 Nov 2011 01:46:00 PM
Progress is moving extremely slower than it has before. Progress is currently at 33%]]> ../ Wed, 23 Nov 2011 01:27:00 PM
Progress is currently at 31%]]> ../ Wed, 23 Nov 2011 01:26:00 PM
We are running the SP Configuration Wizard to repair the corrupted config. The process is at 30%]]> ../ Wed, 23 Nov 2011 01:01:00 PM
The sharepoint service on SP4 has stopped serving Sharepoint sites to clients]]> ../ Tue, 22 Nov 2011 07:55:00 PM
We are rebooting SHAREPOINT4 (65.99.255.43)]]> ../ Sat, 19 Nov 2011 11:16:00 PM
The Dewey Service has been fully restored. All temporary data has been seeded over to the old mailbox data. ]]> ../ Sat, 19 Nov 2011 07:17:00 PM
Under 10 users left on merge, problems caused by old data merged at client side to, *ALL* items req checks for those users mboxs now]]> ../ Fri, 18 Nov 2011 05:22:00 PM
Two passive copy databases have fallen behind in replication and will not resume normally]]> ../ Fri, 18 Nov 2011 09:28:00 AM
The seeding process does not provide a progress bar. The best we can do is a percentage based on the mailbox counts ~ 86% ROUGH est.]]> ../ Fri, 18 Nov 2011 09:27:00 AM
The connection issue between the DB server and Louie SP sites has been resolved. The issue was tied to the NIC.]]> ../ Fri, 18 Nov 2011 09:24:00 AM
The engineers diagnosed an issue with the SQL server not properly communicating with the SharePoint servers. This issue has been resolved.]]> ../ Fri, 18 Nov 2011 07:49:00 AM
The SharePoint Engineers are currently investigating and issue with connectivity to the SharePoint Site on the Louie cluster.]]> ../ Fri, 18 Nov 2011 07:38:00 AM
We are still in the process of seeding all the temporary data from the Temp mailbox to the primary mailbox. ]]> ../ Thu, 17 Nov 2011 10:59:00 PM
isinteg has completed on DEWEY and DB3 has been mounted. We will begin the seeding process shortly]]> ../ Thu, 17 Nov 2011 09:14:00 PM
Isinteg is on step 20/21 with 15% completed]]> ../ Thu, 17 Nov 2011 06:47:00 PM
Isinteg for DB3 is current on step 11/21 with 37% overall completed]]> ../ Thu, 17 Nov 2011 01:13:00 PM
We were able to resolve the delays. No mail was lost during the process.]]> ../ Thu, 17 Nov 2011 11:11:00 AM
We are currently researching mail delays to Livearchive.]]> ../ Thu, 17 Nov 2011 09:22:00 AM
isinteg is at 16% and was at 9% on our previous update -- still on step 9/21]]> ../ Thu, 17 Nov 2011 05:41:00 AM
isinteg is on step 9 of 21. We do not believe that the checks will complete before noon.]]> ../ Thu, 17 Nov 2011 05:38:00 AM
The isinteg tool integrity check on the DB3 database are still processing.]]> ../ Wed, 16 Nov 2011 11:47:00 PM
eseutil has finally completed on DB3. We are running isinteg which took about 8-10 hours on DB2]]> ../ Wed, 16 Nov 2011 03:29:00 PM
Dewey is still performing the DB checks.]]> ../ Wed, 16 Nov 2011 05:10:00 AM
Still monitoring read/write activity on DB3 Integrity checks.]]> ../ Tue, 15 Nov 2011 04:36:00 PM
We've adjusted the balancing logic and we are preferring LOUIECAS4 over LOUIECAS2. We will continue to monitor the connection response times]]> ../ Tue, 15 Nov 2011 04:35:00 PM
We've run IISRESET to correct the issue. We are also rebalancing users on the load balancer to alleviate the pressure]]> ../ Tue, 15 Nov 2011 04:29:00 PM
Our monitoring has sent off alerts about response timeouts on LOUIE.]]> ../ Tue, 15 Nov 2011 09:56:00 AM
DB3 integrity checks are still processing. ]]> ../ Tue, 15 Nov 2011 05:03:00 AM
Integrity checks on DB3 are still processing. Continuing to monitor read/write activity. No progress data available. ]]> ../ Mon, 14 Nov 2011 12:00:00 PM
DB3 integrity checks and repairs are still on going..]]> ../ Mon, 14 Nov 2011 09:20:00 AM
Integrity checks on DB3 are still processing. Progress data unavailable.]]> ../ Mon, 14 Nov 2011 04:22:00 AM
Integrity Check is still on going on DB3, monitoring read/write activity.]]> ../ Sun, 13 Nov 2011 03:59:00 PM
DB3 integrity checks/repairs are still on going.]]> ../ Sun, 13 Nov 2011 09:52:00 AM
DB3 integrity checks and repairs are still continuing.]]> ../ Sat, 12 Nov 2011 03:55:00 PM
Integrity check on DB3 is still ongoing.]]> ../ Sat, 12 Nov 2011 07:01:00 AM
DB3 integrity checks are still continuing.]]> ../ Fri, 11 Nov 2011 09:17:00 PM
Integrity check are still moving along on DB3, a reminder there is no progress bar on this step of the process.]]> ../ Fri, 11 Nov 2011 04:28:00 PM
Integrity checks are still ongoing on the last affect DB: Db3]]> ../ Fri, 11 Nov 2011 02:41:00 PM
Dewey DB3 integrity checks and repairs are ongoing still.]]> ../ Fri, 11 Nov 2011 09:37:00 AM
We are still monitoring the DB3 integrity checks. No progress bar data available.]]> ../ Fri, 11 Nov 2011 07:16:00 AM
Database integrity checks on DB3 are still processing. ]]> ../ Thu, 10 Nov 2011 12:17:00 PM
Dewey is still processing the integrity checks on DB3.]]> ../ Thu, 10 Nov 2011 07:15:00 AM
DB3 integrity check still on going. Monitoring progress through read/write activity. Still no progress bar data Available. ]]> ../ Thu, 10 Nov 2011 05:32:00 AM
RPC client access service restarted. Mail delivery delays were reported but it appears tied to one database/hub combo. Mail is delivering.]]> ../ Thu, 10 Nov 2011 05:30:00 AM
We received reports from clients about inability to access their mailboxes via Outlook.]]> ../ Wed, 09 Nov 2011 07:07:00 PM
DB3 is still ongoing the final step of the integrity checks.]]> ../ Wed, 09 Nov 2011 03:33:00 PM
DB3 is still continuing its integrity checks.]]> ../ Wed, 09 Nov 2011 01:46:00 PM
Db3 integrity checks are still ongoing. This is the lengthy but longer portion of the process, with no progress bar.]]> ../ Wed, 09 Nov 2011 08:23:00 AM
Integrity checks are still ongoing on the final Database on Dewey.]]> ../ Wed, 09 Nov 2011 06:46:00 AM
Continuing to monitor the integrity check on DB3. Still processing.]]> ../ Tue, 08 Nov 2011 08:18:00 PM
Checks on the last DB are still ongoing, ]]> ../ Tue, 08 Nov 2011 03:37:00 PM
The issue with LiveArchive has been resolved, mail is now delivering.]]> ../ Tue, 08 Nov 2011 03:28:00 PM
We're researching reports of message delivery delays to our LA LiveArchive node. ]]> ../ Tue, 08 Nov 2011 02:12:00 PM
The integrity checks on DB3 are still ongoing. ]]> ../ Tue, 08 Nov 2011 08:16:00 AM
DB3 has moved into the final step of the integrity check.]]> ../ Tue, 08 Nov 2011 05:26:00 AM
Integrity checks on DB3 are still processing. ]]> ../ Mon, 07 Nov 2011 12:18:00 PM
DB3 is still ongoing with integrity checks.]]> ../ Mon, 07 Nov 2011 10:21:00 AM
Data from the temp DB2 database has been seeded into the original DB2 database. Users on DB2 should be back to normal operation.]]> ../ Mon, 07 Nov 2011 09:30:00 AM
Restore details posted @ http://bit.ly/vETct3]]> ../ Mon, 07 Nov 2011 04:21:00 AM
Integrity checks on DB3 continue to be monitored. Accounts on DB2 should begin to see their temp data merged with their old data.]]> ../ Sun, 06 Nov 2011 05:42:00 PM
DB2 on DEWEYMBOX1 has been mounted. We will begin the process of seeding back mail from the temp db shortly.]]> ../ Sun, 06 Nov 2011 02:20:00 PM
The inistg process is now at 80% completion.]]> ../ Sun, 06 Nov 2011 10:21:00 AM
The eseutil check has completed on DB2 and we are now running inistg check to correct any reference errors before remounting DB2]]> ../ Sun, 06 Nov 2011 08:23:00 AM
The DB integrity checks are still ongoing.]]> ../ Sat, 05 Nov 2011 09:28:00 PM
The Integrity checks are still ongoing and being monitored. ]]> ../ Sat, 05 Nov 2011 08:22:00 PM
Integrity checks on the DBs are still ongoing.]]> ../ Sat, 05 Nov 2011 05:51:00 PM
DB Checks are still processing.]]> ../ Sat, 05 Nov 2011 12:56:00 PM
Db integrity check is still confirmed as ongoing.]]> ../ Sat, 05 Nov 2011 09:29:00 AM
DB Integrity checks are still ongoing on both DB2 and DB3]]> ../ Fri, 04 Nov 2011 07:54:00 PM
We're still monitoring the DB Integrity scan, the process is still ongoing.]]> ../ Fri, 04 Nov 2011 03:50:00 PM
Investigating connectivity issue to admin.exchangedefender.com]]> ../ Fri, 04 Nov 2011 02:39:00 PM
Integrity checks are still continuing on both DBs.]]> ../ Fri, 04 Nov 2011 10:02:00 AM
The integrity check on DB3 has finally shown some progress, however, DB2 remains the same. We can still see disk activity for the process]]> ../ Fri, 04 Nov 2011 10:01:00 AM
Users on DEWEY experiencing slow speeds can switch their Outlook anywhere server to deweycas2.dewey.exchangedefender.com]]> ../ Fri, 04 Nov 2011 10:00:00 AM
Users on DEWEY experiencing slow speeds can switch their Outlook anywhere server to deweycas2.dewey.exchangedefender.com for an immed]]> ../ Fri, 04 Nov 2011 08:35:00 AM
Both database integrity checks are still active and processing. Continuing to monitor activity.]]> ../ Fri, 04 Nov 2011 04:54:00 AM
The integrity check are still continuing on both databases. Still monitoring via read/write head activity, no numeric value available.]]> ../ Thu, 03 Nov 2011 07:46:00 PM
The database check are still going, we are monitoring the progress through the read/write head activity.]]> ../ Thu, 03 Nov 2011 06:37:00 PM
The database check are still going, we are monitoring the progress through the read/write head activity. No progress bar available]]> ../ Thu, 03 Nov 2011 06:35:00 PM
The database check are still going, we are monitoring the progress through the read/write head activity. No progress bar data Avl.]]> ../ Thu, 03 Nov 2011 04:01:00 PM
The database check are still going, we are monitoring the progress through the read/write head activity. No progress bar data Avl.]]> ../ Thu, 03 Nov 2011 02:49:00 PM
The database check are still going, we are monitoring the progress through the read/write head activity. No progress bar data Avl. ]]> ../ Thu, 03 Nov 2011 08:52:00 AM
The database check are still going, we are monitoring the progress through the read/write head activity. No progress bar data Avl.]]> ../ Thu, 03 Nov 2011 04:11:00 AM
The integrity check are still continuing on both databases in the progress bar-less step. Still no numeric value at this time.]]> ../ Wed, 02 Nov 2011 07:21:00 PM
Guys, both integrity are still active and processing. Again no numeric value to assign as explained.]]> ../ Wed, 02 Nov 2011 05:59:00 PM
LOUIE is confirmed to be back online. We are continuing our investigating to provide a report.]]> ../ Wed, 02 Nov 2011 05:47:00 PM
There may be a hardware issue. We are currently investigating possible hardware issues with LOUIE.]]> ../ Wed, 02 Nov 2011 04:34:00 PM
We are investigating the connectivity with LOUIE. We will provide another update as more details are available.]]> ../ Wed, 02 Nov 2011 04:03:00 PM
We're researching reports of some users having trouble connecting to LOUIE. The server is online, but we're researching the root cause.]]> ../ Wed, 02 Nov 2011 03:55:00 PM
We're researching reports of some users having trouble connecting to LOUIE. The server is online, but we're researching the root cause.]]> ../ Wed, 02 Nov 2011 02:53:00 PM
Integrity chcks are moving. It's going to be tough to give numbers as both DBs are now in progress bar less steps. Activity is confirmed.]]> ../ Wed, 02 Nov 2011 12:24:00 PM
DB3 completed steps 2 and 3 out of 5, and is now on step 4. DB2 is at 72% of its final step. Noc @ http://bit.ly/vETct3]]> ../ Wed, 02 Nov 2011 10:15:00 AM
Both DB Integrity checks are still continuing to progress. Remember your business continuity options: Dailtone mbox and LiveArchive.]]> ../ Wed, 02 Nov 2011 08:31:00 AM
Anticipating DB2 completion by end of business day. Unfortunately DB3 check is taking longer potentially completing its process by Friday. ]]> ../ Wed, 02 Nov 2011 08:24:00 AM
Integrity checks on DB2 is on the final process at 80% while DB3 is on the second passing at 30%. ]]> ../ Wed, 02 Nov 2011 06:34:00 AM
At this time integrity checks on DB2 & DB3 are progressing smoothly. We are on the final processes of DB2 while DB3 is moving flawlessly.]]> ../ Tue, 01 Nov 2011 11:14:00 PM
The integrity check on DB3 is on step 4/5 and at 80% DB2 is on step 5/5 and at 65%. We anticipate DB2 finishing around 4 am and DB3 late Wed]]> ../ Tue, 01 Nov 2011 08:27:00 PM
The integrity checks are still progressing, the last database is through 70% of the Scanning process. Noc @ http://bit.ly/vETct3]]> ../ Tue, 01 Nov 2011 06:40:00 PM
The integrity checks are still progressing, the last database is through 62% of the Scanning process. Visit http://bit.ly/vETct3 for details]]> ../ Tue, 01 Nov 2011 03:38:00 PM
The integrity checks are still progressing, the last database is through 50% of the Scanning. ]]> ../ Tue, 01 Nov 2011 12:46:00 PM
The dial tone migration has completed and users are now able to access their mailboxes on the temporary database. ]]> ../ Tue, 01 Nov 2011 12:25:00 PM
Latest Update - Includes Image ]]> ../ Tue, 01 Nov 2011 12:25:00 PM
We are internally discussing activating a dial tone database switch to restore service to users. We will have a decision by 1pm.]]> ../ Tue, 01 Nov 2011 11:14:00 AM
The integrity checks are still progressing, the last database is through 30% of the Scanning. No ETA as there is no completion estimate.]]> ../ Tue, 01 Nov 2011 09:13:00 AM
The Engineers have been able to successfully mount one more database and are currently monitoring the final DB integrity check.]]> ../ Tue, 01 Nov 2011 08:00:00 AM
The larger DB is about 25% completed while the second largest DB is still checked as errors were detected.]]> ../ Tue, 01 Nov 2011 06:49:00 AM
Update 6:49 AM EST The Engineers are still monitoring their DB integrity checks. At this time the largest DB check is progressing smoothly.]]> ../ Tue, 01 Nov 2011 04:41:00 AM
Update 4:41 AM EST The larger DB check has started showing progress while the second to largest DB on final steps of the integrity check.]]> ../ Tue, 01 Nov 2011 04:00:00 AM
Update 4:00 AM Eastern There hasn't been any significant progress change as the checks are still underway.]]> ../ Tue, 01 Nov 2011 02:30:00 AM
Update 2:30 AM Eastern The integrity check is still underway. Unfortunately the first check is on the last step. ]]> ../ Mon, 31 Oct 2011 10:30:00 PM
Update 10:30 PM EST The integrity check is performing clean up and is on the final step of the check. We've begun the check on the last DB]]> ../ Mon, 31 Oct 2011 08:55:00 PM
Update 8:55 PM Eastern The integrity check is 90% completed.]]> ../ Mon, 31 Oct 2011 08:05:00 PM
Update 8:05 PM Eastern The current DB check is 2/3 completed. At this point we estimate being able to mount the database on or before 10PM]]> ../ Mon, 31 Oct 2011 08:00:00 PM
The larger DB check has started showing progress while the second to largest DB is still on the final step of the integrity check.]]> ../ Mon, 31 Oct 2011 07:30:00 PM
There hasn't been any significant progress change as the checks are still underway.]]> ../ Mon, 31 Oct 2011 07:00:00 PM
The integrity check is still underway. Unfortunately the first check is on the last step which does not have a progress bar.]]> ../ Mon, 31 Oct 2011 06:02:00 PM
The integrity check is performing clean up and is on the final step of the check. We've begun the check on the last DB]]> ../ Mon, 31 Oct 2011 05:45:00 PM
The integrity check is 90% completed.]]> ../ Mon, 31 Oct 2011 05:21:00 PM
The current DB check is 2/3 completed. At this point we estimate being able to mount the database on or before 10PM]]> ../ Mon, 31 Oct 2011 05:00:00 PM
The integrity check is about 1/3 completed. At this point we are going to wait for completion before starting the integrity check on the DB3]]> ../ Mon, 31 Oct 2011 05:00:00 PM
continue to monitor the repairs and will update this posting once the next step is reached.]]> ../ Mon, 31 Oct 2011 04:55:00 PM
Update 4:55 PM EasternThe integrity check is taking slightly longer than calculated. We will continue to monitor the repairs and wi]]> ../ Mon, 31 Oct 2011 04:00:00 PM
Update 4:00 PM EasternThe second to largest database is on step 3 of 5 for the consistency check. We anticipate having this databas]]> ../ Mon, 31 Oct 2011 03:45:00 PM
We've mounted half of the databases and are now running consistency checks on the larger databases.]]> ../ Mon, 31 Oct 2011 03:40:00 PM
The databases were offlined by Exchange and now in a dirty shutdown state. Before we can remount the databases we must run consistency check]]> ../ Mon, 31 Oct 2011 03:38:00 PM
The databases for DEWEY were forcefully taken offline by Exchange. We are working on diagnosing the issue.]]> ../ Mon, 31 Oct 2011 03:33:00 PM
We've replaced a failed drive on the DEWEYMBOX1 server. Unfortunately to bring the server back to a healthy RAID we must rebuild the array. ]]> ../ Fri, 28 Oct 2011 07:34:00 AM
Their may exist a possibility for Delivery Delay issues on Outbound-JR due to an Open Relay. The source of the traffic has been suspended.]]> ../ Thu, 27 Oct 2011 06:08:00 AM
The ED Engineers are currently investigating an issue with Delivery Delay notifications being sent by the LA LiverArchive cluster.]]> ../ Thu, 27 Oct 2011 06:08:00 AM
The NDR issue has been resolved. The issue was caused by a unprepared HUB node being inadvertently deployed by the Exchange system. ]]> ../ Wed, 26 Oct 2011 03:38:00 PM
The issue was caused by a newer HUB node for Los Angeles that was activated automatically by Exchange before it was ready for use.]]> ../ Wed, 26 Oct 2011 02:26:00 PM
This is only affecting the LA node. Normal mail flow is unaffected.]]> ../ Wed, 26 Oct 2011 02:26:00 PM
Researching delays from LiveArchive]]> ../ Wed, 26 Oct 2011 09:07:00 AM
The Engineers have been able to mitigate the issue. It appears to have been an issue with the IIS service. Service restarted and online.]]> ../ Wed, 26 Oct 2011 09:07:00 AM
The Engineers are investigating an issue with accessing the backup74 server.]]> ../ Tue, 25 Oct 2011 07:02:00 AM
The ED Engineers have become aware of a potential issue with Outbound Delivery Delays due to a DDOS attack through an Open Relay.]]> ../ Tue, 25 Oct 2011 07:02:00 AM
The Engineers have been continuously monitoring the Outbound SmartHost traffic. There should be no interruption to the mail flow.]]> ../ Mon, 24 Oct 2011 10:00:00 PM
We will be installing rollup updates across the LOUIE nodes]]> ../ Sat, 22 Oct 2011 07:50:00 PM
We received 2 reports of issues with connections to LOUIE. We were initially unable to reproduce the problem upon out initial research.]]> ../ Sat, 22 Oct 2011 07:50:00 PM
The scripted service completed its restart procedure, logging resumed as normally full connectivity was restored to all client connectivity.]]> ../ Wed, 19 Oct 2011 04:27:00 AM
The Engineers have been able to mitigate the Outlook Anywhere connectivity issue. Outlook Anywhere services should be fully operational.]]> ../ Wed, 19 Oct 2011 04:27:00 AM
The Exchange Engineers are currently investigating an a connectivity issue with the Della Server. OWA access at this time is available.]]> ../ Mon, 17 Oct 2011 11:01:00 AM
We are restarting the information store on DEWEY. We will be scheduling a reboot for after hours to apply memory changes.]]> ../ Mon, 17 Oct 2011 11:01:00 AM
DEWEY has gone down for the reboot and is now in POST.]]> ../ Mon, 17 Oct 2011 11:01:00 AM
The reboot has completed and we are finishing service checks for availability.]]> ../ Mon, 17 Oct 2011 11:01:00 AM
In the interest of bringing live mail up to speed we are going to freeze the current queues and move them off for processing later today.]]> ../ Mon, 17 Oct 2011 11:01:00 AM
Between 2:45PM and 3:10 PM we seen drastic performance issues with DEWEY that was linked to an unused failed drive that activated.]]> ../ Mon, 17 Oct 2011 11:01:00 AM
Mail delivery is back on speed on DEWEY and is still being monitored.]]> ../ Mon, 17 Oct 2011 11:01:00 AM
Between 2:45 and 4:30 PM there were some clients who reported the inability to send out. Solved after restarting the mail submission service]]> ../ Mon, 17 Oct 2011 11:01:00 AM
Half of the upgrades were completed last night and so far the queues and performance are on par.]]> ../ Mon, 17 Oct 2011 11:01:00 AM
We are rebooting DEWEY mbox1 to prevent further service issues throughout the day for DEWEY users. We do not anticipate another reboot ]]> ../ Mon, 17 Oct 2011 11:01:00 AM
Service to DEWEYMBOX2 has been restored.]]> ../ Mon, 17 Oct 2011 11:01:00 AM
We will be powering down DEWEYMBOX2 to add an additional storage controller. Maintenance should take 20 minutes.]]> ../ Mon, 17 Oct 2011 11:01:00 AM
We are adding an additional CAS server for DEWEY. Once in production partners can opt to use this additional RPC/HTTPs server ]]> ../ Mon, 17 Oct 2011 11:01:00 AM
We've created additional databases across the mailbox servers in DEWEY. We are balancing users across the DBs to alleviate pressure]]> ../ Mon, 17 Oct 2011 10:38:00 AM
At this time our Engineers are researching the connectivity issue on the Dewey Server.]]> ../ Mon, 17 Oct 2011 09:52:00 AM
We've received a report from a partner about this affecting outbound mail, but we haven't received additional reports from other partners.]]> ../ Mon, 17 Oct 2011 09:52:00 AM
We've noticed increased mail delivery times for messages inbound to the DEWEY network and affects mailboxes not on DEWEYMBOX2]]> ../ Fri, 14 Oct 2011 04:10:00 AM
The Exchange Engineers are currently Investigating a connectivity Issue with the server. OWA is accessible and mail flow not being affected.]]> ../ Fri, 14 Oct 2011 04:10:00 AM
The server accessibility issue has been resolved. The engineers confirmed full availability of all services and mail flow is unaffected. ]]> ../ Thu, 13 Oct 2011 07:18:00 PM
Service has been restored as scheduled.]]> ../ Thu, 13 Oct 2011 07:18:00 PM
We've detected hardware issues on our redundant hardware. We're replacing faulty node in the cluster.]]> ../ Thu, 13 Oct 2011 11:44:00 AM
Investigating slow mail delivery to dewey]]> ../ Wed, 12 Oct 2011 04:50:00 PM
We've identified an issue when a free mail service user replies from the encryption portal]]> ../ Wed, 12 Oct 2011 04:50:00 PM
We've tested and resolved this issue. It was caused by our free email outbound restrictions to stop spam outbreaks caused by open relays.]]> ../ Wed, 12 Oct 2011 03:48:00 PM
RIM issues with BES service. Details in the following article... http://news.cnet.com/8301-30686_3-20119163-266/blackberry-service-issues-s]]> ../ Wed, 12 Oct 2011 08:11:00 AM
Our Engineers discovered a hardware issue with the Node. Necessary replacements being done. Mail Flow was no affected.]]> ../ Wed, 12 Oct 2011 08:11:00 AM
Server unavailable. Investigating Issue with an outbound node. Outbound mail flow should not be affected. ]]> ../ Sun, 09 Oct 2011 12:19:00 PM
Looks it was a rogue Windows update that caused the issue. The service has been fully restored.]]> ../ Sun, 09 Oct 2011 12:19:00 PM
We're investigating an issue with Matilda. We'll provide more information as it becomes available.]]> ../ Thu, 06 Oct 2011 08:22:00 PM
The issue is fully resolved, total time elapsed was roughly 11 minutes.]]> ../ Thu, 06 Oct 2011 08:22:00 PM
Virus engine update issue caused some scanning processes to back up tonight.]]> ../ Thu, 06 Oct 2011 08:02:00 AM
The issue has been resolved. There was some hardware failure that has been replaced. Mail was not lost and should be delivered accordingly.]]> ../ Thu, 06 Oct 2011 08:02:00 AM
Server Inaccessible]]> ../ Wed, 05 Oct 2011 08:06:00 AM
The ED Engineers have mitigates the issue. It appears to have been a network conflict in ED. SPAM Reports will now flow as scheduled.]]> ../ Wed, 05 Oct 2011 08:06:00 AM
SPAM Reports]]> ../ Mon, 03 Oct 2011 03:06:00 PM
We are currently researching delays with OB3]]> ../ Sun, 02 Oct 2011 11:40:00 PM
outbound1.exchangedefender.com was down for maintenance. No outage should have been suffered for all clients that were setup correctly.]]> ../ Sat, 01 Oct 2011 05:48:00 PM
Our monitoring software is showing a blacklog of a handful of messages on inbound90. ]]> ../ Sat, 01 Oct 2011 04:43:00 PM
Outbound emails from DEWEY are not leaving the HUB server]]> ../ Sat, 01 Oct 2011 04:43:00 PM
We've switched the smart host servers on DEWEY hub to deliver to the smart host via IP and not via DNS. Mail flow has resumed on DEWEY]]> ../ Thu, 29 Sep 2011 12:15:00 PM
Researching reports of web services inaccessible.]]> ../ Mon, 26 Sep 2011 03:23:00 PM
After 48 hours of monitoring and partner reports, this issue is fully resolved.]]> ../ Mon, 26 Sep 2011 08:41:00 AM
We're rolled out a fix since this morning. We are monitoring the traffic to ensure that the issue is resolved.]]> ../ Mon, 26 Sep 2011 08:41:00 AM
Messages intended for our Dallas LiveArchive cluster not being accepted in our LA Cluster]]> ../ Mon, 26 Sep 2011 08:30:00 AM
We've received overnight reports of emails going to Yahoo! being rejected as relaying denied. We found the issue and resolved it.]]> ../ Mon, 26 Sep 2011 08:30:00 AM
Our Yahoo relay was not accepting connections from our backup relay nodes recently installed in Los Angeles.]]> ../ Thu, 22 Sep 2011 08:16:00 AM
This issue has been fully resolved.]]> ../ Thu, 22 Sep 2011 05:48:00 AM
Connectivity issues have been resolved. Our Engineers have informed use that some of the services did not starting as expected.]]> ../ Thu, 22 Sep 2011 05:48:00 AM
Server Inaccessible]]> ../ Wed, 21 Sep 2011 12:03:00 PM
This issue has been fully resolved.]]> ../ Wed, 21 Sep 2011 12:01:00 PM
Folks have reported seeing delay notices from our new LiveArchive nodes.]]> ../ Tue, 20 Sep 2011 08:55:00 AM
We're investigating some connectivity issues to the server.]]> ../ Tue, 20 Sep 2011 08:55:00 AM
Service has been restore however there may be some delays in mail delivery due to this.]]> ../ Tue, 20 Sep 2011 08:20:00 AM
Cross forest authentication was resolved on this case.]]> ../ Tue, 20 Sep 2011 07:26:00 AM
Server Inaccessible]]> ../ Tue, 20 Sep 2011 07:26:00 AM
Connectivity issues have been resolved on Scrooge. There were issues with some Services not starting as expected.]]> ../ Mon, 19 Sep 2011 01:46:16 PM
We are physically moving backup73 which will involve service interruption for up to 30 minutes]]> ../ Mon, 19 Sep 2011 01:43:00 PM
We are physically moving backup90 which will involve service interruption for up to 30 minutes]]> ../ Mon, 19 Sep 2011 09:59:00 AM
We're currently investigating a report authentication looping on the Ludwig Sharepoint farm.]]> ../ Sat, 17 Sep 2011 10:02:00 PM
We're investigating connectivity issues with our linux web farm. We'll provide additional details when they become available]]> ../ Sat, 17 Sep 2011 10:02:00 PM
This was resolved earlier. It appears there was port issue affect some of the connections to the farm. ]]> ../ Fri, 16 Sep 2011 04:26:00 PM
The new upgraded PDU units have been installed. We will be redoing network and power connections to provide easier management in the future.]]> ../ Fri, 16 Sep 2011 04:26:00 PM
We will be shutting off mail delivery to the Dallas network in the next 10 minutes and routing all traffic through our backup data centers.]]> ../ Fri, 16 Sep 2011 04:25:00 PM
Tonight we will be performing data center upgrades including network and power]]> ../ Fri, 16 Sep 2011 06:44:00 AM
The issue appears to have been a routing issue by the DC during our upgrade. Inbound mail was routed to outbound nodes as well as inbound.]]> ../ Fri, 16 Sep 2011 06:44:00 AM
Mail delivery through Dallas appears to be going off without hitch. We believe we've found the cause in LA and are still investigating.]]> ../ Fri, 16 Sep 2011 06:17:00 AM
The issue appears to be limited to a few servers in Los Angeles. We've disabled routing to LA as we continue to investigate this issue]]> ../ Fri, 16 Sep 2011 06:17:00 AM
We've received a couple reports from partners about receiving "Relaying Denied" errors on mail delivery.]]> ../ Thu, 15 Sep 2011 10:39:00 PM
All work has been completed. The new infrastructure has been deployed and the existing infrastructure is back online.]]> ../ Thu, 15 Sep 2011 09:00:00 PM
The physical move of the power, network, and hardware has been completed. We're in the process bringing up the network services.]]> ../ Thu, 15 Sep 2011 09:00:00 PM
At 9:00 PM Eastern we will be replacing the APCs in our Los Angeles datacenter.]]> ../ Thu, 15 Sep 2011 12:14:00 PM
The automated alerts were due to a lowered threshold. Mail delivery is up to speed and users may have seen a brief delay]]> ../ Thu, 15 Sep 2011 12:14:00 PM
We are receiving alerts from our monitoring about increasing latency across the Dallas network]]> ../ Tue, 13 Sep 2011 09:45:00 AM
There was an issue with the networking interface on the Database server. We've resolved that problem and all affected sites are online.]]> ../ Tue, 13 Sep 2011 09:28:00 AM
Sharepoint Issues on LOUIE]]> ../ Tue, 13 Sep 2011 09:28:00 AM
The issue has been linked to the Database server that the farm is linked to. We're continuing work to restore the service ASAP.]]> ../ Tue, 13 Sep 2011 12:11:00 AM
The second replacement drive passed the post testing. Sharepoint service on LAUNCHPAD has been restored.]]> ../ Mon, 12 Sep 2011 05:20:00 PM
Backup restore was completed, however the destination drive tested bad during post testing. We are currently repeating with a new drive.]]> ../ Mon, 12 Sep 2011 12:39:00 PM
The OS primary RAID array suffered a controller failure. We are in the process of restoring a backup taken about 30 minutes before the card ]]> ../ Mon, 12 Sep 2011 12:39:00 PM
We're in the middle of advanced troubleshooting on Launchpad. We will update as more details become available.]]> ../ Mon, 12 Sep 2011 08:12:00 AM
The SharePoint Engineers noticed the issue was with one of the SharePoint Windows services. A restart of the service resolved the problem.]]> ../ Mon, 12 Sep 2011 07:50:00 AM
SharePoint Farm]]> ../ Thu, 08 Sep 2011 06:42:00 PM
"ExchangeDefender does not protect this email address" issue resolved for new clients added today.]]> ../ Thu, 08 Sep 2011 05:58:00 PM
New user additions were taking longer than expected to propagate. Root cause has been identified and resolved.]]> ../ Thu, 08 Sep 2011 10:30:00 AM
We've found and resolved an issue with our peak load balancing and the volume has normalized.]]> ../ Thu, 08 Sep 2011 10:30:00 AM
Our inbound networks in Los Angeles and Dallas are showing higher than normal queue volumes, slightly increasing delivery times.]]> ../ Thu, 08 Sep 2011 05:34:50 AM
Replication of relay access tables across the outbound-jr servers was lagging; Services were cycled and operations resumed as normal.]]> ../ Thu, 08 Sep 2011 12:00:00 AM
On Thursday, 9/8 at 8:00 PM we will be renewing the SSL for LOUIE]]> ../ Thu, 08 Sep 2011 12:00:00 AM
Investigating issues with outbound-jr1]]> ../