As of 9:15 AM EST all services are back online.
We are addressing several performance issues on mail1.ownwebnow.com. We expect the resolution in a few minutes and will update this post at 9:15. As of 9:00, ability to send/receive mail from Own Web Now Linux Web hosted mailboxes is down.
Cause: Kernel page fault
We are receiving reports from partners on our POP3 server, MAIL1 about incoming messages being rejected as “Proper Authentication Required”. This appears to be an issue with the configuration of MAIL1 and our engineers are currently working to diagnose the issue.
Update 3:50 PM Eastern: The issue has been resolved and was related to a routing file for configuration.
Over the weekend we will be performing maintenance on EUROPE and DEWEY to resolve issues that have occurred twice this week. In short, drive space is being exhausted on both servers and preventing new messages from entering or leaving the network.
Saturday (1/29/11) at 9PM Eastern we will be performing the following maintenance
DEWEY: Moving the database path for users on DB1 from the current storage to a new storage array
EUROPE: Moving the database path for users on DB2 from the current storage to the secondary storage array.
Both moves are anticipated to take two hours to complete. During the moves, users will be unable to access their mailboxes through any medium. Mail that arrives during the transition will remain in queue until the work is completed.
Partners are strongly urged to inform clients about livearchive who require access to their mailbox during the maintenance schedule
Overnight many users received errors while attempting to connect to our admin.exchangedefender.com network. This issue was caused by the new backup systems that were recently put in place and has been addressed by 7:00 AM EST.
We apologize for any inconvenience this has caused our users, please rest assured that this issue has been addressed and that no mail was lost.
We are currently going through the public folder databases and replicas to discover the server with the latest copies of all customer content. During the scan, clients will be unable to access their public folders. We anticipate that this task will take up to 2 hours.
01/12/2011 10:00 AM Eastern – We experienced a DDOS attack this morning on our outbound network dedicated to our Exchange clients. There may have been a small amount of emails impacted, however this is only limited to a slight delay while we got the attack under control. The issue is resolved, the queues have been completely caught up and are currently delivering on time.
The monitoring software on LiveArchive has alerted about login issues across a few databases. We are in the process of diagnosing the issue.
Update 5:50 AM: The root cause of the login issues has been discovered. We are applying database upgrades to each database individually. While the database that a user is on is being upgraded, users will receive the warning that their mailbox appears unavailable – the error will go away as soon as the database is upgraded.
Update 9:45 AM: Three of the primary databases have been upgraded. We are continuing the updates across the other databases.
Update 12:28 PM: Five of the primary databases have been upgraded and mounting. We are continuing the updates across the other databases.
Update 10:50 AM: In the interest of restoring service across the board, we are going to move all users that still have databases waiting to be upgraded to a temporary database. This will allow the users to login to their livearchive mailbox and receive live mail, but the previously archive mail will not be available until the DB schema upgrade is completed.
Over the next three hours all the mail that has been queued over the past 24 hours will be delivered to the new temporary profiles.
After the schema upgrades are completed on all databases, we will begin to move mail from the new temporary profile back to the main profile
As part of our on going background maintenance with LOUIE, we’ve moved all user mailboxes from our active mbox server to our passive mbox server in preparation of the reinstall of mbox1.
Tonight (1/6/11) at 9PM we will be uninstalling the mbox role and HUB transport role from mbox1. Upon competition of the role uninstall, we will change the louiembox1.louie.exchangedefender.com DNS record to a CNAME record pointing to louiembox2.louie.exchangedefender.com. Finally, we will disconnect mbox1 for the rest of the week to ensure all dependent services have been moved before the reinstallation
This change is expected to be completely transparent to the users, and the only users that will be potentially affected are POP3/IMAP based users who rely on SMTP services to transmit mail. Outlook, OWA, ActiveSync and BES users will not be affected by the change.
If any POP3/IMAP clients are unable to send mail after the change, either flush their DNS so louiembox1 will resolve to louiembox2, or change the client configuration to use louiembox2.louie.exchangedefender.com