We’ve experienced explosive growth in the current quarter and some of you have experienced a few of those a growing pains with us. This weekend we will be working on a massive build out to our secondary data center on the mail processing side of the services. Los Angeles will not be available for mail processing this weekend. This should NOT have any impact on standard MTAs that can handle multiple DNS results.
The only true impact expected is mail releases. Releases which reside in LA , that generally are a lot of less due to the lack of parity between the sites will not release until the expansion is complete. Once the expansion begins its last stage as the servers begin to come online all releases will still be queued and at that juncture should release.
Please follow the NOC above in case anything unexpected becomes affected.
-Update 4:55 EST this work has started
-Update 5:40 EST XD Data Replication is online, nodes to follow
-Update 6:55 EST 80% capacity moved for exchangedefender
-Update 11:09 EST All work completed across all services.
Sunday Post Report
We located an issue with the SPAM actions not being correct for all users, we are working to resolve that this morning if you receive a report of this that goes beyond today at noon please open a ticket, if the timestamp of the message is before noon today, please consider the issue resolved for your end users.
We received additional reports that some SPAM actions still weren’t functioning correctly. We have resolved this issue if you see any emails time stamped after 7 PM EST please let us know. If its before then, please consider the issue resolved.
This weekend we will be performing maintenance on the DEWEY cluster. This will be done to improve the performance and overall experience for users whom opted to remain on the legacy Exchange 2007 servers. Throughout the weekend users may notice a disconnect from Outlook while we move their mailbox to a new database. Note that this will not affect all users. When the move has been completed Outlook will prompt the user with the following message:
Once the users clicks “OK” and restarts Outlook they will be automatically connected to their mailbox on the new database.
We have launched phase 1 of the LiveArchive4 succesfully. This means that real time email is available through https://admin.exchangedefender.com/livearchive.php.
Phase 2 which is in progress and will take a few weeks is the migration of old items. This is currently ongoing. To avoid any confusion we’ll cut to the bottom line. If your mail is not at the new URL, then its still at the old URL. So your email for up to a year is still with us.
Current gremblins we have noticed:
Dead: There WAS (as it was resolved yesterday) an issue with the RBL zone reloads that would cause some DNS timeouts and rejections to livearchive.
Current: This one is more of an annoyance than anything else but be aware. Due to the fact that we’re moving platforms, there’s no “database=>database” move. We’re moving content. Unfortunately, this is triggering read-receipts.
Phase 3. Vlad will blog about it once it’s ready.
We are currently doing massive redesigns to Compliance Archiving some aspects are not available. Your client’s data is still safe alot of the front end is getting some re-balancing done from the back end. By that I mean, its inaccessible for viewing at times but your data is fine. We’re working on balancing the rendering as well.
One of the benefits of ExchangeDefender is that we allow our clients to process both their inbound and outbound mail through ExchangeDefender and eliminate SPAM in both directions. Until now we allowed IP based relay on our outbound network even if the domain name was not protected by ExchangeDefender. While this was a matter of convenience, the explosion of compromised servers and virus distribution is starting to create an “open relay” problem for ExchangeDefender: where third party domains are relaying through our clients servers and forging the From: address.
Effective March 22nd, ExchangeDefender will only permit outbound relay from email domains that are protected by ExchangeDefender and have their MX records pointed to ExchangeDefender nodes. Any attempt to send email through the ExchangeDefender by a 3rd party domain not protected by ExchangeDefender will be rejected.
If you have a business case scenario that requires you to relay third party domains that are not protected by ExchangeDefender (because you don’t own the domain name, do not have administrative control, shared domain with another organization) we recommend that you relay those messages directly via your SMTP service / IP address instead of attempting to route them through the ExchangeDefender smarthost.
Note: This policy change will not affect any legitimate traffic going through our outbound network. It only picks up illegal spoofed traffic based on the From: line (if it includes a domain name not protected by ExchangeDefender)
There was an issue earlier today with one of the RBLs we use listing non-spam URLs. This resulted in issues where some clients’ outgoing messages were being blocked with a message from Exchange Defender saying the message contained a Spam URL.
That issue was resolved but there was a problem in the process that the RBL used, which made it so that it timed out and didn’t function at all, causing sporadic blocks in messaging inbound. This is not on our servers, nor on the recipient or sender MTAs.
The issue is resolved completely now and mail flow should be back to normal. You may still get complaints for the next few hours as the issues get ironed out completely. We would like to extend our deepest apologies for the inconvenience caused by this to your clients. We’re in the process of rewriting some of the code to ensure that the dependability on some of our providers is not so absolute.
As an edit to address follow up questions:
This DNS issue would affect certain mail delivery and processing speeds. But we’re flushing through it as quickly as possible.
We’re working on UI enhancements for the Service Manager within the our Support Portal. If you’re continuing to see any issues with please open a ticket with screenshots so we can make sure the development team is aware of any bugs they may have to address.
Our sip provider is currently having issues. Remember our portal is SLA backed 24/7. If you need help open a ticket or a LiveChat within the portal for support issues.
Update: Our provider has resolved their issues.
We’re currently working an issue with DB4 which is similar to the issues faced previously with DB3_1 . The issue is related to a storage controller issue and we are currently running an integrity check/repair on the database
We sincerely apologize for the length of this issue. We understand this is extremely frustrating, as it is extremely frustrating for us to provide this level of service. This is currently not only our top priority but only priority and our top Exchange engineers are on this task and this task alone until resolved.
Update 6:00 PM Eastern: We are still performing work on DB4 for the affected mailboxes. We would like to extend the option for users to utilize “recovery mode” / dial tone. Recovery mode is a temporary new “blank” mailbox where users can work with their live mail (after the outage occurred) as they normally would. Since the recovery mailbox is a “new” mailbox, there wont be contacts, calendars, etc. If you have active sync users or iPhone users then you should have then set their Email, Calendar, and Contact sync to a manual sync as the phone will sync with the empty mailbox.
When a mailbox is manually switched to a temp database Outlook will now prompt the user to either use their Temporary data (new data) or open their previous cached mailbox data
Outlook “Recovery Mode” prompt is a feature set introduced in Outlook 2007 to safely handle mailbox or server side issues that lead to mailbox configuration changes. After the server goes through a dial tone migration, Outlook will notice the content change in the mailbox and locks the old OST cache file from downloading any new mail. Once Outlook goes into Recovery Mode, the user will receive a prompt upon launching an Outlook profile asking if the user wants to use “Old data” or “Temp data” – in simple terms, Old data refers to the OST cache file on the client machine, and temp data provides online access to the mailbox to view new items that arrive – while in temp data, Outlook will run in Online mode and not Cached mode, so the overall experience will be slower for users who are used to the speeds from cached mode.
If a partner would like to utilize a recovery mailbox for mailboxes in their domain then we would need a new support request opened with following subject:
Dialtone Request: domain.com
Where domain.com is the client domain. By opening the request as a new request this will allow us to ensure all requests are properly completed and documented.
January 2 2013
Update 4:30 PM Eastern: The repair / integrity check is near 55% completed on the last step. We anticipate being able to mount the database by 8PM Eastern tonight which we will then deliver all queued mail to the mailboxes. Once all queues are flushed we will then swap users who are in dial tone mode back to their home database and subsequently merge the new and existing data. We want to thank everyone again for their patience as we’ve had a positive response to the dial tone mailbox setups.
Update 5:45 PM Eastern: We’ve successfully mounted DB4 on LOUIE. We’ll be resuming mail delivery shortly.
Users on LOUIE DB3_1 are currently unable to access their mailboxes due to a hardware level issue with the storage device. The database was a temporary database used as a temporary holding spot for mailboxes that were moving from DB3 which was being phased out on LOUIE. Unfortunately since this was a temporary database there was no anticipated need to make multiple database copies available as the extra over head could cause production performance issues. To rectify the issue we’ve been running a hardware RAID level resync to ensure all data is in the best state possible. We do anticipate the issue being resolved before the start of business on Monday.
As a reminder, partners are able to use LiveArchive (https://livearchive.exchangedefender.com) to access their live running mail during the outage
Update 9:07 AM: We are currently running an eseutil integrity check on the log files for DB3_1. This check will ensure that all the data that is present on the database is correct and true, as well as any uncommitted data can be finally committed. This process is expected to take up to two hours to complete. If the replay and check complete without error we will then be able to safely mount the database. All mail that hasn’t been delivered is still queued and waiting for delivery.
Update 5:30 PM: The database was successfully mounted after the integrity check completed. In order to prevent this from reoccurring we are adding a database copy to this database. Please keep in mind that this database was a transitional database, used temporarily to move users from DB3 which was in the process of being phased out. We’re extremely sorry for the inconvenience this has caused and we will include transitional databases in DAG setups going forward.
The servers in the Australia network (2007 Exchange) has suffered a major failure which has ultimately led to the decision to decommission it as a restoration will not bring service online seamlessly (without recovery and resetup by clients)
Unfortunately the recovery from Australia is not something we believe will be completed in the next 48 hours at a minimum. To accommodate this and provide users with an immediate solution to continue working we’ve proactively recreated all users on Australia (2007) on to Matilda (2010). Matilda has redundancy and we’ve already sent over an additional server to add to the Matilda cluster.
The new server address is cas.matilda.exchangedefender.com and the autodiscover record should be setup as autodiscover.matilda.exchangedefender.com
We’ve changed the target delivery location in ExchangeDefender to matilda. You can also login to livearchive to work with mail while the change over to matilda occurs.
Partners who have clients with cached outlook profiles can open the current profile (On Australia) and then export the cached data to .pst files. Any users who do not have cached data or any public folder data will not have an ETA on restoration for at least 48 hours (until we can process the drive’s data)
Update 10/25/12 9:45 PM Eastern: We are in the process of copying over items from livearchive to matilda mailboxes to import the past three days of mail from livearchive. We anticipate the process being completed in under 4 hours.
This morning around 3:30 AM Eastern we received alert that the hosted Exchange 2010 network in Australia (MATILDA) went offline. Upon login via KVM we noticed the server was repeatedly rebooting after faulting to a blue screen, however no diagnostic information was provided. It was soon determined that the operating system needed to be repaired which completed around 5:20 AM Eastern. Once we were able to successfully boot into Windows we performed a database integrity check to ensure no actual data was corrupted or lost which completed around 7:45 AM Eastern and service was restored.
Once service was restored we looked into the server logs which provided no information or logged entries regarding the server fault, however, a memory dump was created. Unfortunately the memory dump wasn’t of much help and the issue appeared to possibly be hardware related. We temporarily switched service for Matilda over to the backup node as we tested the hardware components. We were able to determine that the issue was related to a power supply that randomly dropped output voltage during high load. After replacing the power supply we ruled that the server faulted while running a backup. With regard to the OS repair we can only deduce that a system file was corrupted from the fault and needed to be replaced.
Over the weekend our Linux web hosting infrastructure was upgraded to PHP 5.3. The previous release of PHP 5.1.6 was getting out of date for a lot of social applications that dominate todays deployment base. You will now be able to run the latest WordPress, Joomla and other CMS and shopping cart sites without a problem.
If you have any legacy PHP applications that were written in PHP4 and early PHP5 days, you may notice that some functionality does not work. Consult your software vendor for an update that functions with PHP 5.3 and above.
If your application is not supported on the new platform of PHP and you do not have the resources to make an immediate switch to an alternative, we do have a www2legacy web infrastructure in place that will support older applications. We will make the accommodations and the switch to the legacy platform for a small one-time fee but do keep in mind that the legacy platform will be discontinued in March of 2013. We have provided more than a 1 year announcement of our intention to upgrade our network infrastructure to the latest PHP so if this caught you by surprise we will still do what we can to make sure you have a smooth transition.
For a list of features in PHP 5.3.x and old deprecated functions:
This is probably an excellent place to start your troubleshooting on outdated code. If you decide you’d like to stay with the legacy code please open a support request at https://support.ownwebnow.com if you’d like further information.
This Friday night we will be performing maintenance to the ROCKERDUCK cluster that may interrupt client connections for up to 10 minutes. The purpose of this upgrade is to provide a more intelligent network routing for our mailbox database replication service. We will work our hardest to make the required changes without interrupting client connections.
We anticipate starting the work around 9PM on Friday August 24th.
Users on RDDB1 are unable to access their mailbox data as the mailbox database is currently offline. We are performing surface check on the database as the database health is indicating minor corruption. We are unable to activate the passive copies of the database without the surface check being completed as we are unsure when the corruption occurred and could have been copied to the passive copies. We anticipate that the database will be offline until at least 3PM Eastern. Only 9% of user base is affected and we are working diligently to restore access to customer mailboxes on RDDB1
Update 3:06PM Eastern: We are running two processes to try and restore access as soon as possible: repairing the live database and running a restore from the last backup (7/16/12) and then replaying the committed data to the restored backed up database to bring it up to date. We will go with whatever process completes first as both will bring the database back online in a healthy state. The restored backed up data base is copying log files and will complete around 5PM. The live data base repair is at 50% on step 1/5 and its ETA is unknown
Update 4:13 PM Eastern: We’ve successfully mounted RDDB1 on ROCKERDUCK and confirmed user access.
Unfortunately this was not something that could have been avoided by any redundancy available as this was a software level corruption which isn’t protected by a DAG. Essentially, the DAGs prevent against issues with unavailability between mailbox nodes or networking issues, but when a corrupted log file gets committed, it then gets distributed to the passive nodes. Since we couldn’t confirm if the corrupted file was or wasn’t distributed to specific nodes when caught we felt it wasn’t safe to remount the databases with the lingering possibility of encountering a more serious issue in the near future.
We are planning on introducing a lagged mailbox database copy which will ‘lag’ when it commits log files to the passive copy. By implementing a lagged copy if we ever experience corruption again we can restore instantly to the previous day as the lagged copy wouldn’t have copied/committed the corrupted data
The work described below is schedule to begin at 9:00 PM Eastern May 29th 2012
- LOUIE – LOUIEMBOX1 & LOUIEMBOX2 – Update Network Driver
Will cause brief interruption, 15-30 seconds while the driver updates
- LOUIE – update exchange to service pack 2
Will not cause interruption to clients
- ROCKERDUCK – Reseeding databases between RDMBOX1 and RDMBOX2 for fail over
Will not cause interruption, but OWA users may see slight delays in accessing content (including public folders) since the replication is going to use the MAPI NICs instead of the replication NIC. This will only be during night as to not flood the network during the day
- ROCKERDUCK – Redistributing disk layout on RDMBOX3
RDMBOX3 is one of the additional fail over clusters and does not actively hold any mailbox databases
Today (5/14/12) there was an outage with ROCKERDUCK between 11:50 AM – 12:20 PM (Eastern) that affected client access to mailboxes. In short the outage was caused by our Los Angeles site experiencing Active Directory communication issues with Dallas, eventually taking LA offline. Normally, this event wouldn’t cause any issue as Dallas would be able to maintain quorum as Dallas holds the majority vote in the DAG quorum. Unfortunately the tie breaking voter (MBOX3) was offline for maintenance (As it does not actively host DBs – it’s the internal failover server for Dallas), which before the communication issue between sites left the vote as 4/5 online. Once LA experienced connectivity issues the vote dropped to 2/5 online and was forced offline. Our first alerts from monitoring started to come in at 11:52AM and we were able to respond by 11:55AM to the issue and post an advisory. Around 12:10 PM Eastern we were able to reestablish quorum in Dallas and bring the cluster online. Around 12:15 PM MBOX1 mounted all active databases and by 12:20 PM MBOX2 mounted all databases. Since this event was a communication issue between sites, the passive site was marked as Blocked (Unable to automatically mount databases for fail over) as network interruptions were detected.
This weekend our international SIP trunk provider will be doing weekend long maintenance on the infrastructure that hosts our accounts. During the period of 4/27/12 – 4/30/12 our international partners will have to dial our main number, 877-546-0316
Update 5/2/2012: The reinstallation has completed and service is resumed normal operation.
Update 5/1/2012: We are in the process of reloading the operating system on LOUIEMBOX1. As of now the only affected service is mail enabled public folders, however, all mail will be queued until the reinstallation of Exchange has completed. We anticipate being done with all work by midnight.
Update : This has been scheduled for Friday at 3:00 PM Eastern
In preparation of the reload of LOUIEMBOX1 we’ve moved all mailboxes hosted on to other mailbox servers in the cluster. Last month we created a replication of all Public Folder content to all mailbox servers in LOUIE from LOUIEMBOX1. The final step is to change all ExchangeDefender delivery points for LOUIE customers away from LOUIEMBOX1. This step should fall in line with our seamless upgrades and should not be noticed by the client. Unfortunately in past experiences some public folders would not receive messages from the outside after a replica is taken offline. We believe this was a previous bug with Exchange and has since been resolved, however, we’d like to make all possibilities known ahead of time.
Due to the number of public folders and public folder content, we will be unable to validate mail delivery across all mail enabled public folders during the reload.
If any clients experience mail delivery delays to their public folders, please open a support request with the email address of the mail public folder and our support team will immediately look into the issue.
Update: The below work has been partially completed. We’ve changed the migration strategy to use public folder replication instead of an in place reload. This decision will extend the maintenance interval into next weekend but will provide no downtime for mailboxes or public folders
Over the weekend of March 30th– April 1st we will be performing upgrades to the LOUIE DAG (Mailboxes) and Public Folder databases. The changes that will begin during this maintenance cycle will improve the native automatic redundancy for LOUIE mailboxes and bring an overall greater experience for users. Due to the number of users and the size of data on LOUIE, the only way to complete the migration with the overall least amount of service interruption is to dismount the public folder database on Friday evening, and leaving it dismounted until Saturday afternoon.
Taking the public folder database offline will prevent ALL access to public folder data housed on LOUIEMBOX1, including mail enabled public folders between Friday evening until Saturday afternoon.
All public folders are replicated between multiple databases, however, unlike DAG protection, Public Folder databases work off a referral based system for locating copies and is highly unreliable for automated continuity.
Any users/partners who are concerned about the availability of their public folder data can open a support ticket and request that our team ensures their public folder data has an active, local replica.
We do not anticipate any interruption in the service of actual mail delivery or mailbox access on LOUIE.