One of the benefits of ExchangeDefender is that we allow our clients to process both their inbound and outbound mail through ExchangeDefender and eliminate SPAM in both directions. Until now we allowed IP based relay on our outbound network even if the domain name was not protected by ExchangeDefender. While this was a matter of convenience, the explosion of compromised servers and virus distribution is starting to create an “open relay” problem for ExchangeDefender: where third party domains are relaying through our clients servers and forging the From: address.
Effective March 22nd, ExchangeDefender will only permit outbound relay from email domains that are protected by ExchangeDefender and have their MX records pointed to ExchangeDefender nodes. Any attempt to send email through the ExchangeDefender by a 3rd party domain not protected by ExchangeDefender will be rejected.
If you have a business case scenario that requires you to relay third party domains that are not protected by ExchangeDefender (because you don’t own the domain name, do not have administrative control, shared domain with another organization) we recommend that you relay those messages directly via your SMTP service / IP address instead of attempting to route them through the ExchangeDefender smarthost.
Note: This policy change will not affect any legitimate traffic going through our outbound network. It only picks up illegal spoofed traffic based on the From: line (if it includes a domain name not protected by ExchangeDefender)
Throughout the week of October 31st – November 4th we will be performing maintenance on our Dallas LiveArchive network. During the maintenance cycle we will redirect all LiveArchive requests to our newest LiveArchive network in Los Angeles, CA. This means that any user who browses https://livearchive.exchangedefender.com will be forwarded off to https://la.livearchive.exchangedefender.com
To allow users access to both networks during maintenance we’ve changed the IP value for livearchive.exchangedefender.com to 22.214.171.124 which has a global 302 redirect to la.livearchive.exchangedefender.com
Maintenance will include upgrades to the storage architecture, rebalancing users and implementing a new Web Services based API to provide greater control and logging during new user creation.
During maintenance users can attempt to login to their Dallas LiveArchive mailbox by browsing to it directly @ https://livearchive3.exchangedefender.com however there will be periods where mailbox databases are dismounted and data will be inaccessible.
We ask that partners keep in mind that the Los Angeles version of LiveArchive only retains 3 months of mail and was initially deployed at the end of September.
Between 4:30 PM and 5:30 PM few clients may have received bounce back messages referencing delivery to delivery to xd286#### from LiveArchive.
These notices were sent in error and was caused by LiveArchive tests that we’re conducting to setup redundancy for LiveArchive to Los Angeles.
On 7/20/2011 around 3:35 PM Eastern we started experiencing random packet loss across various services including Hosted Exchange and OWN Websites. Roughly around 3:45 PM Eastern, the random packet loss turned into a wide-spread service outage and lasted until 4:12 PM Eastern.
The incident appears to be faulted network driver on a Exchange monitoring server. Upon automatic recovery of the driver, the machine began to flood nearby network switches with invalid requests. Unfortunately the internal floods prevented access to the network analytic servers behind the DMZ. Since all machines received and responded to the request, all machines showed up as ‘flooding’ to the router and IDS was unable to determine the ‘source’ IP.
All services were essentially taken offline when the IDS started blocking traffic from the internal hosts. After we disabled the offending machine from the network and cleared IDS we were able to resume service across the board.
The biggest area of concern was the inability to contact us as the outage was occurring as the outage took down our support board and primary phone lines. We deeply apologize for the grief and trouble that this unexpected event caused and without saying, this has been the most impacting network event that we’ve experienced. We’ve implemented a new redundancy plan to our phone systems to handle global outages as this was the first time our phone systems were completely offline during a critical event.
We appreciate everything that our partners do for us and the patience that was extended yesterday as we definitely know that it was a very stressful event for our partners and their end users. As always we will continue to bring improvements to our solution stacks and address the areas where we may fall short.
As per the post on our NOC blog we are beginning to prepare for the hardware upgrades and maintenance that will last throughout the weekend.
As part of the preparation, we are disabling the indexing service on the current mailbox server to allow the disk IO resources to be dedicated to the user moves.
Tomorrow night (2/17/11) at 9PM Eastern we will be making adjustments to the CAS cluster for the LIVEARCHIVE network. During the adjustments, users may experience intermittent issues in accessing their mailbox. The maintenance is scheduled to last 30 minutes and service is expected to be restored by 9:30 PM Eastern.
Around 9:45 PM Eastern we started to receive alerts from our monitoring software about queue sizes and delivery speed in ExchangeDefender. Upon investigation, three ExchangeDefender nodes were being DDoS SMTP attacked..unfortunately, our servers were able to keep up with the number of open connections and corrupted messages, that it began to clog up delivery across the board.
There may have been some limited delays in email delivery for any messages that were in-scanning when the DDoS started.
The attack has been thwarted and we are currently implementing the blocks across the entire node grid.
Over the past two months we’ve seen a tremendous growth of ExchangeDefender users and utilization of LiveArchive. The increased load has prompted us to begin planning an upgrade for the LiveArchive hardware and network.
Throughout the month (Starting Friday) we will begin to bring the new hardware online and move selected databases to the new hardware. Due to the sizes of the databases, we will have to do a “dial tone” like migration. A dial tone migration will allow us to instantly switch the database that a user is assigned to without moving their previous mail. The whole purpose of the dial tone based migration is to constantly provide service to users without any downtime.
LiveArchive was designed to provide on demand access to new mail for a domain in the event of a local outage. In keeping this core concept in mind, users will always have access to their mailbox and any live mail during the migration. Unfortunately, in doing a dial tone based move, we will then have to merge the old mail content and the new mail content back together.
We will be doing the upgrades one database at a time, and moving a total of 10 databases to the newer hardware. The entire upgrade is estimated to take up to a month to complete.
In short, as we move each database, each user on that database will lose access to their previous mail until the move and merge is complete, but they will have the ability to work with live mail during the transition.
The monitoring software on LiveArchive has alerted about login issues across a few databases. We are in the process of diagnosing the issue.
Update 5:50 AM: The root cause of the login issues has been discovered. We are applying database upgrades to each database individually. While the database that a user is on is being upgraded, users will receive the warning that their mailbox appears unavailable – the error will go away as soon as the database is upgraded.
Update 9:45 AM: Three of the primary databases have been upgraded. We are continuing the updates across the other databases.
Update 12:28 PM: Five of the primary databases have been upgraded and mounting. We are continuing the updates across the other databases.
Update 10:50 AM: In the interest of restoring service across the board, we are going to move all users that still have databases waiting to be upgraded to a temporary database. This will allow the users to login to their livearchive mailbox and receive live mail, but the previously archive mail will not be available until the DB schema upgrade is completed.
Over the next three hours all the mail that has been queued over the past 24 hours will be delivered to the new temporary profiles.
After the schema upgrades are completed on all databases, we will begin to move mail from the new temporary profile back to the main profile
Access to livearchive is currently deprecated. Yesterday we noticed mail submission would randomly drop, so we increased our monitoring. Today, we’re noticing the same behavior, but it’s pointing more to the mailbox databases. We are currently taking multiple databases off at a time and running integrity checks.
We highly apologize for the inconvenience and any partner who has a down server that needs access to livearchive, please open a case with support and we will work out an alternative solution.
We received alerts from our monitoring software that messages on outbound1 were being rejected as relaying denied. We killed the sendmail process and resolved the issue. The bounce backs occurred between 11:20 AM Eastern and 11:35 AM Eastern.
Update 6:44 PM Eastern: The issue reappeared around 6:20 PM and began rejecting messages. We located the root cause, which was a faulty update from an AV vendor. We applied the previous definitions and await an update from the vendor.
Gmail is apparently experiencing technical issues. Several users have reported trouble sending mail to Gmail over the past hour. While we are unable to isolate it at this point we have contacted Gmail and are awaiting response.
The issue does not seem to be widespread, only about 10% of messages sent to Gmail terminate with the 5.x.x series error (permanent failure) and it is being issued by Gmail servers, not ExchangeDefender.
We have alerted Gmail and will update this blog post when they notify us of any problems being resolved on their end.
The livearchive service has been degraded and new mail is not being delivered to the server. We are in the process of truncating log files and optimizing the storage controller.
You may have noticed an unusual rise in the amount of SPAM getting through to select users on your domain. We have isolated dozens of domains against which large scale SPAM attacks have been raised, typically from trusted (reputable) mail infrastructure.
We have been isolating recent outbreak of SPAM that is originating from sources that are typically trusted and managed by larger providers. We have designed new rules, tested them against known sites and our own production domains and we can confirm that we expect the performance as of 2 AM EST to improve significantly. As usual, if you continue to see unusual SPAM levels please use the Outlook 2007 addin and report anything that gets through as the email@example.com alias will be discontinued shortly.
We are in the process of finalizing the upgrades to LiveArchive 3.0. Throughout the week we received complaints from a few partners about delays in mail being delivered to the server and mail being delivered to livearchive. We identified the problem as a bottleneck, which we implemented a temporary solution of la-edge.
During this final upgrade we will have to reboot servers in the livearchive network once before installing the new transport server. Throughout the night we will be flushing out overflow queues from the past 48 hour into LiveArchive mailboxes.
To address several new SPAM patterns and strains that are able to get around the current restrictions (and whitelist implementations) we will be rolling out an update to our AntiSPAM engine at 11:00 EST tonight.
During the 15 minute window around the switch, there may be an unusual amount of SPAM passing through as our databases are flushed, reloaded and propagated through the network.
We have already started receiving support requests regarding missing messages – all caused by not including the latest IP addresses in the IP restriction set.
If you are currently enforcing restrictions, please make sure you have the 126.96.36.199/24 IP address range in your access list:
Earlier today we launched ExchangeDefender LiveArchive 3, powered by Microsoft Exchange 2010. LiveArchive is a free business continuity solution that is constantly copying all your inbound and outbound messages as they are scanned by ExchangeDefender and in addition to delivering to your mail server or your mail recipients, it delivers to an enterprise deployment of Exchange 2010 in our data centers.
When your servers go down, or production environment is interrupted in any way, LiveArchive can be accessed through any browser or mobile web device and users can have access to up to 1 year worth of email (note: not a compliance or archiving solution)
When we went from Exchange 2007 to Exchange 2010 we simply created a new environment and created all the users from ExchangeDefender user databases. You can access your old accounts and your old email at http://livearchive2.exchangedefender.com
Your new accounts are at https://livearchive.exchangedefender.com and they all hold the same username as the one in our ExchangeDefender portal. If you changed your passwords via Outlook Web Access keep in mind that the password will now be synced to our ExchangeDefender user databases and only supported passwords are the ones at https://admin.exchangedefender.com. We will soon restrict the ability to change the password through OWA because users forget their passwords and it creates an enormous amount of support issues.
If you are running into password issues when logging into LiveArchive today please use your ExchangeDefender password, not your OWA password or anything you may have reset it to via OWA. If you need to change the password please do so at https://admin.exchangedefender.com
Have a great weekend and we hope you love Exchange 2010!
We are in the process of cutting over mail flow from our LiveArchive 2 (Exchange 2007) to our LiveArchive 3 (Exchange 2010) cluster. During this transition mail may be delivered to either location, however, users will be able to access their LiveArchive 3 account by 2AM Eastern.
Over the course of the next month mail is scheduled to be imported from the previous LiveArchive cluster.
We’re currently addressing a large scale DDoS attack on our outbound network; We’re slowly clearing out the queues from the attack and the latency has been reduced down to normal levels.