ExchangeDefender Network Operations

Update 11:01 AM Eastern: We are in the process of moving the ROCKERDUCK CAS & HUB servers.

Update 10:13 AM Eastern: We are beginning the process of moving ROCKERDUCK servers to the new cabinet in our cage. Service is not expected to be affected at the moment.

During the weekend of March 23rd – March 24th we will be performing the following maintenance and upgrades to servers across multiple platforms:

Friday March 23rd 2012 22:00 – Saturday March 24th 06:00 Eastern:

NETWORK/ALL SERVICES: All services in our main cage in Dallas will undergo a quick network reconfiguration which will momentarily impact service availability for the following services
•    Sharepoint 2007 and 2010 (Excluding Rockerduck)
•    Exchange 2007 (Scrooge, Huey, Dewey)
•    Offsite backup
•    ExchangeDefender inbound nodes
•    ExchangeDefender apps (Encryption, Webfile Share)
•    OwnWebNow Support Portal
•    LiveArchive

The network reconfiguration consists of replacing network switches and rerouting network uplink distribution.

ROCKERDUCK: All servers to be relocated to new cabinet in Dallas. Service will not be interrupted during server moves.

Saturday March 24th 13:00 Eastern
LOUIE: LOUIEMBOX3 will be placed into production and pre-process steps to phase out LOUIEMBOX1 will begin. Service will not be interrupted during the turn up of LOUIEMBOX3


During the first half of March we will be performing upgrades to the LOUIE network which include adding mailbox servers, phasing out older servers, upgrading Exchange to SP2, and most importantly, DAG redesign.

On the first week (March 5th-9th 2012) we will add two new mailbox servers for LOUIE (one intended to phase out LOUIEMBOX1).

On the second week (March 12th – 16th 2012) we will create a new DAG for LOUIE and add two new mailbox databases into the DAG. Throughout the week users hosted on LOUIEMBOX1 will be moved to the new databases in the DAG. Finally once all users are moved from LOUIEMBOX1 we will begin replicating public folder content to the new mailbox servers.

All changes are intended to be transparent to users and should not interrupt service access.

Tonight beginning at 10:30 PM Eastern we will be performing the following maintenance

  • Rockerduck Load Balancer
    • Increasing physical resources. Restart required.
      • Clients will be disconnected from their mailboxes for up to 5 minutes.
  • LiveArchive: Los Angeles SP2
    • Installing Service Pack 2 on Exchange 2010
      • Client access will not be affected as we are currently running LiveArchive out of Dallas.

This weekend (02/24/12 – 02/25/12 19:00 Eastern [00:00 GMT]) we will be performing SP2 upgrades to the Europe Exchange 2010 Cluster: Della. Upgrade to Exchange 2010 SP2 will be performed on all passive nodes in Della. Upon successful upgrade clients will be moved from the active server to a passive node. . This upgrade is not expected to impact customer access however, there will be critical changes prior to the upgrade.


· New load balancer will be activated across the passive nodes.

· IP address for will be modified to the new load balancer (Expected to be

On Friday evening users on the active node will be moved to the passive nodes. The switch over from active to passive should be transparent to users.

Unfortunately BES services may be interrupted as BES does not detect and handle upgrades seamlessly. If BES service is interrupted we will work on restoring service after SP2 has been successfully applied.

Update 3:05 AM: Exchange 2010 SP2 has successfully been applied to DELLA.

This Friday (02/18/12) beginning at 07:00 Eastern (-5 GMT) [Saturday, 23:00 NSW] we will be upgrading the Exchange 2010 cluster Matilda to SP2. Clients should expect to see minimal downtime as services are restarted. Maintenance is expected to last one hour.

Update 2/17/12 6:50 AM Eastern: We are preparing the server for installation of SP2. We are expecting to begin the installation around 7:30 AM.

Update 2/17/12 7:30 AM Eastern: We are beginning the installation of SP2 on matilda.

Update 2/18/12 8:55 AM Eastern: SP2 installation has completed. We are testing services to verify a successful installation.

Update 9:07 AM Eastern: The installation was successfully verified.

During the week of Feb 4 –  Feb 11th 2012 the ExchangeDefender staff performed various upgrades to our ExchangeDefender service and to our Hosted Exchange network for ROCKERDUCK. The upgrades included additional nodes for ExchangeDefender, installation of additional power and a replacement VPN router for ROCKERDUCK internal communication (active directory, exchange services, etc). 

Unfortunately during the upgrade process we encountered various issues in which some cases affected about half the clients on ROCKERDUCK.

On February  9th  RDMBOX2 lost heartbeat communication with the DAG which prompted the passive mailbox database copies on ROCKERDUCK:LA to actively mount the database. Failover to Los Angeles kept client mailbox access available but prompted issues with Active Sync users with redirection requests and with mail delivery. The issue was completely resolved by 2PM Eastern and only affected users on RDDB8 and RDDB13.

On February 10th and 11th  Rockerduck EDGE servers experienced issues with inbound and outbound mail flow. The issue was quickly identified as an issue with a corrupted Exchange 2010 SP2 install on the EDGE servers. After reapplying the update mail flow was resumed and service was fully restored.

Although we made a lot of headway on our maintenance schedule this weekend there were a few tasks that were unable to be completed and have been rescheduled for tonight and next weekend.

Tonight we will replace the VPN device for ROCKERDUCK internal communication.  The VPN device upgrade will bring a more responsive experience for users in accessing their mailboxes.

Next weekend we will be testing the rockerduck failover procedure. Currently we have this on schedule for Saturday at 1am Eastern.

Both maintenance tonight and next weekend (2/17 – 2/19) will not interrupt clients from accessing their mailboxes.

Our primary DC will be performing maintenance this weekend as outlined below:

EVENT ID:            COLO402112012-NM

DATE:                02/11/2012



SERVICES/EQUIPMENT:  Colo4 IP Plus/Value Network

PURPOSE OF WORK:    Hardware maintenance/code upgrade

IMPACT OF WORK:      Momentary loss of connectivity as the IP Plus primary router is rebooted to accomodate code upgrade.

Maintenance Description:

Colo4 will be performing hardware maintenance and a software image upgrade.  A reboot of the router will be required.

IP Plus/Value customers can expect a brief period of connectivity loss while the router is rebooted.

On Friday (1/27/12 11:00 PM Eastern – 1:00 AM Eastern) and Saturday (1/28/12 11:00 PM Eastern – 2:30 AM Eastern) we will be performing maintenance on Rockerduck to wrap up new additions to the mailbox server high availability which will disrupt service to small population.

During maintenance we will be moving 5 mailbox active databases across new storage arrays to improve overall performance. Databases will be moved one by one and only one database will lose service availability to clients at a time. During each database move users on the respective database will be unable to access their mailbox on Rockerduck. Since we will be making architectural changes to the active mailbox database we will be unable to activate the standby copy as the passive and active copies must reside in the same location across all nodes.

The current time estimates include a 30 minute buffer in case of unforeseen events. During maintenance users should expect to be disconnected from their mailbox, however, clients can utilize livearchive during the maintenance interval.

Update 11:15 PM: We are beginning work on RDMBOX1 and moving the path of RDDB1

We’re currently testing the Business Continuity component of ExchangeDefender Essentials. We’ve scoped the test to a couple of nodes to avoid any major inconveniences. All of the backend tests with test accounts were completed successfully, so we’re moving to the next step which is testing it versus live mailflow. What does this mean to you and your clients?

A. You may see a limited (most likely not) NDRs, please review them carefully and look for the within the content. If your clients receive any please see B.

B. We’re giving the Essentials product something to separate it from other products in its price range, business continuity, where previously there was none!

Tomorrow @ 430AM We will be relocating the sites on WWW2 (Windows web hosting) to a new physical host

This change will allow us to resolve the outstanding disk space issues with the server. The move is expected to take up to an hour to complete.

The following work will be performed in our primary datacenter in Dallas Texas this weekend.


Hardware upgrade for ROCKERDUCK mailbox servers

  • Replacing RAID controller battery RDMBOX2
  • Upgrading RAM RDMBOX1 & RDMBOX2
  • Additional mailbox server for ROCKERDUCK

During the upgrades for ROCKERDUCK service is not anticipated to be interrupted. We will switch the active role between servers during maintenance.

APC Power Unit Upgrades – 7:30 AM Saturday 12/10/11

We will be upgrading two APC units in our Dallas DC. The following services will be interrupted between 7:30 AM – 8:00 AM Eastern

  • DEWEY 2007 & 2010 Exchange
  • HUEY 2007 Exchange
  • LUDWIG 2007 Sharepoint
  • All 2007 BES servers and LOUIE BES
  • –Web
  • – Web
  • – Web

Update 11:08 PM 11/17/11

DB3 has been mounted successfully on DEWEY. We’ve switched all users back to the original DB3 off from the temporary DB. We will be seeding in data from the temporary mailboxes to the primary mailbox.

Update 11:10 AM 11/17/11

The integrity check on DB3 completed around 10:00 PM Eastern on 11/16/11. Upon completion we began the process of running isinteg before mounting the database to ensure any fixed corruption gets remapped properly in the database. The check is currently at 22% completion and is estimated to complete tonight. Upon completion we will switch all users that were on DB3 back to the live running DB3 and we will then merge mail from the tempDB3 to DB3.

Update 9:24 AM 11/07/11

The integrity check and repair on DB2 completed early Sunday morning. After completing eseutil, we ran isinteg which completed around 6PM Eastern. Once we mounted DB2 and confirmed data, we begun to seed the data from the temporary database back to the original user database. Unfortunately we’ve had some partners who’ve imported their previous cached data into their temporary mailbox instead of attaching it as an archived PST on the user computer. We understand partners wanted to restore their customer back to a normal state, but that wasn’t the intention or purpose of the temporary mailbox. The restore process now must check some mailboxes with 36k+ items in the temporary mailbox which puts an extreme delay on the restore time.

Update 9:59 AM 11/04/11

Users on DEWEY experiencing slow speeds can switch their Outlook anywhere server to for an immediate performance improvement.



Update 1:09 PM Eastern

The dial tone migration has completed and users are now able to access their mailboxes on the temporary database.

Update 12:30 PM Eastern

We will be performing a dial tone migration to DEWEYMBOX2 for users on the affected databases. A dial tone migration will allow users to reconnect to their user mailbox on DEWEYMBOX2 via Outlook, OWA and Active Sync however the mailbox will have no information other than the mail from the previous day when the outage occurred and any new live running mail.

Users will see the following prompt after restarting Outlook

If the user wants to access their new mail they’ll select “Use Temporary Mailbox”

After the databases are back online we will move users back to their original databases and then restore mail from the temporary mailbox.

If the user does not receive the dial tone prompt or they stay disconnected after restarting Outlook then open their profile settings in Outlook and select ‘Check Name’ on the user.



Original Post:

On 10/31/11 at 3:45 PM DEWEYMBOX1 suffered a major outage with the databases hosted on it. Around 3:15 PM our staff replaced a failed drive on the OS RAID for the server. As a result, the server began to rebuild the array and we seen slightly increased queue sizes in which we responded by issuing the original NOC report. Shortly after the rebuild began, the controller detected the new drive as bad and activated the global hot spare policy. Unfortunately this action is what caused the DEWEY outage.

A few months ago the Information Store logs for DEWEMYBOX1 had a drive fail in the RAID array. The RAID hot spare policy activated and automatically repaired the array.

Yesterday when the outage occurred, the global hot spare policy overrode the hot spare policy of the log drive hot spare policy and forcefully took the drive to become a spare for the DB RAID array (as this had a higher weight). Once the drive was removed from the logs array, the controller faulted and the log array went offline, causing the databases to shutdown dirty.

These series of events lead to roughly 10 uncommitted log files being lost per database. As the database knows that there are uncommitted logs, the information store wouldn’t mount any databases after we replayed the available logs. Unfortunately the only way to recover was by repairing the database.

Due to the sizes of the databases, the repair is an extremely lengthy process as each record in the database gets checked for corruption.

At this point we know of roughly 30 emails across all clients (not each) that were lost because of the automated forced removal, however these emails can be recovered from livearchive. Any mail that wasn’t committed to the transport server and delivered to deweymbox1 is still in queue and pending delivery once the databases are activated.

Throughout the week of October 31st – November 4th we will be performing maintenance on our Dallas LiveArchive network. During the maintenance cycle we will redirect all LiveArchive requests to our newest LiveArchive network in Los Angeles, CA. This means that any user who browses will be forwarded off to

To allow users access to both networks during maintenance we’ve changed the IP value for to which has a global 302 redirect to

Maintenance will include upgrades to the storage architecture, rebalancing users and implementing a new Web Services based API to provide greater control and logging during new user creation.

During maintenance users can attempt to login to their Dallas LiveArchive mailbox by browsing to it directly @ however there will be periods where mailbox databases are dismounted and data will be inaccessible.

We ask that partners keep in mind that the Los Angeles version of LiveArchive only retains 3 months of mail and was initially deployed at the end of September.

Over the weekend we performed massive updates to our network, infrastructure and even provisioned new products that will be announced in October. We wanted to give you a heads up about the work being done so if you do get complaints from end users you have some knowledge about the work being done.

ExchangeDefender Inbound Network

ExchangeDefender received a huge upgrade to the load balancers, log infrastructure and even the number of inbound servers that process all mail has been increased as we continue to grow. In the new infrastructure each data center operates independantly from one another and gives us the ability to have a full failover for all services without issues with one network causing issues with the other. This was the concern that came from our issues during August where a power failure in Dallas slowed down mail processing in Los Angeles, and I am happy to report that the new infrastructure will not only improve availability when we face issues but also improve performance when everything is working as it should.

Please note: During the rebalancing process and as the new nodes are introduced to the environment the load on the network increases which increases processing times (ie: email delays). While this is counter-intuitive please consider what is happening on the backend: all nodes need to become aware of the new nodes. The configuration load times are increased as the new nodes come online and receive the initial ExchangeDefender image and definitions. During the replication process all configurations are updated across all systems and nodes are pulled online/offline as tests are ran to make sure all nodes perform in the same way. So even though there are more processing systems, initially the load on the existing nodes is increased.

ExchangeDefender Outbound Network

We have added outbound servers in Los Angeles, providing full geographic redundancy for the outbound services as well. These will be online shortly.

ExchangeDefender LiveArchive

ExchangeDefender LiveArchive LA (LA3) is online and processing mail at this time. We will be making it available for access shortly as soon as we are happy with the performance figures on both the primary and backup systems that power ExchangeDefender LiveArchive.


We are quite pleased with the performance and the updates that were made during the weekend. While we regret that there are some issues and inconveniences temporarily such as email delays, this will improve our clients ability to communicate as the network becomes more reliable and more redundant. Keep in mind that our clients pay us to keep them and their users secure and that is the primary mission here. If we could make that happen and account for both email and client growth without massive infrastructure upgrades we would love nothing more than to do more with less – but the threats are getting more serious and more numerous and protecting our client base continues to require additional resources.

Thank you for your business and your patience and please assure your clients we are working hard to keep them safe and keep their business continuity in place.

Over the course of the next few days Own Web Now Corp will be undergoing massive upgrades to our network infrastructure across our Dallas and Los Angeles data centers. We will take every precaution to limit service interruptions and while all maintenance will be performed well outside of business hours, many users will not even notice any issues as upgrades are being made to redundant systems.

The upgrades will impact all of our global services: Web hosting, ExchangeDefender, Exchange Hosting and more. Keep in mind that these are not individual computers – work is extremely complex and equipment is very sensitive. While we have planned our course of action very carefully there is always a possibility that certain service availability will be impacted or degregated. We will work diligently to communicate those issues here so you can stay in the loop. Here is a summary of maintenance work:

Thursday, September 15th, 2011

9 PM EST – ExchangeDefender in Los Angeles will be receiving new processing clusters for mail logs, inbound and outbound mail as well as LiveArchive.

Friday, September 16th, 2011

Noon – ExchangeDefender in Los Angeles will receive a new redundant LiveArchive network to improve geographic redundancy of our business continuity systems.

Saturday, September 17, 2011

Midnight EST – Power maintenance. We will be upgrading our power feeds, PDUs and UPS infrastructure across ExchangeDefender, hosting and more. We anticipate the work to be completed by 9AM EST.

Saturday Noon EST – ExchangeDefender inbound network upgrade. We are adding 25% more capacity to compensate for the growth in subscriber base.

Sunday, September 18, 2011

3 AM – 8 AM EST – ExchangeDefender will introduce geographic redundancy to our encryption, web file sharing and inbound mail routing capabilities.

All of these enhancements have been on the drawing board for months but were obviously reprioritized after the outage our Exchange 2010 network experienced in August. We moved extremely aggressively to make sure we provide a full geographically redundant network. Yes, experiencing one outage in 10 years related to power is not bad but we expect 100% uptime and this upgrade will help assure it.

The outages in the northeast are slowing down deliveries to some customers but have been improving throughout the morning as more customers restore power. We’re increasing capacity as we speak to adjust for the increased rejection/reattempt rates.

Our Engineers have informed us that they will be performing maintenance work on our Backup73 server. They have decided to perform this maintenance during business hours as most of the scheduled jobs are set begin before or after regular business hours. We thank you for your patience with us will the maintenance is performed and apologize for any inconvenience this issue may cause.

We are currently receiving alerts from our monitoring software that tracks average mail delivery time in ExchangeDefender that nodes in Dallas are experiencing larger than normal queues.

We’ve identified the issue with our logging server that receives the SMTP transaction logs from each message as it transverses the network. We are currently working to alleviate the pressure and rebalance the message load across all nodes.

Users should expect to see up to a 15 minute delay on messages until 1:50 PM Eastern when delivery services are back to speed.

Update 1:44 PM: The logs server has been reactivated and we are watching traffic pour in. Just to note, the site has been offlined while the log server is finishing the start up check.

Update 2:00 PM: Mail delivery is catching back up, however, the mail logs tables for domains are being checked for consistency before being brought back online. We expect the site to be up within 30 minutes

Update 2:20 PM: Access to has been restored. The inbound servers are still actively processing the queue loads

Update 4:35 PM: All inbound / outbound nodes have flushed their queues and are operating at normal processing time. We will continue to work on resolving the pressure issues seen by the logs server throughout the weekend.

We’ve stopped the OBS service on backup73 as we rebalance users across the drives. Maintenance is expected to last for 45 minutes and service is expected to resume at 1:30 PM Eastern.

We will be updating MBOX1 and CAS2 on the ROCKERDUCK cluster tonight starting at 9:00 PM Eastern to Exchange 2010 SP1 RU4-v2

During the upgrade, users connected to CAS2 will automatically switch over to CAS1 as we will begin to drain the server connections around 8:45 PM Eastern.

Users with mailboxes on MBOX1 will have the passive copy of their database on MBOX2 activated around 8:50 PM Eastern.

The maintenance schedule should be transparent to users and should not interrupt or disrupt any service.


Our readiness kit contains valuable resources designed specifically to help businesses with GDPR requirements.


IoT Security Solution

Introducing our newest security solution for IoT devices. Protect and secure your IoT environment with robust built in Security.


Are you an MSP?

See why you should consider our partner program. Become a partner at no cost, with no annual commitment, cancel anytime.