Tue Wed Thu Fri Sat
ExchangeDefender Inbound
ExchangeDefender Outbound
ExchangeDefender Apps Web
Exchange 2010 Hosting
Exchange 2007 Hosting
Offsite Backups
Web Hosting
Blackberry Enterprise Server
SharePoint 2007
SharePoint 2010
Exchange 2013 Hosting
SharePoint 2013
Virtual Servers
ExchangeDefender Network Operations

January 3, 2013

Phones

Filed under: Uncategorized — carlos @ 1:10 pm

Our sip provider is currently having issues. Remember our portal is SLA backed 24/7. If you need help open a ticket or a LiveChat within the portal for support issues.

Update: Our provider has resolved their issues.


January 1, 2013

LOUIE DB4 issue

Filed under: Uncategorized — admin @ 7:51 am

We’re currently working an issue with DB4 which is similar to the issues faced previously with DB3_1 .  The issue is related to a storage controller issue and we are currently running an integrity check/repair on the database

We sincerely apologize for the length of this issue. We understand this is extremely frustrating, as it is extremely frustrating for us to provide this level of service. This is currently not only our top priority but only priority and our top Exchange engineers are on this task and this task alone until resolved.

Update 6:00 PM Eastern: We are still performing work on DB4 for the affected mailboxes. We would like to extend the option for users to utilize “recovery mode” / dial tone. Recovery mode is a temporary new “blank” mailbox where users can work with their live mail (after the outage occurred) as they normally would. Since the recovery mailbox is a “new” mailbox, there wont be contacts, calendars, etc. If you have active sync users or iPhone users then you should have then set their Email, Calendar, and Contact sync to a manual sync as the phone will sync with the empty mailbox.

When a mailbox is manually switched to a temp database Outlook will now prompt the user to either use their Temporary data (new data) or open their previous cached mailbox data

dialtone prompt

Outlook “Recovery Mode” prompt is a feature set introduced in Outlook 2007 to safely handle mailbox or server side issues that lead to mailbox configuration changes. After the server goes through a dial tone migration, Outlook will notice the content change in the mailbox and locks the old OST cache file from downloading any new mail. Once Outlook goes into Recovery Mode, the user will receive a prompt upon launching an Outlook profile asking if the user wants to use “Old data” or “Temp data” – in simple terms, Old data refers to the OST cache file on the client machine, and temp data provides online access to the mailbox to view new items that arrive – while in temp data, Outlook will run in Online mode and not Cached mode, so the overall experience will be slower for users who are used to the speeds from cached mode.

If a partner would like to utilize a recovery mailbox for mailboxes in their domain then we would need a new support request opened with following subject:
Dialtone Request: domain.com
Where domain.com is the client domain. By opening the request as a new request this will allow us to ensure all requests are properly completed and documented.

January 2 2013

Update 4:30 PM Eastern: The repair / integrity check is near 55% completed on the last step. We anticipate being able to mount the database by 8PM Eastern tonight which we will then deliver all queued mail to the mailboxes. Once all queues are flushed we will then swap users who are in dial tone mode back to their home database and subsequently merge the new and existing data. We want to thank everyone again for their patience as we’ve had a positive response to the dial tone mailbox setups.

Update 5:45 PM Eastern: We’ve successfully mounted DB4 on LOUIE. We’ll be resuming mail delivery shortly.


December 30, 2012

LOUIE DB Issue

Filed under: Uncategorized — admin @ 6:51 pm

Users on LOUIE DB3_1 are currently unable to access their mailboxes due to a hardware level issue with the storage device. The database was a temporary database used as a temporary holding spot for mailboxes that were moving from DB3 which was being phased out on LOUIE. Unfortunately since this was a temporary database there was no anticipated need to make multiple database copies available as the extra over head could cause production performance issues. To rectify the issue we’ve been running a hardware RAID level resync to ensure all data is in the best state possible. We do anticipate the issue being resolved before the start of business on Monday.

As a reminder, partners are able to use LiveArchive (https://livearchive.exchangedefender.com) to access their live running mail during the outage

Update 9:07 AM: We are currently running an eseutil integrity check on the log files for DB3_1. This check will ensure that all the data that is present on the database is correct and true, as well as any uncommitted data can be finally committed. This process is expected to take up to two hours to complete. If the replay and check complete without error we will then be able to safely mount the database. All mail that hasn’t been delivered is still queued and waiting for delivery.

Update 5:30 PM: The database was successfully mounted after the integrity check completed. In order to prevent this from reoccurring we are adding a database copy to this database. Please keep in mind that this database was a transitional database, used temporarily to move users from DB3 which was in the process of being phased out. We’re extremely sorry for the inconvenience this has caused and we will include transitional databases in DAG setups going forward.


October 25, 2012

Australia 2007 Outage

Filed under: Uncategorized — Tags: — admin @ 8:53 am

The servers in the Australia network (2007 Exchange) has suffered a major failure which has ultimately led to the decision to decommission it as a restoration will not bring service online seamlessly (without recovery and resetup by clients)

Unfortunately the recovery from Australia is not something we believe will be completed in the next 48 hours at a minimum. To accommodate this and provide users with an immediate solution to continue working we’ve proactively recreated all users on Australia (2007) on to Matilda (2010). Matilda has redundancy and we’ve already sent over an additional server to add to the Matilda cluster.

The new server address is cas.matilda.exchangedefender.com and the autodiscover record should be setup as autodiscover.matilda.exchangedefender.com

We’ve changed the target delivery location in ExchangeDefender to matilda. You can also login to livearchive to work with mail while the change over to matilda occurs.

Partners who have clients with cached outlook profiles can open the current profile (On Australia) and then export the cached data to .pst files. Any users who do not have cached data or any public folder data will not have an ETA on restoration for at least 48 hours (until we can process the drive’s data)

Update 10/25/12 9:45 PM Eastern: We are in the process of copying over items from livearchive to matilda mailboxes to import the past three days of mail from livearchive. We anticipate the process being completed in under 4 hours.


September 16, 2012

Matilda outage

Filed under: Uncategorized — admin @ 7:58 pm

This morning around 3:30 AM Eastern we received alert that the hosted Exchange 2010 network in Australia (MATILDA) went offline. Upon login via KVM we noticed the server was repeatedly rebooting after faulting to a blue screen, however no diagnostic information was provided. It was soon determined that the operating system needed to be repaired which completed around 5:20 AM Eastern. Once we were able to successfully boot into Windows we performed a database integrity check to ensure no actual data was corrupted or lost which completed around 7:45 AM Eastern and service was restored.

Once service was restored we looked into the server logs which provided no information or logged entries regarding the server fault, however, a memory dump was created. Unfortunately the memory dump wasn’t of much help and the issue appeared to possibly be hardware related. We temporarily switched service for Matilda over to the backup node as we tested the hardware components. We were able to determine that the issue was related to a power supply that randomly dropped output voltage during high load. After replacing the power supply we ruled that the server faulted while running a backup. With regard to the OS repair we can only deduce that a system file was corrupted from the fault and needed to be replaced.


August 27, 2012

Linux Web Hosting PHP Upgrade

Filed under: Uncategorized — carlos @ 12:34 pm

Over the weekend our Linux web hosting infrastructure was upgraded to PHP 5.3. The previous release of PHP 5.1.6 was getting out of date for a lot of social applications that dominate todays deployment base. You will now be able to run the latest Wordpress, Joomla and other CMS and shopping cart sites without a problem.

If you have any legacy PHP applications that were written in PHP4 and early PHP5 days, you may notice that some functionality does not work. Consult your software vendor for an update that functions with PHP 5.3 and above.

If your application is not supported on the new platform of PHP and you do not have the resources to make an immediate switch to an alternative, we do have a www2legacy web infrastructure in place that will support older applications. We will make the accommodations and the switch to the legacy platform for a small one-time fee but do keep in mind that the legacy platform will be discontinued in March of 2013. We have provided more than a 1 year announcement of our intention to upgrade our network infrastructure to the latest PHP so if this caught you by surprise we will still do what we can to make sure you have a smooth transition.

For a list of features in PHP 5.3.x and old deprecated functions:

http://php.net/manual/en/migration53.deprecated.php

This is probably an excellent place to start your troubleshooting on outdated code. If you decide you’d like to stay with the legacy code please open a support request at https://support.ownwebnow.com if you’d like further information.


August 22, 2012

ROCKERDUCK Maintenance

Filed under: Uncategorized — admin @ 9:49 am

This Friday night we will be performing maintenance to the ROCKERDUCK cluster that may interrupt client connections for up to 10 minutes. The purpose of this upgrade is to provide a more intelligent network routing for our mailbox database replication service. We will work our hardest to make the required changes without interrupting client connections.

We anticipate starting the work around 9PM on Friday August 24th.


July 19, 2012

ROCKERDUCK Outage RDDB1

Filed under: Uncategorized — admin @ 11:33 am

Users on RDDB1 are unable to access their mailbox data as the mailbox database is currently offline. We are performing surface check on the database as the database health is indicating minor corruption. We are unable to activate the passive copies of the database without the surface check being completed as we are unsure when the corruption occurred and could have been copied to the passive copies. We anticipate that the database will be offline until at least 3PM Eastern. Only 9% of user base is affected and we are working diligently to restore access to customer mailboxes on RDDB1

Update 3:06PM Eastern: We are running two processes to try and restore access as soon as possible: repairing the live database and running a restore from the last backup (7/16/12) and then replaying the committed data to the restored backed up database to bring it up to date. We will go with whatever process completes first as both will bring the database back online in a  healthy state. The restored backed up data base is copying log files and will complete around 5PM. The live data base repair is at 50% on step 1/5 and its ETA is unknown

Update 4:13 PM Eastern: We’ve successfully mounted RDDB1 on ROCKERDUCK and confirmed user access.

Unfortunately this was not something that could have been avoided by any redundancy available as this was a software level corruption which isn’t protected by a DAG. Essentially, the DAGs prevent against issues with unavailability between mailbox nodes or networking issues, but when a corrupted log file gets committed, it then gets distributed to the passive nodes. Since we couldn’t confirm if the corrupted file was or wasn’t distributed to specific nodes when caught we felt it wasn’t safe to remount the databases with the lingering possibility of encountering a more serious issue in the near future.

We are planning on introducing a lagged mailbox database copy which will ‘lag’ when it commits log files to the passive copy. By implementing a lagged copy  if we ever experience corruption again we can restore instantly to the previous day as the lagged copy wouldn’t have copied/committed the corrupted data


May 29, 2012

Maintenance May 29th

Filed under: Exchange Hosting — travis @ 4:31 pm

The work described below is schedule to begin at 9:00 PM Eastern May 29th 2012

  • LOUIE – LOUIEMBOX1 & LOUIEMBOX2 – Update Network Driver

Will cause brief interruption, 15-30 seconds while the driver updates

  • LOUIE – update exchange to service pack 2

Will not cause interruption to clients

  •  ROCKERDUCK – Reseeding databases between RDMBOX1 and RDMBOX2 for fail over

Will not cause interruption, but OWA users may see slight delays in accessing content (including public folders) since the replication is going to use the MAPI NICs instead of the replication NIC. This will only be during night as to not flood the network during the day

  • ROCKERDUCK – Redistributing disk layout on RDMBOX3

RDMBOX3 is one of the additional fail over clusters and does not actively hold any mailbox databases


May 14, 2012

ROCKERDUCK Outage:

Filed under: Uncategorized — travis @ 1:41 pm

 

Today (5/14/12) there was an outage with ROCKERDUCK between 11:50 AM – 12:20 PM (Eastern) that affected client access to mailboxes. In short the outage was caused by our Los Angeles site experiencing Active Directory communication issues with Dallas, eventually taking LA offline. Normally, this event wouldn’t cause any issue as Dallas would be able to maintain quorum as Dallas holds the majority vote in the DAG quorum. Unfortunately the tie breaking voter (MBOX3) was offline for maintenance (As it does not actively host DBs – it’s the internal failover server for Dallas), which before the communication issue between sites left the vote as 4/5 online. Once LA experienced connectivity issues the vote dropped to 2/5 online and was forced offline. Our first alerts from monitoring started to come in at 11:52AM and we were able to respond by 11:55AM to the issue and post an advisory. Around 12:10 PM Eastern we were able to reestablish quorum in Dallas and bring the cluster online. Around 12:15 PM MBOX1 mounted all active databases and by 12:20 PM MBOX2 mounted all databases. Since this event was a communication issue between sites, the passive site was marked as Blocked (Unable to automatically mount databases for fail over) as network interruptions were detected.


April 27, 2012

International numbers for Support

Filed under: Uncategorized — travis @ 3:29 pm

This weekend our international SIP trunk provider will be doing weekend long maintenance on the infrastructure that hosts our accounts. During the period of 4/27/12 – 4/30/12 our international partners will have to dial our main number, 877-546-0316


April 23, 2012

LOUIE Public Folder Changes

Filed under: Exchange Hosting — travis @ 2:55 pm

Update 5/2/2012: The reinstallation has completed and service is resumed normal operation.

Update 5/1/2012: We are in the process of reloading the operating system on LOUIEMBOX1. As of now the only affected service is mail enabled public folders, however, all mail will be queued until the reinstallation of Exchange has completed. We anticipate being done with all work by midnight.

Update : This has been scheduled for Friday at 3:00 PM Eastern

In preparation of the reload of LOUIEMBOX1 we’ve moved all mailboxes hosted on to other mailbox servers in the cluster. Last month we created a replication of all Public Folder content to all mailbox servers in LOUIE from LOUIEMBOX1. The final step is to change all ExchangeDefender delivery points for LOUIE customers away from LOUIEMBOX1. This step should fall in line with our seamless upgrades and should not be noticed by the client. Unfortunately in past experiences some public folders would not receive messages from the outside after a replica is taken offline. We believe this was a previous bug with Exchange and has since been resolved, however, we’d like to make all possibilities known ahead of time.

Due to the number of public folders and public folder content, we will be unable to validate mail delivery across all mail enabled public folders during the reload.

If any clients experience mail delivery delays to their public folders, please open a support request with the email address of the mail public folder and our support team will immediately look into the issue.


March 27, 2012

LOUIE DAG and Public Folder Upgrades

Filed under: Uncategorized — travis @ 10:23 am

Update: The below work has been partially completed. We’ve changed the migration strategy to use public folder replication instead of an in place reload. This decision will extend the maintenance interval into next weekend but will provide no downtime for mailboxes or public folders

Over the weekend of March 30th- April 1st we will be performing upgrades to the LOUIE DAG (Mailboxes) and Public Folder databases. The changes that will begin during this maintenance cycle will improve the native automatic redundancy for LOUIE mailboxes and bring an overall greater experience for users. Due to the number of users and the size of data on LOUIE, the only way to complete the migration with the overall least amount of service interruption is to dismount the public folder database on Friday evening, and leaving it dismounted until Saturday afternoon.

Taking the public folder database offline will prevent ALL access to public folder data housed on LOUIEMBOX1, including mail enabled public folders between Friday evening until Saturday afternoon.

All public folders are replicated between multiple databases, however, unlike DAG protection, Public Folder databases work off a referral based system for locating copies and is highly unreliable for automated continuity.

Any users/partners who are concerned about the availability of their public folder data can open a support ticket and request that our team ensures their public folder data has an active, local replica.

We do not anticipate any interruption in the service of actual mail delivery or mailbox access on LOUIE.


March 12, 2012

Maintenance March 23 and 24th 2012

Filed under: Uncategorized — travis @ 3:20 pm

Update 11:01 AM Eastern: We are in the process of moving the ROCKERDUCK CAS & HUB servers.

Update 10:13 AM Eastern: We are beginning the process of moving ROCKERDUCK servers to the new cabinet in our cage. Service is not expected to be affected at the moment.

During the weekend of March 23rd – March 24th we will be performing the following maintenance and upgrades to servers across multiple platforms:

Friday March 23rd 2012 22:00 – Saturday March 24th 06:00 Eastern:

NETWORK/ALL SERVICES: All services in our main cage in Dallas will undergo a quick network reconfiguration which will momentarily impact service availability for the following services
•    Sharepoint 2007 and 2010 (Excluding Rockerduck)
•    Exchange 2007 (Scrooge, Huey, Dewey)
•    Offsite backup
•    ExchangeDefender inbound nodes
•    ExchangeDefender apps (Encryption, Webfile Share)
•    OwnWebNow Support Portal
•    LiveArchive

The network reconfiguration consists of replacing network switches and rerouting network uplink distribution.

ROCKERDUCK: All servers to be relocated to new cabinet in Dallas. Service will not be interrupted during server moves.

Saturday March 24th 13:00 Eastern
LOUIE: LOUIEMBOX3 will be placed into production and pre-process steps to phase out LOUIEMBOX1 will begin. Service will not be interrupted during the turn up of LOUIEMBOX3


February 28, 2012

LOUIE Upgrades

Filed under: Exchange Hosting — travis @ 5:16 pm

 

During the first half of March we will be performing upgrades to the LOUIE network which include adding mailbox servers, phasing out older servers, upgrading Exchange to SP2, and most importantly, DAG redesign.

On the first week (March 5th-9th 2012) we will add two new mailbox servers for LOUIE (one intended to phase out LOUIEMBOX1).

On the second week (March 12th – 16th 2012) we will create a new DAG for LOUIE and add two new mailbox databases into the DAG. Throughout the week users hosted on LOUIEMBOX1 will be moved to the new databases in the DAG. Finally once all users are moved from LOUIEMBOX1 we will begin replicating public folder content to the new mailbox servers.

All changes are intended to be transparent to users and should not interrupt service access.


February 23, 2012

Maintenance: Rockerduck and Livearchive:Los Angeles

Filed under: Exchange Hosting — travis @ 9:52 am

Tonight beginning at 10:30 PM Eastern we will be performing the following maintenance

  • Rockerduck Load Balancer
    • Increasing physical resources. Restart required.
      • Clients will be disconnected from their mailboxes for up to 5 minutes.
  • LiveArchive: Los Angeles SP2
    • Installing Service Pack 2 on Exchange 2010
      • Client access will not be affected as we are currently running LiveArchive out of Dallas.

February 20, 2012

Della Upgrade SP2 & DAG Activation

Filed under: Exchange Hosting — travis @ 10:30 am

This weekend (02/24/12 – 02/25/12 19:00 Eastern [00:00 GMT]) we will be performing SP2 upgrades to the Europe Exchange 2010 Cluster: Della. Upgrade to Exchange 2010 SP2 will be performed on all passive nodes in Della. Upon successful upgrade clients will be moved from the active server to a passive node. . This upgrade is not expected to impact customer access however, there will be critical changes prior to the upgrade.

02/21/2012:

· New load balancer will be activated across the passive nodes.

· IP address for cas.della.exchangedefender.com will be modified to the new load balancer (Expected to be 213.229.89.253)

On Friday evening users on the active node will be moved to the passive nodes. The switch over from active to passive should be transparent to users.

Unfortunately BES services may be interrupted as BES does not detect and handle upgrades seamlessly. If BES service is interrupted we will work on restoring service after SP2 has been successfully applied.

Update 3:05 AM: Exchange 2010 SP2 has successfully been applied to DELLA.


February 15, 2012

Matilda Exchange 2010 SP2 Install

Filed under: Exchange Hosting — travis @ 9:09 am

This Friday (02/18/12) beginning at 07:00 Eastern (-5 GMT) [Saturday, 23:00 NSW] we will be upgrading the Exchange 2010 cluster Matilda to SP2. Clients should expect to see minimal downtime as services are restarted. Maintenance is expected to last one hour.

Update 2/17/12 6:50 AM Eastern: We are preparing the server for installation of SP2. We are expecting to begin the installation around 7:30 AM.

Update 2/17/12 7:30 AM Eastern: We are beginning the installation of SP2 on matilda.

Update 2/18/12 8:55 AM Eastern: SP2 installation has completed. We are testing services to verify a successful installation.

Update 9:07 AM Eastern: The installation was successfully verified.


February 11, 2012

Maintenance Feb 4–Feb 11 2012

Filed under: Uncategorized — travis @ 4:31 pm

During the week of Feb 4 -  Feb 11th 2012 the ExchangeDefender staff performed various upgrades to our ExchangeDefender service and to our Hosted Exchange network for ROCKERDUCK. The upgrades included additional nodes for ExchangeDefender, installation of additional power and a replacement VPN router for ROCKERDUCK internal communication (active directory, exchange services, etc). 

Unfortunately during the upgrade process we encountered various issues in which some cases affected about half the clients on ROCKERDUCK.

On February  9th  RDMBOX2 lost heartbeat communication with the DAG which prompted the passive mailbox database copies on ROCKERDUCK:LA to actively mount the database. Failover to Los Angeles kept client mailbox access available but prompted issues with Active Sync users with redirection requests and with mail delivery. The issue was completely resolved by 2PM Eastern and only affected users on RDDB8 and RDDB13.

On February 10th and 11th  Rockerduck EDGE servers experienced issues with inbound and outbound mail flow. The issue was quickly identified as an issue with a corrupted Exchange 2010 SP2 install on the EDGE servers. After reapplying the update mail flow was resumed and service was fully restored.

Although we made a lot of headway on our maintenance schedule this weekend there were a few tasks that were unable to be completed and have been rescheduled for tonight and next weekend.

Tonight we will replace the VPN device for ROCKERDUCK internal communication.  The VPN device upgrade will bring a more responsive experience for users in accessing their mailboxes.

Next weekend we will be testing the rockerduck failover procedure. Currently we have this on schedule for Saturday at 1am Eastern.

Both maintenance tonight and next weekend (2/17 – 2/19) will not interrupt clients from accessing their mailboxes.


February 8, 2012

Dallas DC Maintenance

Filed under: Data Center Ops — carlos @ 1:58 pm

Our primary DC will be performing maintenance this weekend as outlined below:

EVENT ID:            COLO402112012-NM

DATE:                02/11/2012

WINDOW START TIME:  1:00 AM CST

ESTIMATED END TIME:  4:00 AM CST

SERVICES/EQUIPMENT:  Colo4 IP Plus/Value Network

PURPOSE OF WORK:    Hardware maintenance/code upgrade

IMPACT OF WORK:      Momentary loss of connectivity as the IP Plus primary router is rebooted to accomodate code upgrade.

Maintenance Description:

Colo4 will be performing hardware maintenance and a software image upgrade.  A reboot of the router will be required.

IP Plus/Value customers can expect a brief period of connectivity loss while the router is rebooted.


« Newer PostsOlder Posts »

Powered by WordPress