NOC STATUS


ExchangeDefender provides real-time alerts and information about our routine and emergency maintenance through our NOC Blog.

Support – ExchangeDefender Blog

Most Popular Products

EMAIL SECURITY

Services that protects your mail from spam, viruses, and malware.

ARCHIVING

Secure long term message storage and ediscovery reporting.

BUSINESS CONTINUITY

Constantly archiving your sent and received mail.

Today is a very exciting day for our Hosted Exchange team as we have finally approved the use of Address Book Policies in our provisioning process (Introduced in Exchange 2010 SP2).

mac-address-bookAddress Book Policies

Address Book Policies allow Exchange Administrators to easily assign and separate address lists for clients so that clients only see objects from their organization without using ACLs or AD splicing. Prior to the approval of ABP we’ve always used ACLs to control address list segregation in our Hosted Exchange offering. In short, when a new “company” is added to Exchange, we take their primary domain and group all users based on original domain. For the most part, this approach worked well.  However, there were certain situations that this would not fare well. For instance, if a new company was split into two sub companies and they joined hosted Exchange then each sub company would only be able to see their own users UNLESS all mailboxes were added with the same primary domain in the order.

With Address Book Policies we now have the ability to create an overall policy for the “Company” which links the appropriate address lists and offline address books to each user. This change will allow companies to utilize multiple primary domains and still see all users in the company. By default we will still create companies as “separate” organizations, but partners can now request that domains for a company be linked together.

The change to automation will be introduced to Rockerduck and LOUIE this week and monitored for a week before rolling the change to all 2010 servers.

Travis Sheldon
VP, Network Operations, ExchangeDefender
(877) 546-0316 x757
travis@ownwebnow.com

Support RefresherHere we are again, another week and another quick reminder of how to test mail flow since it appears that ever week people forget the troubleshooting steps from last week.

Guys these steps are fool proof and give the answer every single time.

http://www.exchangedefender.com/blog/2011/12/where-is-my-mail-flow/

One of the tools recommended in the guide above is the Mail Log for Service Providers. Some of you are aware that we provide you with line by SMTP logs for every email. If your client is claiming their mail is not coming in this is the place to go. You log into https://admin.exchangedefender.com with your SP ID and its right there “Mail Log”. You put the to/from for the test message you sent them and voila. You have your answer, no opening a ticket, no calling anyone, no chatting with anyone. If the message isn’t listed there, check the MX record, if it’s not there then there are only two possibilities:

You typed the info in incorrectly.
We did not receive the message.

Please keep your eye on Social Media as we have a ton of new stuff coming down the pipe and that’s the best way to keep in touch.

Carlos Lascano
VP Support Services, ExchangeDefender
carlos@ownwebnow.com
(877) 546-0316 x737

Throughout the past week users with accounts on backup90 have noticed sporadic periods where service was interrupted (events were posted on the NOC blog), restored, and would randomly go back offline after 8 hours. Since this was an event that wasn’t reproducible on demand the amount of time to diagnose the issue was slightly longer than normal. Once the issue was discovered to be a faulty disk we elected to upgrade the entire storage controller for backup90. Over the past quarter backup90 has seen a 340% growth in accounts and doesn’t appear to be slowing down – unfortunately the current controller risks slower performance to clients as growth continues in the next month.

To alleviate this issue we’ve shutdown service on backup90 to start a copy process to the new controller. After all data has been copied to the new controller we will start a random write/read seek test to ensure the stability and performance of the new controller. Unfortunately since backups are an extremely intensive IO operation, leaving Ahsay service running while trying to copy would yield extremely poor results and more than likely would require multiple attempts to copy the data – all in all, it would eventually lead to a longer downtime for clients.

We estimate that the copy process will begin tonight (3/7/12) around 5:30 PM and is expected to continue throughout Friday and Saturday. We will create a new NOC entry once the upgrade process on backup90 has begun.

Travis Sheldon
VP, Network Operations, ExchangeDefender
(877) 546-0316 x757
travis@ownwebnow.com

Anybody that works in technology is familiar with ticketing systems and while at times they can be effective, it’s merely as effective as the two people interacting with the ticket. So here are some tips that can help make your tickets “one update” answer. The most important thing you can consider and rely on when opening your ticket is to maximize the effectiveness of the information you’re providing. If you give the person on the other end all the information they need, you’re improving your chances for a speedy resolutions. “But Carlos! I do this every time!”, well if you’re not following the guidelines below you’re now.

The Ticket Title has a character limit.

That’s correct if you write your entire ticket.
See? That didn’t help me get my point across, what I wanted to say is. If you type your entire ticket into the title we only see what fits in the title.

Please don’t copy and paste your end users report to you.

When we receive a ticket that says it’s from Mary (with no address) and it says I emailed Tom (with no address) and it bounced back, we basically have zero information. While we’d love to help, we don’t know your client and to that extent their clients on first name basis.

After you submit a ticket, click on it to see what we see.

This becomes pivotal, when you’re pasting data with markup codes in it, it’s important that you see what you’re sending us. Because sometimes your ticket may have too much markup to the point that it won’t render and we’re left with no content or partial content. And knowing your luck and mine, it’ll be the information we need to research your issue.

Carlos Lascano
VP Support Services, ExchangeDefender
carlos@ownwebnow.com
(877) 546-0316 x737

This weekend (02/24/12 – 02/25/12 19:00 Eastern [00:00 GMT]) we will be performing SP2 upgrades to the Europe Exchange 2010 Cluster: Della. Upgrade to Exchange 2010 SP2 will be performed on all passive nodes in Della. Upon successful upgrade clients will be moved from the active server to a passive node. . This upgrade is not expected to impact customer access however, there will be critical changes prior to the upgrade.

02/21/2012:

· New load balancer will be activated across the passive nodes.

· IP address for cas.della.exchangedefender.com will be modified to the new load balancer (Expected to be 213.229.89.253)

On Friday evening users on the active node will be moved to the passive nodes. The switch over from active to passive should be transparent to users.

Unfortunately, BES services may be interrupted as BES does not detect and handle upgrades seamlessly. If BES service is interrupted we will work on restoring service after SP2 has been successfully applied.

Travis Sheldon
VP, Network Operations, ExchangeDefender
(877) 546-0316 x757
travis@ownwebnow.com

On February 9th – February 11th 2012 the ExchangeDefender staff performed maintenance on the Rockerduck cluster in which two separate individual ‘outages’ affected client access on the late half of the evenings and early half of the following morning on February 9th -10th and February 10th-11th .

under-construction_l9wi2On the eve of Feb. 9th 2012 (~11:30 PM Eastern) we began upgrades on a failed/failing VPN device that is used to connect ROCKERDUCK:DAL and ROCKERDUCK:LA active directory and internal communication between sites. During the upgrade we began to notice random network related events in which communication seemed saturated and sluggish and randomly affected across the entire network. After various attempts (and configurations) to bring the new VPN router online we determined that the new VPN device was occasionally malfunctioning and flooding the network with ‘dead packets’. Unfortunately the massive flood of packets from the VPN device caused the Database Availability Group (DAG) on ROCKERDUCK to lose communication between nodes and eventually lose quorum. Once quorum was lost between nodes all databases between both sites were automatically dismounted as the DAG was considered unhealthy to Exchange. For the next few hours we worked to restore service to RD clients by replacing the failed VPN routers with our backup VPNs (new vendor) and restoring communication with Los Angeles. After communication was re-established clients were able to access their mailboxes. This outage affected all clients and lasted between the hours of midnight and roughly 3:15 AM.

On the eve of Feb. 10th (~10:30 PM Eastern) we began work to finalize the VPN communication by consolidating both VPN devices in California to the one backup vendor VPN device. The reason we elected to replace the ‘working’ VPN device in California was due to the fear of the abnormal workings of the similar VPN device in Dallas. As part of our protocol to ‘down’ a data center in Exchange hosting we paused SMTP services on Rockerduck. After replacing the VPN device in California we resumed all services (including SMTP) and mail resumed normal flow. Around 5:30 AM Eastern we started to receive alerts about back pressured queues in Rockerduck which would amount to delivery delays. Upon investigation it was discovered that the issue was mail delivery between the EDGE server network and the HUB server network on RD. After two hours of investigating the issue internally (and opening a case with Microsoft) we were able to determine that our course of action would be reapplying the SP2 update to the edge networks. Once SP2 was reapplied to all EDGE nodes mail delivery returned on ROCKERDUCK by 9:15 AM Eastern.

Finally there were about 5% of users who were left in a disconnected state through Outlook but had service through OWA (and some through active sync) between Saturday and Sunday as the database their mailboxes were housed was moved to Los Angeles for the content index database in Dallas to rebuild for RDDB9. Service was restored to these users by noon Eastern.

Travis Sheldon
VP, Network Operations, ExchangeDefender
(877) 546-0316 x757
travis@ownwebnow.com

Maintenance 1This weekend a big part of our team will be doing some massive infrastructure upgrades to improve our network performance and stability in our Dallas DC. These changes should not have any effect on service if we find that the planned change may impact a service of some sort we will be sure to update the NOC blog accordingly. Remember that it’s available at http://www.exchangedefender.com/noc and it’s RSS enabled at http://www.exchangedefender.com/noc/feed/ for you to subscribe for yourself and your staff.

We will also be increasing our power capability by 20% this will help us with any required growth we may need due to business growth. We’ll also begin deploying a new 2010 cluster, but I can’t provide any additional details on that as Vlad will share that news once it’s live as far its purpose and target.

Maintenance 2The last change will impact one of our backup servers, backup90 as it will receive a storage upgrade to accommodate increasing demand for that service in the upcoming months. The service impact there should be minimal and will be blogged.

So we are basically doing a big push to create a big buffer on the availability and performance so that your teams can continue to focus on just moving the product, we will take care of the ugly part of the business for you!

Carlos Lascano
VP Support Services, ExchangeDefender
carlos@ownwebnow.com
(877) 546-0316 x737

Last quarter we launched a slimmed down version of ExchangeDefender to be packaged with Exchange Essentials 2010 and as a standalone product. This product was launched to provide a similar price point and feature set as some less robust Spam & Security solutions out in the channel. However, we were never comfortable with not offering a bundled-in business continuity solution. Enter… ExchangeDefender Essentials Emergency.

Emergency is the business continuity solution that is now bundled in with ExchangeDefender Essentials. It will capture a copy of all incoming email in similar fashion, but only inbound mail with a retention policy of 5 days for all items. This email is accessible via web portal and POP3/IMAP4 (although currently we are limiting the ability on the POP/IMAP to just download messages (to avoid open relay situations). So your clients will be able to continue to do business with the ability to receive, reply to, and create new emails from their real email address during an outage. Remember, that our 7 day spooling/mail bagging system is still in place so a combination of the two should minimize your client’s inconvenience.

The web portal is available at:

https://emergency.exchangedefender.com

Credentials:

Your primary user email address with ExchangeDefender (you cannot log in with an alias address) and your current ExchangeDefender password.

 

 

Once you log in all of your email will be available, with your identity pre-configured for use. There is no additional set up required. You can start reading and firing off emails as quickly as you can type.

Setting up Outlook (remember Read/DL message access only currently)

Fill out the information as below:

 

 

Your email address and user name are the same as your primary address in ExchangeDefender. The POP3/IMAP4 server is emergency.exchangedefender.com. Everything is on the standard ports for both SSL and non.

 

 

Our CEO, Vlad Mazek, will be providing a broader overview on emergency and its feature set when we officially roll it out in the next 2 weeks.

Carlos Lascano
VP Support Services, ExchangeDefender
carlos@ownwebnow.com
(877) 546-0316 x737

Over the past couple of weeks we have been researching some reports regarding encryption not handling attachments correctly. During the process, the error that kept printing on the back end processing was ““Content-Type: application/ms-tnef; name=”winmail.dat” Content-Transfer-Encoding: base64””. If Outlook sends a message using the RTF format (which is not very common outside Outlook) for bold text and other text enhancements, it includes the formatting commands in the winmail.dat file. Receiving email clients that do not understand the code therein display it as a stale attachment. To make matters worse, Outlook may also pack other, regular file attachments in the winmail.dat file. That’s the bad news, the good news is that fix is a piece of cake.

In Outlook 2010 you go through File, then Options and check the box below:

1

In Outlook 2007 you go through Tools, then Options:

2

1. Go to the Mail Format tab.

2. Under Compose in this message format:, make sure either HTML or Plain Text is selected.

3. Click Internet Format.

4. Make sure either Convert to Plain Text format or Convert to HTML format is selected under When sending Outlook Rich Text messages to Internet recipients, use this format:

5. Ok to submit.

Carlos Lascano
VP Support Services, ExchangeDefender
carlos@ownwebnow.com
(877) 546-0316 x737

Imagine it’s 4:00 PM, you’re getting ready to close for the day and your cell phone starts ringing off the hook; it is your biggest clients’ CFO and he is very upset. The incoming caller complains that “email is slow” and “it is taking forever to do xyz”, but any attempt to get more information is greeted with either hostility or an abrupt “I don’t know”.

Does that sound familiar?

Almost every single request that has very little detail from the client in terms of what is “slow” magically gets fixed and then the client is convinced that the issue is Exchange…how do you fight back? How do you know for sure that a server / network out of your control is performing up to par. How do you know that your hosting vendor is keeping redundancy healthy and performing backups? For the most part you can’t… or can you?

One of the most common inquiries that we receive from partners is “Is XYZ server experiencing delays today?” after the partner gets alerted by their client that things seem to be “slow”. Our staff then tries to qualify the phrase “slow”…is it email? Is it Outlook response? How about OWA? After we have an idea of what the client is reporting as slow then we have to dig through logs and statistics files for performance data to provide back to the client…this process takes forever.

What if we could automate it? What if we could provide partners with an “at a glance” view of the server’s health and their client’s statistics? What if we could provide you with a list of available backup dates so you can choose what date you’d like to restore from? What if we could provide you the number of messages in queue for Exchange or overall latency for clients and response times?

What if we could provide you with up to the minute stats of the CAS server your user is on, the CPU percentage used by the client, the amount of latency experienced by the client.

We can, and we will…

As far as I know this level of information and statistics has never been provided by a service provider before…

Below is the draft version of the User Monitor that will be adding to our Staff control panel and will more than likely find it’s way into service manager.

 

xyz

 

Travis Sheldon
VP, Network Operations, ExchangeDefender
(877) 546-0316 x757
travis@ownwebnow.com

DEMO

Schedule a live demo to ask questions and more!

REQUEST A DEMO

QUOTE

Learn more about our plans and pricing.

REQUEST A QUOTE

CONTACT

Have questions for us? We are here to help!

CONTACT US