Support – ExchangeDefender Blog

Most Popular Products

EMAIL SECURITY

Services that protects your mail from spam, viruses, and malware.

ARCHIVING

Secure long term message storage and ediscovery reporting.

BUSINESS CONTINUITY

Constantly archiving your sent and received mail.

trash-canI am sure that we have all heard the adage of “garbage in, garbage out” when it comes to software.  However, based on my experience, the phrase also applies to support as well. It applies in all directions and regardless of perspective. Allow me to illustrate it…

End user #1: Hey local tech support, my _________ is not working right.
Local Tech: Ok I will call our service provider.
End user waits.
Cloud Tech: How can I help you?
Local Tech: Hey, Bob’s email is not working right! We are getting killed over here!
Cloud Tech: Ok who is Bob? And what exactly is not working right?
Local Tech: Umm… I don’t know, can you hold on?
The call goes on hold and at least 10 minutes go by.

Eventually the local guy comes back and tries his best to describe an image as you would to a blind person (because phones do not have video, completely). Mind you from your perspective as an MSP, while all these shenanigans are going on your end user is STILL down.

Now this is where I start yelling at my guys. Unfortunately, technology requires data not a verbal origami that somehow turns into a flying bird from a napkin. “Get that guy off the phone and tell him to paste you the headers, get the .msg file if you can, and get a screenshot.” Show me the problem!

It is at this stage that we either have a problem we fix, or a problem you fix. But sadly, we (yes both you and us) wasted 80% of the time saying something like this:

Tech 1: What does it say?
Tech 2: It says…
Tech 1: No no that line, the next line.
Tech 2: It says…
Tech 1: You skipped a line, I need the one in between.
Tech 2: Oh that one!

Trust me I get it, I love to “talk to someone” too, every day of the week. But you know what I would really like for myself and my customer every time? I would like for my problem to be solved in a timely fashion. And this is something that happens both ways, sometimes we ask bad questions, but other times we get a ticket that is composed of a PSA ticket forward that says “My email is down”, that’s it. That’s just as bad as reply that says “That sucks”. Yet they both contain the same amount of useful information.

Carlos Lascano
VP Support Services, ExchangeDefender
carlos@ownwebnow.com
(877) 546-0316 x737

This blog post is a follow up to Hosted Exchange Outbound Suspensions posted on June 20th, 2012.

Yesterday we began to roll out a custom transport agent on our Hosted Exchange network to perform automated monitoring of sent message counts for each user. The first phase of this roll out is to collect analytics and fine tune our threshold values.

Starting Monday, September 10th 2012 all hosted exchange users will be monitored for threshold breaches of outgoing (external) messages against a “Short time” and a “Daily” threshold value.

Short period
Monitor length: 15 minutes
Value: 100 messages

Long period
Monitor length: 24 hours
Value: 500

Any user who breaches threshold during the analytics period will receive a courtesy email to warn them about their threshold breach against our current threshold values. Once the analytics period ends (Estimated for the end of September 2012) we will send warning messages once the user is near 80% of the threshold and then an automated suspension will be issued once threshold is breached.

Travis Sheldon
VP, Network Operations, ExchangeDefender
(877) 546-0316 x757
travis@ownwebnow.com

clip_image002In order to ensure on time delivery and limit the network’s exposure to items with a negative impact on the network reputation, we’ve started enforcing an additional layer of security. You may have come across a message rejection that looks like this.

“Rejected. Please contact your service provider: Your message contained a SPAM URL and was blocked by ExchangeDefender.”

ExchangeDefender scans all outbound messages for possible SPAM patterns. This particular message contained a SPAMvertised URL which is typically a compromised web site or URL used in thousands of confirmed SPAM messages by a third party authority (Realtime Blacklist). In order to keep you from blacklists and RBL systems, we automatically block messages with SPAM web addresses and return to the sender. This way your legitimate messages do not end up delayed or in junk folders with other service providers.

clip_image004ExchangeDefender outbound relays cannot deliver outbound mail with the SPAMvertised URL(s) in it until it is removed from the blacklists. In order to protect your server ExchangeDefender will automatically block these messages. You can choose not to use ExchangeDefender outbound servers but if you do so your mail server will likely end up on an RBL and then you will not be able to relay through ExchangeDefender even after you remove your server from the URIBL lists.

If you run into these rejections please give my staff the from, to, and relay IP and they will be able to help you find out the content triggering the rejection.

Carlos Lascano
VP Support Services, ExchangeDefender
carlos@ownwebnow.com
(877) 546-0316 x737

Today is a very exciting day for our Hosted Exchange team as we have finally approved the use of Address Book Policies in our provisioning process (Introduced in Exchange 2010 SP2).

mac-address-bookAddress Book Policies

Address Book Policies allow Exchange Administrators to easily assign and separate address lists for clients so that clients only see objects from their organization without using ACLs or AD splicing. Prior to the approval of ABP we’ve always used ACLs to control address list segregation in our Hosted Exchange offering. In short, when a new “company” is added to Exchange, we take their primary domain and group all users based on original domain. For the most part, this approach worked well.  However, there were certain situations that this would not fare well. For instance, if a new company was split into two sub companies and they joined hosted Exchange then each sub company would only be able to see their own users UNLESS all mailboxes were added with the same primary domain in the order.

With Address Book Policies we now have the ability to create an overall policy for the “Company” which links the appropriate address lists and offline address books to each user. This change will allow companies to utilize multiple primary domains and still see all users in the company. By default we will still create companies as “separate” organizations, but partners can now request that domains for a company be linked together.

The change to automation will be introduced to Rockerduck and LOUIE this week and monitored for a week before rolling the change to all 2010 servers.

Travis Sheldon
VP, Network Operations, ExchangeDefender
(877) 546-0316 x757
travis@ownwebnow.com

Support RefresherHere we are again, another week and another quick reminder of how to test mail flow since it appears that ever week people forget the troubleshooting steps from last week.

Guys these steps are fool proof and give the answer every single time.

http://www.exchangedefender.com/blog/2011/12/where-is-my-mail-flow/

One of the tools recommended in the guide above is the Mail Log for Service Providers. Some of you are aware that we provide you with line by SMTP logs for every email. If your client is claiming their mail is not coming in this is the place to go. You log into https://admin.exchangedefender.com with your SP ID and its right there “Mail Log”. You put the to/from for the test message you sent them and voila. You have your answer, no opening a ticket, no calling anyone, no chatting with anyone. If the message isn’t listed there, check the MX record, if it’s not there then there are only two possibilities:

You typed the info in incorrectly.
We did not receive the message.

Please keep your eye on Social Media as we have a ton of new stuff coming down the pipe and that’s the best way to keep in touch.

Carlos Lascano
VP Support Services, ExchangeDefender
carlos@ownwebnow.com
(877) 546-0316 x737

Throughout the past week users with accounts on backup90 have noticed sporadic periods where service was interrupted (events were posted on the NOC blog), restored, and would randomly go back offline after 8 hours. Since this was an event that wasn’t reproducible on demand the amount of time to diagnose the issue was slightly longer than normal. Once the issue was discovered to be a faulty disk we elected to upgrade the entire storage controller for backup90. Over the past quarter backup90 has seen a 340% growth in accounts and doesn’t appear to be slowing down – unfortunately the current controller risks slower performance to clients as growth continues in the next month.

To alleviate this issue we’ve shutdown service on backup90 to start a copy process to the new controller. After all data has been copied to the new controller we will start a random write/read seek test to ensure the stability and performance of the new controller. Unfortunately since backups are an extremely intensive IO operation, leaving Ahsay service running while trying to copy would yield extremely poor results and more than likely would require multiple attempts to copy the data – all in all, it would eventually lead to a longer downtime for clients.

We estimate that the copy process will begin tonight (3/7/12) around 5:30 PM and is expected to continue throughout Friday and Saturday. We will create a new NOC entry once the upgrade process on backup90 has begun.

Travis Sheldon
VP, Network Operations, ExchangeDefender
(877) 546-0316 x757
travis@ownwebnow.com

Anybody that works in technology is familiar with ticketing systems and while at times they can be effective, it’s merely as effective as the two people interacting with the ticket. So here are some tips that can help make your tickets “one update” answer. The most important thing you can consider and rely on when opening your ticket is to maximize the effectiveness of the information you’re providing. If you give the person on the other end all the information they need, you’re improving your chances for a speedy resolutions. “But Carlos! I do this every time!”, well if you’re not following the guidelines below you’re now.

The Ticket Title has a character limit.

That’s correct if you write your entire ticket.
See? That didn’t help me get my point across, what I wanted to say is. If you type your entire ticket into the title we only see what fits in the title.

Please don’t copy and paste your end users report to you.

When we receive a ticket that says it’s from Mary (with no address) and it says I emailed Tom (with no address) and it bounced back, we basically have zero information. While we’d love to help, we don’t know your client and to that extent their clients on first name basis.

After you submit a ticket, click on it to see what we see.

This becomes pivotal, when you’re pasting data with markup codes in it, it’s important that you see what you’re sending us. Because sometimes your ticket may have too much markup to the point that it won’t render and we’re left with no content or partial content. And knowing your luck and mine, it’ll be the information we need to research your issue.

Carlos Lascano
VP Support Services, ExchangeDefender
carlos@ownwebnow.com
(877) 546-0316 x737

This weekend (02/24/12 – 02/25/12 19:00 Eastern [00:00 GMT]) we will be performing SP2 upgrades to the Europe Exchange 2010 Cluster: Della. Upgrade to Exchange 2010 SP2 will be performed on all passive nodes in Della. Upon successful upgrade clients will be moved from the active server to a passive node. . This upgrade is not expected to impact customer access however, there will be critical changes prior to the upgrade.

02/21/2012:

· New load balancer will be activated across the passive nodes.

· IP address for cas.della.exchangedefender.com will be modified to the new load balancer (Expected to be 213.229.89.253)

On Friday evening users on the active node will be moved to the passive nodes. The switch over from active to passive should be transparent to users.

Unfortunately, BES services may be interrupted as BES does not detect and handle upgrades seamlessly. If BES service is interrupted we will work on restoring service after SP2 has been successfully applied.

Travis Sheldon
VP, Network Operations, ExchangeDefender
(877) 546-0316 x757
travis@ownwebnow.com

On February 9th – February 11th 2012 the ExchangeDefender staff performed maintenance on the Rockerduck cluster in which two separate individual ‘outages’ affected client access on the late half of the evenings and early half of the following morning on February 9th -10th and February 10th-11th .

under-construction_l9wi2On the eve of Feb. 9th 2012 (~11:30 PM Eastern) we began upgrades on a failed/failing VPN device that is used to connect ROCKERDUCK:DAL and ROCKERDUCK:LA active directory and internal communication between sites. During the upgrade we began to notice random network related events in which communication seemed saturated and sluggish and randomly affected across the entire network. After various attempts (and configurations) to bring the new VPN router online we determined that the new VPN device was occasionally malfunctioning and flooding the network with ‘dead packets’. Unfortunately the massive flood of packets from the VPN device caused the Database Availability Group (DAG) on ROCKERDUCK to lose communication between nodes and eventually lose quorum. Once quorum was lost between nodes all databases between both sites were automatically dismounted as the DAG was considered unhealthy to Exchange. For the next few hours we worked to restore service to RD clients by replacing the failed VPN routers with our backup VPNs (new vendor) and restoring communication with Los Angeles. After communication was re-established clients were able to access their mailboxes. This outage affected all clients and lasted between the hours of midnight and roughly 3:15 AM.

On the eve of Feb. 10th (~10:30 PM Eastern) we began work to finalize the VPN communication by consolidating both VPN devices in California to the one backup vendor VPN device. The reason we elected to replace the ‘working’ VPN device in California was due to the fear of the abnormal workings of the similar VPN device in Dallas. As part of our protocol to ‘down’ a data center in Exchange hosting we paused SMTP services on Rockerduck. After replacing the VPN device in California we resumed all services (including SMTP) and mail resumed normal flow. Around 5:30 AM Eastern we started to receive alerts about back pressured queues in Rockerduck which would amount to delivery delays. Upon investigation it was discovered that the issue was mail delivery between the EDGE server network and the HUB server network on RD. After two hours of investigating the issue internally (and opening a case with Microsoft) we were able to determine that our course of action would be reapplying the SP2 update to the edge networks. Once SP2 was reapplied to all EDGE nodes mail delivery returned on ROCKERDUCK by 9:15 AM Eastern.

Finally there were about 5% of users who were left in a disconnected state through Outlook but had service through OWA (and some through active sync) between Saturday and Sunday as the database their mailboxes were housed was moved to Los Angeles for the content index database in Dallas to rebuild for RDDB9. Service was restored to these users by noon Eastern.

Travis Sheldon
VP, Network Operations, ExchangeDefender
(877) 546-0316 x757
travis@ownwebnow.com

Maintenance 1This weekend a big part of our team will be doing some massive infrastructure upgrades to improve our network performance and stability in our Dallas DC. These changes should not have any effect on service if we find that the planned change may impact a service of some sort we will be sure to update the NOC blog accordingly. Remember that it’s available at http://www.exchangedefender.com/noc and it’s RSS enabled at http://www.exchangedefender.com/noc/feed/ for you to subscribe for yourself and your staff.

We will also be increasing our power capability by 20% this will help us with any required growth we may need due to business growth. We’ll also begin deploying a new 2010 cluster, but I can’t provide any additional details on that as Vlad will share that news once it’s live as far its purpose and target.

Maintenance 2The last change will impact one of our backup servers, backup90 as it will receive a storage upgrade to accommodate increasing demand for that service in the upcoming months. The service impact there should be minimal and will be blogged.

So we are basically doing a big push to create a big buffer on the availability and performance so that your teams can continue to focus on just moving the product, we will take care of the ugly part of the business for you!

Carlos Lascano
VP Support Services, ExchangeDefender
carlos@ownwebnow.com
(877) 546-0316 x737

GDPR - GET STARTED

Our readiness kit contains valuable resources designed specifically to help businesses with GDPR requirements.

DOWNLOAD OUR GDPR READINESS KIT

IoT Security Solution

Introducing our newest security solution for IoT devices. Protect and secure your IoT environment with robust built in Security.

READ MORE

Are you an MSP?

See why you should consider our partner program. Become a partner at no cost, with no annual commitment, cancel anytime.

MORE INFORMATION