Sun Mon Tue Wed Thu
ExchangeDefender Inbound
ExchangeDefender Outbound
ExchangeDefender Apps Web
Exchange 2010 Hosting
Exchange 2007 Hosting
Offsite Backups
Web Hosting
Blackberry Enterprise Server
SharePoint 2007
SharePoint 2010
Exchange 2013 Hosting
SharePoint 2013
Virtual Servers
Network Intermittence – ExchangeDefender Network Operations

January 31, 2015

Network Intermittence

Filed under: Uncategorized — admin @ 8:32 am

Friday January 30, 2015 we experienced a network issue that affected all of our services intermittently. The problem was initially detected around 11:35 by our monitorin and then reported and confirmed around 11:40 in the morning. At this juncture we continued our network tests to find the culprit. The issue was a bit tough to track down as we were able to get the of majority of mtr tests through and into our network, came back clean. And we only had a single customer being affected. Eventually, we were able to track it down to a specific level3 hop outside our network and datacenter. We continued to work alongside our ISP and their providers to locate and resolve their issue.

We completed internal tests of all our network hardware and routing with no issues being detected. The connectivity stabilized for about an hour and half stretch as the ISP investigation continued and some of latency and intermittence returned and this time it intensified. We were notified that other customers of the isp were being affected as well. As it appeared a DDOS attack vector revealed itself in an amplified manner enough to be detected and nailed down. We started a process of null routing some items that were necessary but redundant, once that was done the attack abaited.

Always remember that any issues are always listed in the noc, and if its not because the noc (website) is affected it is directly tweeted at twitter.com/xdnoc. As we want to make sure the information is accessible and not try to update information on a site that may not be accessible.

Also for folks who want emails about email being down, both the noc and twitter support rss/notifications you can set up on your clients.

No Comments

No comments yet.

RSS feed for comments on this post.

Sorry, the comment form is closed at this time.


Powered by WordPress