Alerts – ExchangeDefender Blog

September 8, 2017

Schedule and SLA during Hurricane Irma

Filed under: Alerts,Announcements — vlad @ 11:02 pm

As many of you know, ExchangeDefender HQ is in Orlando, FL and as of noon today that office is closed until further notice. As of the latest hurricane projection, Orlando is likely facing the worst case scenario of being in the NE quadrant very close to the eye of the storm that will roll over the county next to us. Best case scenario, we will be returning to our Orlando offices on Tuesday, Septembrer 12th. Our capacity at that point is obviously questionable because roads, infrastructure and staff may be impacted.

ExchangeDefender is a global business and we have made several contingency plans and do not expect any interruption to our services. Although the majority of our staff is in Florida, we don’t have a single piece of computer/network infrastructure here so the show will go on as usual. The SLA for normal support cases will still be within 4 hours, we will still answer support tickets and phone calls. You should not be able to tell the difference for ordinary issues.

Where we expect to have the most significant challenge is in the areas of projects and special “courtesy” assistance that we typically go well above and beyond the call of duty – things like exporting mail, adding accounts by hand, reaching out directly to clients on your behalf, sales engineering and proposal writing, legal advisory services, compliance officer training, marketing collateral design and event support services are highly dependant on manpower which is something we may be lacking. We will still not say no but we do hope you’ll extend us some extra patience as we go on.

I feel it is my responsibility to be honest and up front about our priority process as we go through next week. This is the typical disaster priority schedule our entire staff is briefed and drilled on but with obvious bias towards Florida and the hurricane.

1. Most important priority will be given to the US and Florida government organizations that are coordinating efforts in the disaster zone. Staff is directed to drop literally anything else they may be doing to assist them.

2. Urgent tickets, system outages and performance issues will be treated as a secondary priority.

3. Ordinary support tickets will still be resolved within the SLA 4 hour window.

We understand that any issue impacting our service is a critical issue to those unable to go about their work because of something in our realm of responsibility. We have made numerous contingencies, failover and backup plans and expect to execute the plan accordingly. We do not employ warm bodies and people that do not care – everyone that is a part of a team understads service delivery is a team effort and wants to see everything continue to work for you as flawlessly and transparently as possible. Hopefully this blog post gives you additional insight as to how we will be handling that with regards to Irma. Personally, I understand that many of our partners in Texas are reeling, that many of our partners and clients in Florida/Alabama/Georgia/Carolinas are scared and we have been working unprecedented hours to make sure everyone has their contingency and failover plans implemented, tested and documented. To everyone in the way, we’re in this together. There is no hurricane party or a 5 day weekend for us, it’s all hands on deck with that deck now spanning 4 states. We’ll get through this. We may move a bit slower than usual, but we will get through this.

Sincerely,
Vlad Mazek, CEO.


July 18, 2013

ExchangeDefender SQL Balancer Maintenance Window

Filed under: Alerts,Announcements,ExchangeDefender — vlad @ 5:46 pm

This Friday, July 19th, 2013 we will be conducting an upgrade, stress testing and DDoS simulation against ExchangeDefender SQL backend, load balancer backend and the overall infrastructure fault tolerance starting at 3PM and ending at 6PM EST.

During the maintenance window mail flow will not be interrupted and mail delivery will not be delayed. Access to email, LiveArchive, mobile services and so on will not be interrupted either. Access to https://admin.exchangedefender.com site, including the ability to make configuration changes or release SPAM, will be interrupted periodically. Again, no email will be lost or delayed.

Following the upgrade our capacity across the core ExchangeDefender network will be increased which will enable us to launch additional services that will be discussed in the upcoming webinar. We apologize for the inconvenience this causes your clients as they may be presented with the message “We are currently conducting routine maintenance, please try your request again in a few minutes”

Scope of the effort

Isn’t this something you could do outside of the peak hours?

Yes, and we routinely make changes and upgrades during the standard weekend maintenance intervals.

For the purposes of this upgrade and an assessment of the new failover systems brought to the new version of ExchangeDefender, we wanted to run a maintenance interval during the peak usage and ascertain both our ability to handle traffic with the brains of the system offline (ExchangeDefender SQL) as well as test geographic redundancy during peak hours.

We felt it was better to have this done at the same time on a scheduled and controlled circumstances than under the unplanned service outage or DDoS.

As I will discuss next week, we have spent a fair amount of money and time upgrading the network not just to be able to sustain excessive outages but also distributed attacks, hacks and so on. We expect to be able to operate the network in a distributed and autonomous environment should we experience massive changes in Internet routing and availability of global networks. If you’ve been paying attention to the news, international government response to NSA espionage, rise in sophisticated hacking and network exploit activity, the reasons and the manner in which we are executing this maintenance window should be self explanatory.

As always, our staff will be available in higher than usual counts so please count on us.

We will also be providing a play-by-play updates as usual via NOC and our @xdnoc twitter.

Sincerely,
Vlad Mazek
CEO, ExchangeDefender
877-546-0316 x500


September 15, 2011

Massive Network Infrastructure Updates

Filed under: Alerts — vlad @ 7:29 pm

Over the course of the next few days Own Web Now Corp will be undergoing massive upgrades to our network infrastructure across our Dallas and Los Angeles data centers. We will take every precaution to limit service interruptions and while all maintenance will be performed well outside of business hours, many users will not even notice any issues as upgrades are being made to redundant systems.

Even with the best plans and best intentions, there can be brief service interruptions as we go along. We will continue to communicate with all of our clients and partners here: http://www.exchangedefender.com/noc

Thursday, September 15th, 2011

9 PM EST – ExchangeDefender in Los Angeles will be receiving new processing clusters for mail logs, inbound and outbound mail as well as LiveArchive.

Friday, September 16th, 2011

Noon – ExchangeDefender in Los Angeles will receive a new redundant LiveArchive network to improve geographic redundancy of our business continuity systems.

Saturday, September 17, 2011

Midnight EST – Power maintenance. We will be upgrading our power feeds, PDUs and UPS infrastructure across ExchangeDefender, hosting and more. We anticipate the work to be completed by 9AM EST.

Saturday Noon EST – ExchangeDefender inbound network upgrade. We are adding 25% more capacity to compensate for the growth in subscriber base.

Sunday, September 18, 2011

3 AM – 8 AM EST – ExchangeDefender will introduce geographic redundancy to our encryption, web file sharing and inbound mail routing capabilities.

All of these enhancements have been on the drawing board for months but were obviously reprioritized after the outage our Exchange 2010 network experienced in August. We moved extremely aggressively to make sure we provide a full geographically redundant network. Yes, experiencing one outage in 10 years related to power is not bad but we expect 100% uptime and this upgrade will help assure it.

To reiterate: We expect to minimize any issues and all maintenance will be done well outside of business hours. To stay informed stay tuned to http://www.exchangedefender.com/noc or @xdnoc


August 15, 2010

Pardon the sluggishness

Filed under: Alerts — vlad @ 10:59 pm

Typically this type of post will be made on our Network Operations site, but since it will only affect Own Web Now corporate web sites we decided to make it here.

Over the next week you may experience some network sluggishness while accessing corporate web pages, forums and blogs that belong to Own Web Now Corp. We are rebalancing and re-routing some internal and external connections on the 65.99.192.x range in order to optimize and correctly prioritize traffic on our corporate network. 

At times you may experience slow load times at www.ownwebnow.com and www.exchangedefender.com, including our support portal and our forums.


October 3, 2007

OWN Discontinues Email Based Support

Filed under: Alerts,Announcements,General — vlad @ 1:51 am

For the past eight months we have been testing our new support portal at https://support.ownwebnow.com and have had a lot of success in making sure support issues do not “fall off the table” as they tend to via email. Going forward, this will be the only way we will provide support, purchasing, cancellations and other account management.

Any mail sent to our previous support aliases will yield this automatic response:

*************************************
This is an automated message
*************************************

The address you sent an email to is not a monitored email address, your reply was destroyed.

Own Web Now Corp only provides support via our support portal at:
https://support.ownwebnow.com

If you have a support request or would like to update a support request please login to our support portal and create/update the support request directly. Doing so allows us to provide you with an SLA, guaranteed support request tracking, escalation and more.

Sincerely,
Own Web Now Corp Support Team
https://support.ownwebnow.com

Check out our blog:
http://www.ownwebnow.com/blog

There are several reasons we chose to eliminate the email support completely:

  • We have no tracking of when/if a support request was emailed
  • If the message is delayed because the support request was required to address a mail server issue we would get it far too late
  • Clients would forget to provide updates
  • Clients would keep the message thread going for support requests that were solved and were looking to address other issues
  • E-mail is an insecure way of conducting e-commerce
  • “Did you get my email?” <that they probably never sent>

There are far more reasons to drop email support but at the end of the day we needed a structured and fixed process of requesting support. At the moment there is only one way to get support, https://support.ownwebnow.com and we stand behind it with our SLA.

Thank you for doing business with us and I hope this change makes your support experience more enjoyable.


September 23, 2007

Global Operations Cycle Update

Filed under: Alerts,Announcements — vlad @ 3:37 am

4:35 AM EST Update – Network shutdown proceeded as planned. The network recovery process is beginning. Please stand by as all the services are restored. Please be aware that we will not be able to work any Urgent or High priority cases while the maintenance cycle is active. We will resume regular SLA at noon EST. Please keep an eye on the blog as we update it throughout the morning.

10:51 AM EST Update – Powergrid maintenance complete, network maintenance complete, cage “shuffles” complete. Moving on to application layer testing.

1:21 AM EST, The day after update – All services, networks, systems and operations back to normal. We will provide additional details later, we’re also lobbying for a day off


September 19, 2007

Huge Maintenance Cycle Announcement

Filed under: Alerts,Offsite Backups — vlad @ 1:33 pm

Three times a year Own Web Now Corp conducts a global network maintenance cycle. These maintenance cycles are meant to double-check the equipment, swap out aging infrastructure, improve cable management as execute a disaster recovery procedures. In plain terms, we take the network down at an announced time and work on it during off-peak hours so it doesn’t crash unannounced in the middle of the day.

Our global network maintenance cycle is scheduled for this Sunday, September 23, 2 AM – 5 AM Central (GMT -5).

All networks, all services, all customers will be affected. We will literally be shutting the NOC down and restarting it from scratch.

We will also have a minor ExchangeDefender Policy Engine upgrade during Saturday afternoon, the services should not be affected beyond perhaps a few minutes without control panels while we swap out the switches and nodes.


August 30, 2007

DDoS In Effect 8/30/2007 6AM – 2PM EST

Filed under: Alerts — vlad @ 1:15 pm

We are dealing with a fairly significant distributed denial of service attack (DDoS) at the moment and are doing all we can to mitigate the traffic. Please stand by.


August 15, 2007

ExchangeDefender Delivery Alert

Filed under: Alerts,ExchangeDefender — vlad @ 7:46 pm

Over the past 24 hours we have received a lot of complaints about the non-delivery of email to random recipients from random hosts on the Internet. At this point we have been able to narrow it down to Microsoft Exchange 2003 servers running IMF v2 and Microsoft Exchange 2007 servers also running IMF. In all instances the mail was acknowledged by ExchangeDefender, processed and delivered to the target mail server. In several instances mail disappeared completely. In others only certain recipients “didn’t get” the message.

We cannot stress this strongly enough: Turn OFF any SPAM filtering enabled on your servers, along with any SPF checking, validation or firewall port 25 data inspections. 

The only rules that should be enabled are those described in previous posts dealing with port 25 restrictions.

How come an email was received by one user but not on the other if they were cc’ed?

Frequent question asked today in our support portal. For example, two users in the same organization were receiving an email. Or perhaps a distribution list where a single email address delivers to Outlook/mbox and other relays externally to a Blackberry address. If one gets it and the other doesn’t thats ExchangeDefender’s fault, no?

No. ExchangeDefender does not modify/alter the message body nor does it split the message that is being delivered. The message is delivered in a single direct stream at which point Exchange further processes the message and delivers it to the delivery agent (which further relays the message or forwards it depending on local rules.) Exchange also uses technology called single instance storage, allowing a single message to be stored in the database if it is received for multiple recipients.

So, if a message was received by one user it must have been received by the other. Why can one see it while the other can’t? Turn up message tracking and see. However, we are currently seeing this as an issue related to Outlook Junk Filters and IMF, both of which need to be disabled for reliable mail delivery.


July 17, 2007

Extended Maintenance Cycle for 7/21/2007

Filed under: Alerts,Announcements — vlad @ 10:43 am

We are extending the maintenance cycle window for July 21, 2007. Our regular maintenance cycle is from 3AM to 7AM EST but due to the number of systems going online this week we are going to have to extend that window until noon EST. We will follow up with full details of the work being done but suffice to say it is significant.

If you are not familiar with our maintenance cycle activities do not worry, they are routine and they happen every week. Generally they are things you would expect – reboots to add more memory, storage, reboot for software patch installation, etc. At times there are other items such as electrical or network maintenance.

This particular weekend involves nearly all of the above plus a significant upgrade to our core infrastructure of ExchangeDefender, offsite backups and virtual servers. We’re bringing online 3 new data centers to top it off so we wanted to give ourselves more time to get everything done right.


Older Posts »