Friday – Sunday, January 5-7 we will be conducting routine network equipment maintenance.
Friday – Sunday, January 19-21 we will be conducting routine network equipment maintenance.
Remaining LiveArchive clusters will be moving from Los Angeles to Dallas. The infrastructure and the network layer is 100% owned by us and the DC exceeds our regulatory compliance requirements. We are moving our LiveArchive infrastructure away from our primary DC/network operations to improve fault tolerance and prepare for critical/catastrophy features that will be coming to LiveArchive in 2018.
Impact: We expect minimal service interruptions during this time as these systems are redundant but as per our policy anytime there is significant move of network resources and potential for isolated outages, we will be putting up a notice.
Over the next few days we will be conducting massive upgrades to the core of our ExchangeDefender network to add capacity and modernize some of the infrastructure that has served us really well. Here is a brief summary of the systems that will be affected:
support.ownwebnow.com – We will be upgrading our support portal and all the associated systems related to it from Saturday, June 24 – July 4th. Only time that user access will be affected is in the early hours of Sunday, June 24th. Systems are getting hardware and software upgrades and any interruption will be minimal and announced.
www.exchangedefender.com – We are upgrading our web site as well. Some portions of it may not be available from time to time on June 24 and June 25th.
outbound-jr, outbound-xd, outbound-enterprise, outbound-gov – Our entire outbound network is getting a massive overhaul to both address the growth and deliver the new features our clients have been asking for.
name services – We will be performing a minor upgrade to our DNS infrastructure. The current ns1.ownwebnow.com system has an uptime of 1,612 days (well over 4 years) and frankly, we’re just pushing our luck. It’s hot spare (that would take over immediately in the event of a failure) has already gone through two different Dell servers over the same time window and they are being replaced by something more robust.
These are not small changes and the work behind the scenes to make these upgrades has been going on for several months (and support.own case for nearly a year) and all the new systems that are coming in as a replacement have been burned in and online for months. We will do everything we can to minimize impact and will be putting announcements on @xdnoc handle on Twitter as well as site banners as the work is being done so that any interruptions can be minimized. As a global company we really don’t have a “downtime” when people are not using the network and our systems are massively scalable whereever technology allows it. That said, we are in an unprecedented time for hackers both in scale of attacks and the income potential so the threats our clients face are far more significant than interruptions during maintenance windows. As always, we appreciate the trust you put in us to manage your IT and we’ll keep you in the loop the whole way.
Over the course of the next few days Own Web Now Corp will be undergoing massive upgrades to our network infrastructure across our Dallas and Los Angeles data centers. We will take every precaution to limit service interruptions and while all maintenance will be performed well outside of business hours, many users will not even notice any issues as upgrades are being made to redundant systems.
The upgrades will impact all of our global services: Web hosting, ExchangeDefender, Exchange Hosting and more. Keep in mind that these are not individual computers – work is extremely complex and equipment is very sensitive. While we have planned our course of action very carefully there is always a possibility that certain service availability will be impacted or degregated. We will work diligently to communicate those issues here so you can stay in the loop. Here is a summary of maintenance work:
Thursday, September 15th, 2011
9 PM EST – ExchangeDefender in Los Angeles will be receiving new processing clusters for mail logs, inbound and outbound mail as well as LiveArchive.
Friday, September 16th, 2011
Noon – ExchangeDefender in Los Angeles will receive a new redundant LiveArchive network to improve geographic redundancy of our business continuity systems.
Saturday, September 17, 2011
Midnight EST – Power maintenance. We will be upgrading our power feeds, PDUs and UPS infrastructure across ExchangeDefender, hosting and more. We anticipate the work to be completed by 9AM EST.
Saturday Noon EST – ExchangeDefender inbound network upgrade. We are adding 25% more capacity to compensate for the growth in subscriber base.
Sunday, September 18, 2011
3 AM – 8 AM EST – ExchangeDefender will introduce geographic redundancy to our encryption, web file sharing and inbound mail routing capabilities.
All of these enhancements have been on the drawing board for months but were obviously reprioritized after the outage our Exchange 2010 network experienced in August. We moved extremely aggressively to make sure we provide a full geographically redundant network. Yes, experiencing one outage in 10 years related to power is not bad but we expect 100% uptime and this upgrade will help assure it.
On 7/20/2011 around 3:35 PM Eastern we started experiencing random packet loss across various services including Hosted Exchange and OWN Websites. Roughly around 3:45 PM Eastern, the random packet loss turned into a wide-spread service outage and lasted until 4:12 PM Eastern.
The incident appears to be faulted network driver on a Exchange monitoring server. Upon automatic recovery of the driver, the machine began to flood nearby network switches with invalid requests. Unfortunately the internal floods prevented access to the network analytic servers behind the DMZ. Since all machines received and responded to the request, all machines showed up as ‘flooding’ to the router and IDS was unable to determine the ‘source’ IP.
All services were essentially taken offline when the IDS started blocking traffic from the internal hosts. After we disabled the offending machine from the network and cleared IDS we were able to resume service across the board.
The biggest area of concern was the inability to contact us as the outage was occurring as the outage took down our support board and primary phone lines. We deeply apologize for the grief and trouble that this unexpected event caused and without saying, this has been the most impacting network event that we’ve experienced. We’ve implemented a new redundancy plan to our phone systems to handle global outages as this was the first time our phone systems were completely offline during a critical event.
We appreciate everything that our partners do for us and the patience that was extended yesterday as we definitely know that it was a very stressful event for our partners and their end users. As always we will continue to bring improvements to our solution stacks and address the areas where we may fall short.
As of 9:15 AM EST all services are back online.
We are addressing several performance issues on mail1.ownwebnow.com. We expect the resolution in a few minutes and will update this post at 9:15. As of 9:00, ability to send/receive mail from Own Web Now Linux Web hosted mailboxes is down.
Cause: Kernel page fault
Overnight many users received errors while attempting to connect to our admin.exchangedefender.com network. This issue was caused by the new backup systems that were recently put in place and has been addressed by 7:00 AM EST.
We apologize for any inconvenience this has caused our users, please rest assured that this issue has been addressed and that no mail was lost.
On November 25th between 17:00 – 19:00 Eastern OwnWebNow will be upgrading the OS version on its Value bandwidth router. Estimated period of impact is no more than fifteen minutes. During this time, the core router will require a reboot, interrupting service momentarily
We are seeing an unusually high load of outbound mail along with delays at major free email destinations like Yahoo, Google, Hotmail, etc. Please advise your users that outbound mail tends to have less of it’s usual real time feel around the holidays as most users take to shopping, opening up new computers that become immediately infected and turned into SPAM bots, time online increases and we see more and more overloading of free email providers.
We have posted a remarkable quarter and have focused exclusively on the features and onboarding support. Traditionally, that has been the biggest pain point expressed by our partners and the biggest obstacle to us as a growing organization. In order to provide the level of service and assurance required going forward, we’ve had to reprioritize how we do business in Q4 2009 and Q1 2010.
This weekend we will be revealing the first set of patches that have plagued nearly all of our products. Additionally, we are migrating away from several technologies that have, frankly, failed us and failed our partners for the level of quality you should expect from Own Web Now. We are not willing to point any fingers, you and your customers trust us to deliver rock solid solutions and at the end of the day it’s our decision to make the right choices.
Unfortunately, the reality is that we haven’t made some right choices in the past two quarters and in April we will be moving to address those issues, in all products across services.
There will be extended maintenance hours each weekend outside of business hours as we move to address the many problems noted in the NOC blog over the past few months.
As always, I appreciate your business and I speak to partners every day that love what we’re able to do. That is what drives us and my team strives to deliver more every single day. That is what we are good at – competing. We have dedicated 2010 to raising Own Web Now to the next level, so I hope you pardon the dust throughout April as we hold extended maintenance hours to bring a level of consistency.
I promise you that each and every step taken, to address some of the issues the few of you have been very vocal about, will be fully communicated through this blog.
mail1 network will be going offline for maintenance in 5 minutes. We expect the service interval to be completed within 10 minutes. This maintenance cycle is to add capacity to the network switching and failover configuration.
Maintenance period completed, services back at 100%.
There haven’t been many updates on the NOC site for quite some time, one of you noticed last Friday when we had an issue on the outbound network that was reported only once (SPAM load was too high that in turn corrupted a database and caused SPAM and real mail during a window to get delayed). The question is: Has OWN lost focus on the NOC site?
The quick answer is: No, we have not diminished the importance of the NOC blog.
The long form answer is that our service levels have improved significantly and with the improved staff, training, redundancy and several long-term problems addressed, we have not had many network events that warranted posts. The staff is busy working on the new products we will be launching this year and with the more aggressive growth in 2009 we have been far more aggressive in adding resources before the demand, not after.
As always, our service has improved thanks to your feedback and your continued business. We appreciate it all very much!
Happy New Year!
We have had an exciting 2008 with all the ExchangeDefender and Offsite Backup work. On behalf of my team I am sure you will appreciate that we hope for a far less exciting 2009 😉
Exchange Hosting – We are rolling out a new Exchange 2007 grid, huey.exchangedefender.com today. This is primarily to address the growth in our Exchange Hosting operation and eliminate any growing pains down the road.
ExchangeDefender AntiSpam – As announced on our official blog, the new ExchangeDefender engine is in play. We will be keeping an eye on this for any unusual behavior over the next few days.
Web Hosting – Migration to new hardware continues. We expect to complete web server migrations this week with SQL scheduled for the weekend.
Another installment of Monday updates on the state of our network and ongoing maintenance that takes place across our global network. One of the best parts of this job is that the challenges never stop even when most people in the world are taking a break.
Week in Review
Last week included extensive performance enhancements to our Enterprise Storage network and our ExchangeDefender network. Our ExchangeDefender network had a few network topology scenarios isolated that would create random mail delivery delays, something we have now addressed. Both our XD and ES networks struggled with a network hardware upgrade that really did not live up to the expectations. Unfortunately, when something doesn’t live up to expectations in an enterprise production network, discontinuing the use of that hardware is never a simple unplug – it is a slow migration and waiting for massive amounts of data to be replicated onto the replacement systems.
Great news is, our ExchangeDefender network and our Enterprise Storage network are now rock solid and we feel the additions and modifications over the past week will allow us to easily scale up the additional 30% we are predicting for the first quarter of 2009. Seeing what we’ve done with the “projections” in the past we are well under way to announcing the new Los Angeles data center currently being built out to support our ExchangeDefender network alone.
We wish you a Merry Migration
We are currently working on a massive upgrade to our virtual hosting infrastructure, the biggest one we’ve had since 2003. While this work is scheduled to last close to a month our first objective is to get the new hardware in place and move all the services to the new systems and establish a working baseline before performing the management software update towards the end of January.
Starting this week you can expect slight downtime (less than a minute per vserver) as the data rsync’s between the old and the new systems.
Good morning, happy Monday, welcome to another edition of tales from the network crypt!
We are light on entertainment today but very big on caffeine-induced energy because the weekend network infrastructure upgrades were quite uneventful. We have swapped out a good amount of aging hardware and hope that the upgrades to the network can keep up with the growth cycle. Speaking of which, we are officially invading Canada this week with native Exchange 2007, Offsite Backups, Web Hosting, SharePoint and then some. Over the past year we have grown significantly in Canada and the demand for localized services has been great.
Issues with RBLs – We constantly monitor our outbound mail flow and the RBL submissions and notifications. Because we own our IP address space we are notified whenever an abuse notification is filed against our IP space and we move to quickly deactivate the offending client. In instances when the IP address is blacklisted without our knowledge, or by a list that is not reputable, we follow the usual procedure of requesting delisting and re-routing the mail out through an IP that is not blacklisted.
What should you do? First, if you received a notification that the IP address was blocked you need to contact the recipients email server administrator. ExchangeDefender will make the best effort to contact the postmaster@ of the sites we have issues getting mail through but due to the volume and backlog our action will never be as fast as yours. Second, you should attempt to resend the message. Most of the RBL complaints we field at ExchangeDefender are not legitimate but just random rejections that overloaded servers make. Third, if this is a frequent / important client you should create a direct SMTP connector to their mail server and establish a trust with the organization. Finally, if you participate in any kind of broadcast mail, distribution group, personal mail lists or in any way send identical messages to multiple people we will not be able to assist you. Although you may not think it is SPAM that you are sending you have to understand that most antispam products use what is called Heuristics and bayesian analysis that tracks identical messages being relayed – computers are blind to your relationships with your customers, the double opt-in or even joke messages. Some of the largest service providers even keep a statistical model of messages and senders that have most of their mail moved to the Junk folder. So if you start to see and hear notifications that your mail is suddenly not getting to the recipients please consider outsourcing your mass/bulk mailing services to another email address or preferably any company. Important: Using ExchangeDefender for any mass mailing activity is a violation of AUP and outbound services can be suspended if the client gets caught. At the end of the day it really just comes down to courtesy: You can’t pay people to protect you from the junk mail you don’t want to read while you are directly contributing to the problem.
Issues with slipping SureSPAM – We have seen a significant pickup in these. Spammers, having been taken down in every imaginable way, are now exploiting the last bit of user stupidity – mirror whitelist. Here is how it works: Spammer will forge a message with your email address and send it to you. Your message will naturally end up in a SPAM bucket but far too many people have gotten used to clicking Trust Sender or WhiteList and have now created what is called a mirror whitelist – from me to me. When future SPAM messages come from this forged sender (you) they will automatically pass on through because no SPAM checks will be executed against a message placed in the whitelist. We often even find email administrators adding their own domains to whitelists on the global settings so please if you see [SURESPAM] or [SPAM] make sure you clean up your whitelist.
Looking Forward: Big launch week, keep an eye on http://www.ownwebnow.com/blog
Every Monday we publish network operations task list, important events and issues we are working on. We hope these updates help you get more in tune with the efforts taking place on the backend to keep OWN at 99.999% uptime.
On behalf of the NOC team thank you for your feedback. Many of you ask why we only publish the negative and service-affecting issues. That is what this blog is all about, things that are down or are going down that we are trying to fix because they are supposed to work, always. But if we must brag, the upgrades and hotfixes on our Exchange 2007 network have been incredible, all systems have been rock solid for a while now without a second of outage. The recent update to ExchangeDefender XD engine are filtering out more SPAM than we’ve ever been able to with the record new lows in false positives. So much for the good news 🙂
Items we are working on:
Upgrades: Thanksgiving weekend is when we make massive hardware upgrades to our shared services infrastructure. As devices age we routinely replace them (even if they are in perfect functional order) so Thursday – Sunday will involve a lot of new parts replacing the old parts. There should be no system outage until we replace the load balancers and that switch will be announced separately in advance.
Offsite Backup Scaling: Our Enterprise Storage Network is growing again, there have been some sporadic issues that we are still trying to address with EMC. Typical issues involve no email reports being generated because the backup job hangs at the last process (report email) and so far we’ve been able to isolate it to a write timeout.
Items that we are concerned about:
Exchange 2007 SP 1 Update Rollup 5: We will not be immediately rolling out the backup in production even after it clears our labs. We will be using the package on our intranet in production and the two volunteer sites before rolling it out to the remainder of the Exchange 2007 network.
See you in December!
AT&T is having RBL issues again, we are working with them to resolve the problem. You may receive this problem when emailing the AT&T network for the time being:
<<< 521-220.127.116.11 blocked by sbc:blacklist.mailrelay.att.net.
<<< 521 DNSRBL: Blocked for abuse. See http://att.net/blocks
554 5.0.0 Service unavailable
We have put in place a workaround and are working with AT&T to resolve the issue. You should not continue to see this problem. However, the issue is still open.
Good morning and happy Monday! This posting is part of our effort to keep you more in tune with the network operations group at Own Web Now because you rely on our services and should be a part of the conversation.
Extended maintenance windows over the weekend have been completed successfully and have addressed numerous growing pains and performance issues. Biggest changes were applied to our Exchange 2007 network and all the systems are now performing remarkably well. We would like to thank Microsoft CSS Exchange team for all the assistance they have rendered over the past two weeks in isolating creeping and unusual issues with the product that we have never encountered before.
Issues we are currently working on:
ExchangeDefender Reports Replication – At times the replication between our master maillog servers and slave reporting servers tends to lag. Sometimes the replication lags beyond our tolerance and appears to nearly stall. We have not yet isolated the issue that is casing this problem but are monitoring it very closely and adjusting replication points.
Offsite Backup Scaling – We are preparing our offsite backup network for another product addition to the matrix. You may experience slight availability issues as the management consoles reload while changes are taking place. This process will not interfere with the execution of offsite backups because the backup agent software automatically reloads.
ExchangeDefender SPAM Load – New ExchangeDefender antispam engine went online last week and the detection has been improved dramatically. We still haven’t kicked it up all the way but expect to by Wednesday. We have had several complaints about the level of SPAM and in our investigation have found that almost all of them are related to end user stupidity – whitelisting null senders <> and whitelisting their own domain or email address. When you whitelist any address you suppress ALL SPAM detection mechanisms, so when you whitelist your own address you open yourself up to a ton of SPAM. We have a process that we are preparing that will automatically remove null sender and mirror trusts because those should never be programmed in.
Lastly, an apology to our Canadian clients. Last week we let the cat out of the bag about our Canadian offsite backup product. This site is not intended to serve as an announcement board for new products, we just want to keep you informed about what our group is working on. From time to time what we are working on may not match the business side of Own Web Now so if you have a product or business related question we are not the most authorative source to provide an answer.
Earlier today we completed the rollout of 450 new servers to the ExchangeDefender family all over our American network. The introduction and initial sync of the new nodes did allow some junk through as well as introduce a slight today (maximum reported 1 hour from one system that nearly immediately went into maintenance mode) but as of roughly 11:30 AM EST all is good.
Additional 600 nodes are planned in our global expansion leading up to ExchangeDefender 4.0 launch. We are also looking at additional data centers on both coasts at the moment scheduled to go live this fall.
Update: 2:24 PM EST: We are happy to report that all the nodes have now converged in the scanning network and the SPAM filtering is back at its usual levels (and to be tightened up even further later tonight). You may have seen an increase in SPAM over the past few hours while the nodes were joining the network and accepting new programming but you should be seeing far less SPAM going forward.
We are currently investigating a network event in our Dallas region network centers, since approximately 5 AM EST we have been receiving complaints about network connectivity and availability. There are currently no outages and there have been no outages but certain customers are unable to reach the services on our network.
If you are experiencing an issue, please open a support request and include a traceroute to the service you are trying to reach (ex: 18.104.22.168).
We will update this ticket as soon as we have further information.
Update (9:05 AM EST, 14:05 GMT): We believe the network issue some of our customers have experienced has been resolved. Particularly affected were some of our UK customers (not BT) and local customers with Level 3 connectivity. Although we see the traffic back up at usual levels, it may take about an hour or so until all the mail catches up and gets delivered.
We have been informed that a number of customers have reported receiving 0’s in their intraday reports even though there was SPAM in the system at the time. We are still investigating what is causing the issue and expect to have the full restore to the intraday reports within 72 hours. In the meantime, we have suspended the intraday report generation as to eliminate the confusion from the user base.
Please remember that the primary way to check for SPAM is through the ExchangeDefender portal at https://admin.exchangedefender.com and for the customers that need a reliable, realtime reminder of the SPAM that is being kept from their mailbox we recommend the SPAM Monitor software designed to sit on the desktop and show alerts. We still consider email reports to be an important part of our offering but due to the their nature (spam digests of spam messages) we urge you to consider the more reliable, realtime alternatives.