Another installment of Monday updates on the state of our network and ongoing maintenance that takes place across our global network. One of the best parts of this job is that the challenges never stop even when most people in the world are taking a break.
Week in Review
Last week included extensive performance enhancements to our Enterprise Storage network and our ExchangeDefender network. Our ExchangeDefender network had a few network topology scenarios isolated that would create random mail delivery delays, something we have now addressed. Both our XD and ES networks struggled with a network hardware upgrade that really did not live up to the expectations. Unfortunately, when something doesn’t live up to expectations in an enterprise production network, discontinuing the use of that hardware is never a simple unplug – it is a slow migration and waiting for massive amounts of data to be replicated onto the replacement systems.
Great news is, our ExchangeDefender network and our Enterprise Storage network are now rock solid and we feel the additions and modifications over the past week will allow us to easily scale up the additional 30% we are predicting for the first quarter of 2009. Seeing what we’ve done with the “projections” in the past we are well under way to announcing the new Los Angeles data center currently being built out to support our ExchangeDefender network alone.
We wish you a Merry Migration
We are currently working on a massive upgrade to our virtual hosting infrastructure, the biggest one we’ve had since 2003. While this work is scheduled to last close to a month our first objective is to get the new hardware in place and move all the services to the new systems and establish a working baseline before performing the management software update towards the end of January.
Starting this week you can expect slight downtime (less than a minute per vserver) as the data rsync’s between the old and the new systems.
We will be conducting out-of-band maintenance on backup73.ownwebnow.com to introduce new monitoring systems and improve performance. The maintenance will start at 8 PM EST and complete by 10 PM EST (~ 2 hours)
We hope this does not cause undue inconvenience.
Thank you for joining us for the Monday network infrastructure upgrade.
It has been an interesting week at Own Web Now. Following the major hardware swaps during the Thanksgiving break we have found several new pieces of hardware that really did not live up to the standards or the tests we put them through. What can you do, when you work in IT for a living you work to swap out disappointing upgrades.
Now on to something serious. In the past two weeks we have found two critical issues that have already been addressed:
Offsite Backups – We have discontinued the use of 1.5 TB drives due to the high level of failure during production. We have also isolated a single point of failure in our offsite backup strategy with the ways we were storing our security database and our file storage. This resulted in having to move approximately 30 customers from one grid to another and restoring their backups in reverse (from replication partners to masters) which is a painstakingly slow process. Over the past week we have also worked to remove this limitation from our systems and our enterprise backup.
ExchangeDefender – Last Thursday we had several messages with extended delays during which we have discovered several important problems. First, ExchangeDefender should not be used to route inbound mail from your own network to our inbound network. Please rely on a local SMTP service because our inbound network is configured not to expect local mail appearing on an external interface. Seems like common sense but apparently some users have their SMTP servers set to route using DNS lookups instead of using local SMTP servers or connectors. ExchangeDefender inbound network is not designed to function as your SMTP server, in fact quite the opposite – mail for local domains appearing from outside the ExchangeDefender network is put through further scanning as a possible forgery. This is either a flaw or a designed security feature depending on who you ask.
This week we will be rolling out two additional Exchange 2007 networks to help address the demand for the service. We will also be provisioning a shared cluster hosting concept and expanding storage allotments for our FTP and file storage services.
We are conducting an automation out-of-band update between Midnight and 2 AM tonight, Friday, December 12th, 2008. During this time you will not be able to create, delete or modify any Exchange 2007, SharePoint 3 or Offsite Backup accounts.
We are conducting an update to Shockey Monkey to bring in multi-tenant and private branding for our Shockey Monkey customers who wish to offer the same control panels you use at OwnWebNow to your customers within your own Shockey Monkey portal. This was previously mentioned as a concept by Vlad Mazek in his private blog posting “7th Monkey of the Seventh Monkey”
Update: We are currently investigating performance problems from 5 AM – 8:20 AM on our outbound network. One of the load balancers handling the outbound mail failed, creating a performance issue on the other server and backing up customers SMTP queues due to load throttling. No mail was lost.
This issue has been resolved at 8:00 AM EST but is being monitored further as the systems catch up and sync up.
Good morning, happy Monday, welcome to another edition of tales from the network crypt!
We are light on entertainment today but very big on caffeine-induced energy because the weekend network infrastructure upgrades were quite uneventful. We have swapped out a good amount of aging hardware and hope that the upgrades to the network can keep up with the growth cycle. Speaking of which, we are officially invading Canada this week with native Exchange 2007, Offsite Backups, Web Hosting, SharePoint and then some. Over the past year we have grown significantly in Canada and the demand for localized services has been great.
Issues with RBLs – We constantly monitor our outbound mail flow and the RBL submissions and notifications. Because we own our IP address space we are notified whenever an abuse notification is filed against our IP space and we move to quickly deactivate the offending client. In instances when the IP address is blacklisted without our knowledge, or by a list that is not reputable, we follow the usual procedure of requesting delisting and re-routing the mail out through an IP that is not blacklisted.
What should you do? First, if you received a notification that the IP address was blocked you need to contact the recipients email server administrator. ExchangeDefender will make the best effort to contact the postmaster@ of the sites we have issues getting mail through but due to the volume and backlog our action will never be as fast as yours. Second, you should attempt to resend the message. Most of the RBL complaints we field at ExchangeDefender are not legitimate but just random rejections that overloaded servers make. Third, if this is a frequent / important client you should create a direct SMTP connector to their mail server and establish a trust with the organization. Finally, if you participate in any kind of broadcast mail, distribution group, personal mail lists or in any way send identical messages to multiple people we will not be able to assist you. Although you may not think it is SPAM that you are sending you have to understand that most antispam products use what is called Heuristics and bayesian analysis that tracks identical messages being relayed – computers are blind to your relationships with your customers, the double opt-in or even joke messages. Some of the largest service providers even keep a statistical model of messages and senders that have most of their mail moved to the Junk folder. So if you start to see and hear notifications that your mail is suddenly not getting to the recipients please consider outsourcing your mass/bulk mailing services to another email address or preferably any company. Important: Using ExchangeDefender for any mass mailing activity is a violation of AUP and outbound services can be suspended if the client gets caught. At the end of the day it really just comes down to courtesy: You can’t pay people to protect you from the junk mail you don’t want to read while you are directly contributing to the problem.
Issues with slipping SureSPAM – We have seen a significant pickup in these. Spammers, having been taken down in every imaginable way, are now exploiting the last bit of user stupidity – mirror whitelist. Here is how it works: Spammer will forge a message with your email address and send it to you. Your message will naturally end up in a SPAM bucket but far too many people have gotten used to clicking Trust Sender or WhiteList and have now created what is called a mirror whitelist – from me to me. When future SPAM messages come from this forged sender (you) they will automatically pass on through because no SPAM checks will be executed against a message placed in the whitelist. We often even find email administrators adding their own domains to whitelists on the global settings so please if you see [SURESPAM] or [SPAM] make sure you clean up your whitelist.
Looking Forward: Big launch week, keep an eye on http://www.ownwebnow.com/blog