Emergency Maintenance Window on Load Balancers
Earlier this morning we identified an issue with our load balancing network and moved to quickly update the firmware and restart the services. This emergency maintenance window has caused delays between 5 and 15 minutes accross our entire network between 10:00 – 10:30 AM EST (13:00 – 13:30 GMT).
This window was not planned and did not affect everyone at the same time. As we applied the load balancer firmware update and cycled the mail currently being processed on that network was temporarily disconnected from the Internet. As soon as the load balancer restarted the normal operation resumed.
While it is unlikely that anyone would have noticed this outage we felt it was important to let you know every time our service is affected.
Backups are back and bigger than ever
Over the past week we have been working very hard at expanding our storage network capacity. The demand for our new off-site backup system has been far greater than we ever imagined so we had to grow our network significantly to accomodate all of our partners that wanted to deploy this service for their customers. Who can blame them, we use the same system to backup all of the servers we manage as well.
So what’s new? Additional network capacity and additional storage. Newer version of AhSay. Improved replication system. The amount of work that went into putting this together was significant, we have dedicated over 5 days to it.
How does it work?
Each client (workstation or server) installs a small branded agent on the PC that is being backed up. The agent runs on Windows, Linux, Mac OS X, Solaris and Novell. The agent backs data up to the server on the byte level, this way only changes within the file are propagated instead of the entire file each time there is a change. The client keeps the information on the backup schedule and retention policies (how long a file should be available for restore) and is capable of backing up Microsoft SQL Server, Microsoft Exchange (both on store level and optionally mailbox brick level down to the message or folder), Volume Shadow Copy, Oracle, MySQL, etc.
The infrastructure, frankly, is impressive. The main AhSay OBS server runs on the perimeter of our Dallas/InfoMart network and receives/sends data from the client agent. The entire server is backed up daily to a local SAN for quick disaster recovery purposes (in case of RAID failure). The entire server is also replicated in realtime to a geographically redundant data center in Chicago/CBOT using AhSay replication server. We know how important your data is so we do our best to protect it as best as we can, 3–way replication with 2 of them in sync at all times.
We’d like to thank our customers for their patience during this process. If you were on trial and experienced any issues please contact us and we will extend your backup trial by additional 30 days.
Maintenance Followup
All planned and scheduled maintenance for this weekend has been completed and finalized far earlier than we anticipated. The new power switches are in place, new servers have been activated, and our worldwide global mainteanance tasks have been completed. This window has not only addressed many problems we have had over the past three months but has also expanded our capacity and platform ability to deliver further ExchangeDefender and The Office Server products over the next quarter.
Please keep in mind that even though the maintenane window is complete there still may be some sporadic performance and service issues over the next 48 hours. All our data centers are fully staffed following the maintenance cycle while we evaluate and adjust our systems to monitor the network at the new capacity.
Maintenance Window
The next 72 hours will include series of six extensive maintenance tasks spanning three USA-based data centers and two EU-based data centers.
Maintenance Tasks
The first and largest maintenance task involves ExchangeDefender. First we will be adding more servers to the mix. This is a very routine task for us and is expected to produce zero downtime even on the nodes that are active during the transition. After the new nodes are online and older hardware has been swapped out we will proceed to roll out new policy systems which will speed up the daily email SPAM report generation, portal access, MSP control panels and higher-end reseller branding. These processes will continue throughout the weekend and while we will not have any downtime the mail release delays could be delayed up to five minutes. If your message is affected by this delay you will be given a notice when you attempt to release the message.
The second maintenance task involves TheOfficeServer dedicated servers and shared hosted services. We are expanding our portfolio to provide APC MasterSwitch remote reboot to address the complaints regarding Microsoft Windows Server patching problems over the past few months. The APC MasterSwitch software will allow you to remotely power cycle the server through a web interface when the system becomes unresponsive. In order to provide this service we will be gradually powering down and swapping power strips. We expect the downtime related to this task to take roughly 10 minutes. Our engineers will login to your server, trigger a graceful shutdown and move the power cord to the new strip. The downtime exclusively applies to the single switched power supplies, if you have a dual power supply you will not have any downtime associated with this maintenance task.
Own Web Now hosting control panels will be going offline for a period of five minutes during our regularly scheduled maintenance window from 7AM – 8AM GMT on Saturday, December 9th, 2006. This outage will not affect any services (dns, mail, web, databases, etc) but you will not be able to make any configuration changes during this window. If you do have an emergency maintenance issue please contact Own Web Now Corp through the trouble ticket support system.
Warranties
As always, all Own Web Now Corp maintenance tasks are fully scripted, practiced and load tested before being executed on the live production network. We have load tested each ExchangeDefender node at least 500 hours, each power strip has been placed to 80% load (manufacturer recommended maximum load) and held for 24 hours. We have full confidence in the hardware that is being placed into production.
Software Updates
ExchangeDefender has undergone significant changes in underlying infrastructure and reporting services to accommodate the global growth of both our network and our customer base. Although initially you will not see a visual difference in what we have changed you will notice a significant performance boost. This is directly related to the changes in the way we currently collect and build reporting for ExchangeDefender.
Finally,
Thank you for all your support, understanding and patience. Nearly all the changes we are making to ExchangeDefender, Own Web Now Corp and The Office Server systems are directly designed by the feedback submitted by you, our partners and customers. While many of you can appreciate that the recent instabilities have been caused by Microsoft’s diminishing QA of security patches, I personally want to make you aware that we are doing everything we can to both eliminate the problems and continue to work with Microsoft to make sure these issues are minimized in the future.
At no time do we lose sight of the trust you place in us to manage your systems and we’re doing everything we can to deliver on our promise and our message. Thank you for your business!
Sincerely,
Vlad Mazek, CEO, Own Web Now Corp.
Mail Server Upgrades Tonight
We wanted to give all our shared hosting customers a notice that mail servers at Own Web Now Corp will be undergoing and update tonight, Tuesday, November 28, 2006. The maintenance window is scheduled for 07:00 – 08:00 GMT. This translates to 2–3AM EST, 11PM-Midnight PST, 6–7PM Syndey, Australia. This maintenance window is scheduled to occur during the off peak hours where our network experiences the least amount of traffic. Having said all that you should not experience any outages or any downtime.
The upgrades:
- Brand new load balancing infrastructure
- Additional global replication servers for archiving
- Additional webmail options for mobile workers
- Optimization of the mobility components to allow Windows Mobile 5 users to recive similar Push Mail technology via our POP3 servers
- Upgrades to antivirus and antispam infrastructure
Two important notes: This maintenance window does not affect ExchangeDefender customers.You will not experience any downtime. Your users will not see any changes to the interfaces they are currently used to.
Update: Tuesday, November 28, 2006 – 2PM GMT – The upgrades to our hosted mail server network have been completed. No downtime our outages to report. Everything went a according to plan and performance is within expected parameters. We are continuing to monitor the changes and stand by to respond to any unforseen problems. So far there are none to report but if they are they will be announced on this blog.
Own Web Now’s Ultimate Vista Launch
For well over a year we have been beta testing Microsoft’s next operating system – Microsoft Vista. At first we did not like what was on the table and we lost many computers scratching our head, wondering just what this new OS is going to do for us. Year later, we’re very excited about what is coming out and want to give you the opportunity to try out Vista and Office on your own turf.
Starting December 1, 2006 we start shipping 1000 PC’s with the latest release candidate of Microsoft Vista Ultimate Edition with Microsoft Office 2007 Beta 2 Technical Refresh to our managed sites. Here is what’s coming:
Intel Pentium D 930 (Dual Core) 3.0 GHz LGA 755 Processor
2 GB DDR2 SDRAM Desktop Memory
2x 80 SATA II Western Digital Hard Drive
Storage Fault Tollerance (RAID 1 / Mirror)
NVIDIA GeForce 6200 128MB DDR Video Card
NVIDIA nForce 590 & Marvell Gigabit NIC
7.1 Surround Sound
USB, USB 2.0 & Firewire Connections
These workstations will be fully supported by Own Web Now Corp free of charge to give you an idea of what the new OS can do for your company. You will have them in your office for 6 months to try out your line of business applications, experience the productivity benefits of the new Office and Vista first hand and get the support from a team that has been running it for over a year.
The cost? Free. If you’re on our managed services plan, you’re getting one! Yes, really. There is no catch, no gotcha, no naming your first-born Vlad. As a part of our Vista launch we’re sending all our managed site clients a free PC for six months to experience Vista/Office combo because we know that once you try it, you’ll love it. Six months later the software will expire and we can talk about the best way to implement Vista, Longhorn and Office to work for your business.
OWN Blog: First Post
Welcome to the very first post of the Own Web Now Blog. My name is Vlad Mazek and I am the CEO of Own Web Now Corp. If you are our customer you have met or at least spoken to me.
Over the past two years I have been sharing insights into how OWN does business, what we do, what our culture is like, how we work on problems. Some of these posts were quite personal and were meant to give outsiders an idea on what it’s like to run a global company. Lately, we have seen and heard a need for a central place for us to communicate with you – our customer and partner – on what we’re doing.
So again, welcome to the OWN Blog. This is the area where my staff and I will be posting frequent updates about what we’re working on. Sometimes we will have announcements, sometimes we will have alerts, promotions, specials. On a more technical and business side we will talk about our core – technology behind advanced network infrastructure, SMB networks, hosting, virtualization, ExchangeDefender, spam defense, security issues, Shockey Monkey and more.
So welcome to the Own Web Now Blog. Please click on the link below to add us to your RSS reader and keep up with us. We look forward to working with you and as always, we thank you for your business:
http://www.ownwebnow.com/blog/feed/rss2
Sincerely,
Vlad Mazek, MCSE, Exchange MVP
CEO, Own Web Now Corp