Offsite Backup Expansion Complete

The previously mentioned offsite backup work and SAN maintenance has been completed. All systems should be back to normal and performance should be back to rock solid. We will be doing some final stress testing scheduled for tonight, along with a few DR scenarios so its likely that tonight is the last night your backups get interrupted. If you do start experiencing problems with your backups on Tuesday please open up a trouble ticket and we will do our best to get to the bottom of the issues.

Thank you for your patience and sorry about the inconvenience. As mentioned before, we are crediting the month of service to all customers regardless of whether you were affected by the performance issues or not. 

Growing the SAN Infrastructure

The backup SAN infrastructure scaling work has officially started, things are moving along very well. Unfortunately, the data copy and sync process is very time consuming and we expect it to take a better part of the day. We expect this move to drastically improve the performance and reliability of the offsite backup service and allow us to stack additional features that you have been asking for since we started offering this service.

We will post an update here when the upgrades have been completed.

Big Changes at Own Web Now

Dear Clients and Partners,

   I wanted to take a moment to write to you about the changes that will be happening in our company in the very near future.

   I first want to thank you for your business and your loyalty over the years. If you’ve been with us for a while you know of our dedication to service and delivering the services that make sense. As we embark on the next growth stage of our company, I want you to know that the same commitment to service and offerings that make sense will stay at the core of who we are and what we do.

   Over the course of next 30 days we will be bringing online more ways to obtain support as well as an easier way to track, escalate and manage IT support for your business. Following that we will be publishing several training and troubleshooting systems to get you up to speed and let you take the advantage of all the services we offer. Finally, we intend to become more aggressive in our release schedule as the new Microsoft platforms launch, bringing you more applications and services that you can use without a large up-front commitment or long contracts.

   What is really changing, at the core, is that until now we have been an infrastructure building company. The work that has gone into Own Web Now Corp since Jan of 2006 has been to transform it to the solution delivery organization. In plain terms that means: we will no longer only build the networks, we will focus on helping you take advantage of them.

   Literally everything at Own Web Now is changing, expect more communication from me over the next month detailing all that we’ve been working on for the past 15 months. You’ll love what we have built, and we look forward to continuing down the path of offering solutions that make sense.

Sincerely,
Vlad Mazek
CEO, Own Web Now Corp

Backup Systems Maintenance

We have scheduled an emergency maintenance interval on our Offsite Backup System network for Wednesday, June 6th, 2007. We are adding new hardware, upgrading some of the software and replication systems, adding more equipment to the mix and extending the reporting functionality to offer more reporting and branding functionality to you and your clients.

The backup network will be unreachable to the public between 10 PM EST – 11 PM EST.

Furthermore, we will be moving certain users to a more scalable network storage system. Because this change affects a very small group of users and involves a more extended downtime, you will be notified directly with some contingency planning and preparation as we intend to make the move in off-peak hours during the weekend when backup activity and change management is low. While you will have the ability to retrieve data during this interval, you will not be able to store more data to it.

Thank you for your patience. Backups and proper disaster recovery planning are very important to us and we realize that the offsite component is the most important one, after all, it is the one you use for compliance and in case everything else disappears, so we’re doing all we can to make sure that data is there when you need it.

Offsite Backup Apology

Dear Partners,

I wanted to address an issue with our offsite backups that started last week with the rollout of new storage arrays used for Own Web Now offsite backups. As you already know, we take great pride in providing rock solid solutions that scale worldwide. We put in countless hours in design, testing, analysis and performance tuning on all services and you can expect the OWN services to be rock solid. That in fact is what you and your customers pay for.

Last week one of the new storage arrays we have added to the mix started to cause issues with the server code that actually maps the logical partitions and makes them available for access and replication to our backbone, offsite centers and your clients. Offsite backups are a partner service only, we do not offer it in direct retail. You pay what it costs us to provide the same redundant layer of data protection we expect and maintain for ourselves. I know that if I lose a backup, your customer will be out of business, you and I will be in the court. That is why I insist on the highest grade quality when it comes to protecting data. That is also why we give it away to our partners at cost – I want you to be the hero for that business that just lost everything and made it back thanks to you.

So when that promise is not delivered on, believe me, I am the most disappointed one. For close to our week a small portion of our clients had to put up with unreliable backups, with backup jobs not completing. Even though you are paying at cost and there is no SLA to speak of at all, I want you to be aware of how dedicated we are to our partnership. We will waive all the June monthly fees as a result of the issues some of you faced. Thats right, even if you were not impacted you will be getting a free month. We understand the enormity of what you put in our hands, we want you to know that you have a partner with a global presence by your side, one that stands by their products. This not only applies to offsite backups but everything we do.

So in closing, thank you for your business and the faith you put in Own Web Now Corp. I am sorry that we have disappointed some of you and in sign of good faith we will eat the fees associated with the offsite backups through the month of June. Our offsite backups have been rock solid since we started them, we aim to keep that level of reliability.

Thank you.

Sincerely,

Vlad Mazek, MCSE

Note: What actually happened? We use eSATA II chasis rack raid systems for backups. These systems are further replicated off to Chicago via private Gigabit network. We use Samba (SMB/CIFS) software on both the appliance and on the remote network to create replication partners. Due to the incompatibilities between the driver and Samba 3.0.10–1.4E 12.2 software our replication systems created enormous amount of traffic and requests of the NAS systems. As a result, the load on the systems spiked, made those partitions unavailable and as a result made reporting and access difficult for some customers (roughly 300 were affected). I would like to thank our partners at EMC and AhSay for their assistance with troubleshooting of the issue and my staff for putting in very long hours over the Memorial Day weeekend to get to the bottom of it.

Product Demo Webcast

Vlad Mazek, CEO of Own Web Now Corp, is presenting to two small business user groups tonight on a variety of technology and business subjects Own Web Now Corp is involved in. If you’d like to attend the access is free, details below:

Tonight’s presentation is free, live over the web and starts at 7:30 PM EST. It ends at approximately midnight EST. Here is the preliminary schedule.

7:30 – 8:30 EST: Exchange 2007 Overview (Level 100)

8:30 – 9:00 EST: ExchangeDefender, Shockey Monkey (Level 0)

9:30 – Midnight: Virtualized Services (Level 200), Shockey Monkey, ExchangeDefender, and other tools we make (and why) that IT solution providers are using to make more money. These will be product demo’s, not sales pitches.

Is there anything else you’d like to hear? Since we’re pretty much inviting SBSers worldwide, is there something you’d like to see? See you tonight, livemeeting info below:

Meeting URL: Livemeeting Address

Meeting ID: D7QS3H

Meeting Key: bn6′%~P

Emergency Maintenance Window on Load Balancers

Earlier this morning we identified an issue with our load balancing network and moved to quickly update the firmware and restart the services. This emergency maintenance window has caused delays between 5 and 15 minutes accross our entire network between 10:00 – 10:30 AM EST (13:00  – 13:30 GMT).

This window was not planned and did not affect everyone at the same time. As we applied the load balancer firmware update and cycled the mail currently being processed on that network was temporarily disconnected from the Internet. As soon as the load balancer restarted the normal operation resumed.

While it is unlikely that anyone would have noticed this outage we felt it was important to let you know every time our service is affected.

Backups are back and bigger than ever

Over the past week we have been working very hard at expanding our storage network capacity. The demand for our new off-site backup system has been far greater than we ever imagined so we had to grow our network significantly to accomodate all of our partners that wanted to deploy this service for their customers. Who can blame them, we use the same system to backup all of the servers we manage as well.

So what’s new? Additional network capacity and additional storage. Newer version of AhSay. Improved replication system. The amount of work that went into putting this together was significant, we have dedicated over 5 days to it.

How does it work?

Each client (workstation or server) installs a small branded agent on the PC that is being backed up. The agent runs on Windows, Linux, Mac OS X, Solaris and Novell. The agent backs data up to the server on the byte level, this way only changes within the file are propagated instead of the entire file each time there is a change. The client keeps the information on the backup schedule and retention policies (how long a file should be available for restore) and is capable of backing up Microsoft SQL Server, Microsoft Exchange (both on store level and optionally mailbox brick level down to the message or folder), Volume Shadow Copy, Oracle, MySQL, etc.

The infrastructure, frankly, is impressive. The main AhSay OBS server runs on the perimeter of our Dallas/InfoMart network and receives/sends data from the client agent. The entire server is backed up daily to a local SAN for quick disaster recovery purposes (in case of RAID failure). The entire server is also replicated in realtime to a geographically redundant data center in Chicago/CBOT using AhSay replication server. We know how important your data is so we do our best to protect it as best as we can, 3–way replication with 2 of them in sync at all times.

We’d like to thank our customers for their patience during this process. If you were on trial and experienced any issues please contact us and we will extend your backup trial by additional 30 days.

Maintenance Followup

All planned and scheduled maintenance for this weekend has been completed and finalized far earlier than we anticipated. The new power switches are in place, new servers have been activated, and our worldwide global mainteanance tasks have been completed. This window has not only addressed many problems we have had over the past three months but has also expanded our capacity and platform ability to deliver further ExchangeDefender and The Office Server products over the next quarter.

Please keep in mind that even though the maintenane window is complete there still may be some sporadic performance and service issues over the next 48 hours. All our data centers are fully staffed following the maintenance cycle while we evaluate and adjust our systems to monitor the network at the new capacity.

Maintenance Window

The next 72 hours will include series of six extensive maintenance tasks spanning three USA-based data centers and two EU-based data centers.

Maintenance Tasks

The first and largest maintenance task involves ExchangeDefender. First we will be adding more servers to the mix. This is a very routine task for us and is expected to produce zero downtime even on the nodes that are active during the transition. After the new nodes are online and older hardware has been swapped out we will proceed to roll out new policy systems which will speed up the daily email SPAM report generation, portal access, MSP control panels and higher-end reseller branding. These processes will continue throughout the weekend and while we will not have any downtime the mail release delays could be delayed up to five minutes. If your message is affected by this delay you will be given a notice when you attempt to release the message.

The second maintenance task involves TheOfficeServer dedicated servers and shared hosted services. We are expanding our portfolio to provide APC MasterSwitch remote reboot to address the complaints regarding Microsoft Windows Server patching problems over the past few months. The APC MasterSwitch software will allow you to remotely power cycle the server through a web interface when the system becomes unresponsive. In order to provide this service we will be gradually powering down and swapping power strips. We expect the downtime related to this task to take roughly 10 minutes. Our engineers will login to your server, trigger a graceful shutdown and move the power cord to the new strip. The downtime exclusively applies to the single switched power supplies, if you have a dual power supply you will not have any downtime associated with this maintenance task.

Own Web Now hosting control panels will be going offline for a period of five minutes during our regularly scheduled maintenance window from 7AM – 8AM GMT on Saturday, December 9th, 2006. This outage will not affect any services (dns, mail, web, databases, etc) but you will not be able to make any configuration changes during this window. If you do have an emergency maintenance issue please contact Own Web Now Corp through the trouble ticket support system.

Warranties

As always, all Own Web Now Corp maintenance tasks are fully scripted, practiced and load tested before being executed on the live production network. We have load tested each ExchangeDefender node at least 500 hours, each power strip has been placed to 80% load (manufacturer recommended maximum load) and held for 24 hours. We have full confidence in the hardware that is being placed into production.

Software Updates

ExchangeDefender has undergone significant changes in underlying infrastructure and reporting services to accommodate the global growth of both our network and our customer base. Although initially you will not see a visual difference in what we have changed you will notice a significant performance boost. This is directly related to the changes in the way we currently collect and build reporting for ExchangeDefender.

Finally,

Thank you for all your support, understanding and patience. Nearly all the changes we are making to ExchangeDefender, Own Web Now Corp and The Office Server systems are directly designed by the feedback submitted by you, our partners and customers. While many of you can appreciate that the recent instabilities have been caused by Microsoft’s diminishing QA of security patches, I personally want to make you aware that we are doing everything we can to both eliminate the problems and continue to work with Microsoft to make sure these issues are minimized in the future.

At no time do we lose sight of the trust you place in us to manage your systems and we’re doing everything we can to deliver on our promise and our message. Thank you for your business!

Sincerely,

Vlad Mazek, CEO, Own Web Now Corp.