June 10, 2007
The previously mentioned offsite backup work and SAN maintenance has been completed. All systems should be back to normal and performance should be back to rock solid. We will be doing some final stress testing scheduled for tonight, along with a few DR scenarios so its likely that tonight is the last night your backups get interrupted. If you do start experiencing problems with your backups on Tuesday please open up a trouble ticket and we will do our best to get to the bottom of the issues.
Thank you for your patience and sorry about the inconvenience. As mentioned before, we are crediting the month of service to all customers regardless of whether you were affected by the performance issues or not.
Comments Off on Offsite Backup Expansion Complete
June 8, 2007
The backup SAN infrastructure scaling work has officially started, things are moving along very well. Unfortunately, the data copy and sync process is very time consuming and we expect it to take a better part of the day. We expect this move to drastically improve the performance and reliability of the offsite backup service and allow us to stack additional features that you have been asking for since we started offering this service.
We will post an update here when the upgrades have been completed.
Comments Off on Growing the SAN Infrastructure
June 7, 2007
Dear Clients and Partners,
I wanted to take a moment to write to you about the changes that will be happening in our company in the very near future.
I first want to thank you for your business and your loyalty over the years. If you’ve been with us for a while you know of our dedication to service and delivering the services that make sense. As we embark on the next growth stage of our company, I want you to know that the same commitment to service and offerings that make sense will stay at the core of who we are and what we do.
Over the course of next 30 days we will be bringing online more ways to obtain support as well as an easier way to track, escalate and manage IT support for your business. Following that we will be publishing several training and troubleshooting systems to get you up to speed and let you take the advantage of all the services we offer. Finally, we intend to become more aggressive in our release schedule as the new Microsoft platforms launch, bringing you more applications and services that you can use without a large up-front commitment or long contracts.
What is really changing, at the core, is that until now we have been an infrastructure building company. The work that has gone into Own Web Now Corp since Jan of 2006 has been to transform it to the solution delivery organization. In plain terms that means: we will no longer only build the networks, we will focus on helping you take advantage of them.
Literally everything at Own Web Now is changing, expect more communication from me over the next month detailing all that we’ve been working on for the past 15 months. You’ll love what we have built, and we look forward to continuing down the path of offering solutions that make sense.
CEO, Own Web Now Corp
Comments Off on Big Changes at Own Web Now
May 29, 2007
I wanted to address an issue with our offsite backups that started last week with the rollout of new storage arrays used for Own Web Now offsite backups. As you already know, we take great pride in providing rock solid solutions that scale worldwide. We put in countless hours in design, testing, analysis and performance tuning on all services and you can expect the OWN services to be rock solid. That in fact is what you and your customers pay for.
Last week one of the new storage arrays we have added to the mix started to cause issues with the server code that actually maps the logical partitions and makes them available for access and replication to our backbone, offsite centers and your clients. Offsite backups are a partner service only, we do not offer it in direct retail. You pay what it costs us to provide the same redundant layer of data protection we expect and maintain for ourselves. I know that if I lose a backup, your customer will be out of business, you and I will be in the court. That is why I insist on the highest grade quality when it comes to protecting data. That is also why we give it away to our partners at cost – I want you to be the hero for that business that just lost everything and made it back thanks to you.
So when that promise is not delivered on, believe me, I am the most disappointed one. For close to our week a small portion of our clients had to put up with unreliable backups, with backup jobs not completing. Even though you are paying at cost and there is no SLA to speak of at all, I want you to be aware of how dedicated we are to our partnership. We will waive all the June monthly fees as a result of the issues some of you faced. Thats right, even if you were not impacted you will be getting a free month. We understand the enormity of what you put in our hands, we want you to know that you have a partner with a global presence by your side, one that stands by their products. This not only applies to offsite backups but everything we do.
So in closing, thank you for your business and the faith you put in Own Web Now Corp. I am sorry that we have disappointed some of you and in sign of good faith we will eat the fees associated with the offsite backups through the month of June. Our offsite backups have been rock solid since we started them, we aim to keep that level of reliability.
Vlad Mazek, MCSE
Note: What actually happened? We use eSATA II chasis rack raid systems for backups. These systems are further replicated off to Chicago via private Gigabit network. We use Samba (SMB/CIFS) software on both the appliance and on the remote network to create replication partners. Due to the incompatibilities between the driver and Samba 3.0.10–1.4E 12.2 software our replication systems created enormous amount of traffic and requests of the NAS systems. As a result, the load on the systems spiked, made those partitions unavailable and as a result made reporting and access difficult for some customers (roughly 300 were affected). I would like to thank our partners at EMC and AhSay for their assistance with troubleshooting of the issue and my staff for putting in very long hours over the Memorial Day weeekend to get to the bottom of it.
Comments Off on Offsite Backup Apology
February 2, 2007
We’re quite happy to announce the launch of Own Web Now forums. With the launch of ExchangeDefender v3 we wanted to form a closer relationship with our partners and clients – depending on how folks take to the forums we will expand them to provide support and discussion of our other products.
If you have a suggestion for features to be included with ExchangeDefender v3 please sign up for the forums:
If it would make it easier to follow via RSS you can subscribe to it here.
January 29, 2007
Over the past week we have been working very hard at expanding our storage network capacity. The demand for our new off-site backup system has been far greater than we ever imagined so we had to grow our network significantly to accomodate all of our partners that wanted to deploy this service for their customers. Who can blame them, we use the same system to backup all of the servers we manage as well.
So what’s new? Additional network capacity and additional storage. Newer version of AhSay. Improved replication system. The amount of work that went into putting this together was significant, we have dedicated over 5 days to it.
How does it work?
Each client (workstation or server) installs a small branded agent on the PC that is being backed up. The agent runs on Windows, Linux, Mac OS X, Solaris and Novell. The agent backs data up to the server on the byte level, this way only changes within the file are propagated instead of the entire file each time there is a change. The client keeps the information on the backup schedule and retention policies (how long a file should be available for restore) and is capable of backing up Microsoft SQL Server, Microsoft Exchange (both on store level and optionally mailbox brick level down to the message or folder), Volume Shadow Copy, Oracle, MySQL, etc.
The infrastructure, frankly, is impressive. The main AhSay OBS server runs on the perimeter of our Dallas/InfoMart network and receives/sends data from the client agent. The entire server is backed up daily to a local SAN for quick disaster recovery purposes (in case of RAID failure). The entire server is also replicated in realtime to a geographically redundant data center in Chicago/CBOT using AhSay replication server. We know how important your data is so we do our best to protect it as best as we can, 3–way replication with 2 of them in sync at all times.
We’d like to thank our customers for their patience during this process. If you were on trial and experienced any issues please contact us and we will extend your backup trial by additional 30 days.
December 10, 2006
All planned and scheduled maintenance for this weekend has been completed and finalized far earlier than we anticipated. The new power switches are in place, new servers have been activated, and our worldwide global mainteanance tasks have been completed. This window has not only addressed many problems we have had over the past three months but has also expanded our capacity and platform ability to deliver further ExchangeDefender and The Office Server products over the next quarter.
Please keep in mind that even though the maintenane window is complete there still may be some sporadic performance and service issues over the next 48 hours. All our data centers are fully staffed following the maintenance cycle while we evaluate and adjust our systems to monitor the network at the new capacity.
December 8, 2006
The next 72 hours will include series of six extensive maintenance tasks spanning three USA-based data centers and two EU-based data centers.
The first and largest maintenance task involves ExchangeDefender. First we will be adding more servers to the mix. This is a very routine task for us and is expected to produce zero downtime even on the nodes that are active during the transition. After the new nodes are online and older hardware has been swapped out we will proceed to roll out new policy systems which will speed up the daily email SPAM report generation, portal access, MSP control panels and higher-end reseller branding. These processes will continue throughout the weekend and while we will not have any downtime the mail release delays could be delayed up to five minutes. If your message is affected by this delay you will be given a notice when you attempt to release the message.
The second maintenance task involves TheOfficeServer dedicated servers and shared hosted services. We are expanding our portfolio to provide APC MasterSwitch remote reboot to address the complaints regarding Microsoft Windows Server patching problems over the past few months. The APC MasterSwitch software will allow you to remotely power cycle the server through a web interface when the system becomes unresponsive. In order to provide this service we will be gradually powering down and swapping power strips. We expect the downtime related to this task to take roughly 10 minutes. Our engineers will login to your server, trigger a graceful shutdown and move the power cord to the new strip. The downtime exclusively applies to the single switched power supplies, if you have a dual power supply you will not have any downtime associated with this maintenance task.
Own Web Now hosting control panels will be going offline for a period of five minutes during our regularly scheduled maintenance window from 7AM – 8AM GMT on Saturday, December 9th, 2006. This outage will not affect any services (dns, mail, web, databases, etc) but you will not be able to make any configuration changes during this window. If you do have an emergency maintenance issue please contact Own Web Now Corp through the trouble ticket support system.
As always, all Own Web Now Corp maintenance tasks are fully scripted, practiced and load tested before being executed on the live production network. We have load tested each ExchangeDefender node at least 500 hours, each power strip has been placed to 80% load (manufacturer recommended maximum load) and held for 24 hours. We have full confidence in the hardware that is being placed into production.
ExchangeDefender has undergone significant changes in underlying infrastructure and reporting services to accommodate the global growth of both our network and our customer base. Although initially you will not see a visual difference in what we have changed you will notice a significant performance boost. This is directly related to the changes in the way we currently collect and build reporting for ExchangeDefender.
Thank you for all your support, understanding and patience. Nearly all the changes we are making to ExchangeDefender, Own Web Now Corp and The Office Server systems are directly designed by the feedback submitted by you, our partners and customers. While many of you can appreciate that the recent instabilities have been caused by Microsoft’s diminishing QA of security patches, I personally want to make you aware that we are doing everything we can to both eliminate the problems and continue to work with Microsoft to make sure these issues are minimized in the future.
At no time do we lose sight of the trust you place in us to manage your systems and we’re doing everything we can to deliver on our promise and our message. Thank you for your business!
Vlad Mazek, CEO, Own Web Now Corp.
November 27, 2006
We wanted to give all our shared hosting customers a notice that mail servers at Own Web Now Corp will be undergoing and update tonight, Tuesday, November 28, 2006. The maintenance window is scheduled for 07:00 – 08:00 GMT. This translates to 2–3AM EST, 11PM-Midnight PST, 6–7PM Syndey, Australia. This maintenance window is scheduled to occur during the off peak hours where our network experiences the least amount of traffic. Having said all that you should not experience any outages or any downtime.
- Brand new load balancing infrastructure
- Additional global replication servers for archiving
- Additional webmail options for mobile workers
- Optimization of the mobility components to allow Windows Mobile 5 users to recive similar Push Mail technology via our POP3 servers
- Upgrades to antivirus and antispam infrastructure
Two important notes: This maintenance window does not affect ExchangeDefender customers.You will not experience any downtime. Your users will not see any changes to the interfaces they are currently used to.
Update: Tuesday, November 28, 2006 – 2PM GMT – The upgrades to our hosted mail server network have been completed. No downtime our outages to report. Everything went a according to plan and performance is within expected parameters. We are continuing to monitor the changes and stand by to respond to any unforseen problems. So far there are none to report but if they are they will be announced on this blog.
« Newer Posts