PBX Upgrade
You may have had a difficult time reaching us on Wednesday 6/27 and Thursday 6/28.
I have mistakenly redirected my DID to an unmonitored extension at the same time that the PBX was undergoing a software upgrade to Trixbox. This resulted to all incoming calls being routed to an unmonitored extension that didn’t even have a greeting assigned to it. I have been able to track down the voicemails and will be returning calls tomorrow on Friday, 6/29.
All other support mediums (email, web, portal) were working during this time, we did not become aware of the problem until one of our partners let me know. We’re really sorry about the inconvenience this has caused some 40 of our callers, we will match up the caller ID with the customer database and contact you all as soon as possible.
DNS and Time Infrastructure Overhaul
As our network grows even the most optimized of services need to scale. While its unlikely that you may have noticed an issue with DNS services, we have decided to both increase its capacity and reduce the scope of that service. We have also added the ability for you to sync with reliable internal time servers. Both modifications are nearing completion but you can take advantage of them right now as they prepare us for future growth.
DNS Modifications
Going forward our DNS servers will only answer authorative requests for the external network (ie, Internet) and full answers including caching will be provided to internal servers (ie, hosted networks, ExchangeDefender, colocation customers, infrastructure partners). More specifically, we will not provide “recursive lookups” for external users and will only answer authorative requests from the Internet.
Background: DNS servers resolve friendly hostnames such as www.ownwebnow.com into IP addresses such as 65.99.192.50. The DNS server, in our case ns1.ownwebnow.com is said to be authorative for a zone (in our case ownwebnow.com) if it is the official provider of the information that matches the hostname to the IP address. When you use a DNS registry such as Network Solutions to register your domain, you enter a set of name servers (ns1.ownwebnow.com and ns2.ownwebnow.com) which will provide resolution, or be authorative, for that domain. Clients, including remote networks, computers, servers and more use their own DNS servers to resolve hostnames into IP addresses so computers can locate one another over the Internet. When a remote server requests a lookup from their local server the local server checks if its authorative for the domain (ownwebnow.com) and if it is not authorative it starts the recursion process – it first looks at its root hints to find the top level domain (.com) and eventually receives an answer from the authorative server (ns1.ownwebnow.com) which it sends back to the client. By disabling recursion on our name servers we stand to reduce the load and increase performance on our network because we will only be providing the DNS service to our customers, not everyone on the Internet.
Time Server Modifications
As of late there have been many issues with the public pool of NTP servers that help computers and networks around the world syncronize their clocks. To make matters worse, there are many issues with virtual machines and the horrible drift (difference between real time and time in the virtual machine) in time thats introduced with new technologies.
If you are Internal to the Own Web Now network you can use time.ownwebnow.com as your time server. It should (and so far statistically it has) answer the time syncronization requests 100% of the time. Our previous time.ownwebnow.com was a round-robbin implementation that simply aliased time.ownwebnow.com to the various military and research organizations that had public time servers. Over time, that infrastructure has become less and less reliable so we’re providing the time sync for you if you’re on our network. Just use time.ownwebnow.com and you’re all set.
That is all for now, we expect all time and DNS related work to be complete by July 15th but you are welcome to use them now to improve your performance. This will be a very seamless and transparent implementation for our entire user base but we wanted you to be aware of what we’re doing to keep up. As always, thank you for your business.
ExchangeDefender gets tougher on NDR and Backscatter
Over the past year we have seen a steady increase in NDR traffic. We’ve done something about it previously but have since gotten far more aggressive on it to the point that virtually every fake bounce will be automatically quarantined.
It’s important to understand the motivation behind the spoofing and massive NDRs they produce. There are two ways in which spammers abuse the NDR system: one is to steal identity and the other is to diminish the confidence in the SPAM filtering solution. The first is quite easy, they want to use a legitimate sender address so that the remote servers will accept the mail. To combat this you can easilly enable SPF/SenderID on your domain and never worry about it. The second is a little more involved/contrived and involves systematically taking apart the ability of the “installed” SPAM filtering solution to adequately sort out mail. Most installed SPAM filtering solutions (the ones you install on your server) and appliances alike (that are devices on your network) build reputation models based on how often legitimate mail comes from certain addresses and IP blocks. They also build local bayesian databases that index known SPAM and non-SPAM; As such, by flooding the server with mail from all over the place those databases the reputation scores become increasingly less reliable – a process more commonly known as poisoning.
So what are we doing and how does it benefit you? Assuming you are using our outbound servers to relay messages, your messages will contain special tracking that will match up what we have in our internal databases. If an NDR is received with that tracking in tact, the message is allowed through. If the NDR is received without that tracking that means that the message didn’t come from you, from your server, that it was spoofed – and it adequately goes into the SPAM quarantine where you’ll likely let it die.
Offsite Backup Troubleshooting
As blogged here previously, our offiste backup maintenance is complete. Not only do we have the storage to sustain the growth for the forseeable future but things are running near flawlessly.
If you are still experiencing issues please open up a trouble ticket and we will help you get to the bottom of it. One problem some customers have reported is that backups do not seem to work automatically, namely, you receive an email saying “Reminder: Scheduled backup missed > Username > Backup set name”
If you receive this error please try following these steps (click) as they are the first thing we will try to do once the ticket is opened.
If you receive any other error, please open up a trouble ticket in our support portal.
We have seen a remarkable improvement in performance for backups and restores since the maintenance interval, and will shortly be announcing some additional services related to the offline backup service, mostly around the management, support and DR crisis management.
Offsite Backup Expansion Complete
The previously mentioned offsite backup work and SAN maintenance has been completed. All systems should be back to normal and performance should be back to rock solid. We will be doing some final stress testing scheduled for tonight, along with a few DR scenarios so its likely that tonight is the last night your backups get interrupted. If you do start experiencing problems with your backups on Tuesday please open up a trouble ticket and we will do our best to get to the bottom of the issues.
Thank you for your patience and sorry about the inconvenience. As mentioned before, we are crediting the month of service to all customers regardless of whether you were affected by the performance issues or not.
Growing the SAN Infrastructure
The backup SAN infrastructure scaling work has officially started, things are moving along very well. Unfortunately, the data copy and sync process is very time consuming and we expect it to take a better part of the day. We expect this move to drastically improve the performance and reliability of the offsite backup service and allow us to stack additional features that you have been asking for since we started offering this service.
We will post an update here when the upgrades have been completed.
Big Changes at Own Web Now
Dear Clients and Partners,
I wanted to take a moment to write to you about the changes that will be happening in our company in the very near future.
I first want to thank you for your business and your loyalty over the years. If you’ve been with us for a while you know of our dedication to service and delivering the services that make sense. As we embark on the next growth stage of our company, I want you to know that the same commitment to service and offerings that make sense will stay at the core of who we are and what we do.
Over the course of next 30 days we will be bringing online more ways to obtain support as well as an easier way to track, escalate and manage IT support for your business. Following that we will be publishing several training and troubleshooting systems to get you up to speed and let you take the advantage of all the services we offer. Finally, we intend to become more aggressive in our release schedule as the new Microsoft platforms launch, bringing you more applications and services that you can use without a large up-front commitment or long contracts.
What is really changing, at the core, is that until now we have been an infrastructure building company. The work that has gone into Own Web Now Corp since Jan of 2006 has been to transform it to the solution delivery organization. In plain terms that means: we will no longer only build the networks, we will focus on helping you take advantage of them.
Literally everything at Own Web Now is changing, expect more communication from me over the next month detailing all that we’ve been working on for the past 15 months. You’ll love what we have built, and we look forward to continuing down the path of offering solutions that make sense.
Sincerely,
Vlad Mazek
CEO, Own Web Now Corp
Backup Systems Maintenance
We have scheduled an emergency maintenance interval on our Offsite Backup System network for Wednesday, June 6th, 2007. We are adding new hardware, upgrading some of the software and replication systems, adding more equipment to the mix and extending the reporting functionality to offer more reporting and branding functionality to you and your clients.
The backup network will be unreachable to the public between 10 PM EST – 11 PM EST.
Furthermore, we will be moving certain users to a more scalable network storage system. Because this change affects a very small group of users and involves a more extended downtime, you will be notified directly with some contingency planning and preparation as we intend to make the move in off-peak hours during the weekend when backup activity and change management is low. While you will have the ability to retrieve data during this interval, you will not be able to store more data to it.
Thank you for your patience. Backups and proper disaster recovery planning are very important to us and we realize that the offsite component is the most important one, after all, it is the one you use for compliance and in case everything else disappears, so we’re doing all we can to make sure that data is there when you need it.
Offsite Backup Apology
Dear Partners,
I wanted to address an issue with our offsite backups that started last week with the rollout of new storage arrays used for Own Web Now offsite backups. As you already know, we take great pride in providing rock solid solutions that scale worldwide. We put in countless hours in design, testing, analysis and performance tuning on all services and you can expect the OWN services to be rock solid. That in fact is what you and your customers pay for.
Last week one of the new storage arrays we have added to the mix started to cause issues with the server code that actually maps the logical partitions and makes them available for access and replication to our backbone, offsite centers and your clients. Offsite backups are a partner service only, we do not offer it in direct retail. You pay what it costs us to provide the same redundant layer of data protection we expect and maintain for ourselves. I know that if I lose a backup, your customer will be out of business, you and I will be in the court. That is why I insist on the highest grade quality when it comes to protecting data. That is also why we give it away to our partners at cost – I want you to be the hero for that business that just lost everything and made it back thanks to you.
So when that promise is not delivered on, believe me, I am the most disappointed one. For close to our week a small portion of our clients had to put up with unreliable backups, with backup jobs not completing. Even though you are paying at cost and there is no SLA to speak of at all, I want you to be aware of how dedicated we are to our partnership. We will waive all the June monthly fees as a result of the issues some of you faced. Thats right, even if you were not impacted you will be getting a free month. We understand the enormity of what you put in our hands, we want you to know that you have a partner with a global presence by your side, one that stands by their products. This not only applies to offsite backups but everything we do.
So in closing, thank you for your business and the faith you put in Own Web Now Corp. I am sorry that we have disappointed some of you and in sign of good faith we will eat the fees associated with the offsite backups through the month of June. Our offsite backups have been rock solid since we started them, we aim to keep that level of reliability.
Thank you.
Sincerely,
Vlad Mazek, MCSE
Note: What actually happened? We use eSATA II chasis rack raid systems for backups. These systems are further replicated off to Chicago via private Gigabit network. We use Samba (SMB/CIFS) software on both the appliance and on the remote network to create replication partners. Due to the incompatibilities between the driver and Samba 3.0.10–1.4E 12.2 software our replication systems created enormous amount of traffic and requests of the NAS systems. As a result, the load on the systems spiked, made those partitions unavailable and as a result made reporting and access difficult for some customers (roughly 300 were affected). I would like to thank our partners at EMC and AhSay for their assistance with troubleshooting of the issue and my staff for putting in very long hours over the Memorial Day weeekend to get to the bottom of it.
Product Demo Webcast
Vlad Mazek, CEO of Own Web Now Corp, is presenting to two small business user groups tonight on a variety of technology and business subjects Own Web Now Corp is involved in. If you’d like to attend the access is free, details below:
Tonight’s presentation is free, live over the web and starts at 7:30 PM EST. It ends at approximately midnight EST. Here is the preliminary schedule.
7:30 – 8:30 EST: Exchange 2007 Overview (Level 100)
8:30 – 9:00 EST: ExchangeDefender, Shockey Monkey (Level 0)
9:30 – Midnight: Virtualized Services (Level 200), Shockey Monkey, ExchangeDefender, and other tools we make (and why) that IT solution providers are using to make more money. These will be product demo’s, not sales pitches.
Is there anything else you’d like to hear? Since we’re pretty much inviting SBSers worldwide, is there something you’d like to see? See you tonight, livemeeting info below:
Meeting URL: Livemeeting Address
Meeting ID: D7QS3H
Meeting Key: bn6′%~P