In a few moments we will begin the upgrade of our network to ExchangeDefender 7. The process will be seamless to our users and we don’t expect any issues. We are prepared for the unexpected issues and here are some suggestions:
1. Sign up for the Network Operations Blog updates (http://www.ownwebnow.com/noc)
2. If you don’t like web / RSS feeds, check out our Twitter feed (http://www.twitter.com/XDNOC)
3. Make sure you have distributed the User Guides to your end users so they can navigate the new User Interface. Link to end user collateral is here.
During The Upgrade
Check here first. We will be posting updates here and to the Twitter feed as soon as we identify and document a bug.
If you see an issue that has not been addressed, open a support request at https://support.ownwebnow.com
Call us at +1 (407) 209-3276 if the issue is urgent. Note: This number will be discontinued after the launch on June 1, 2011.
May 30, 11 PM EST: All services have been upgraded to ExchangeDefender 7. All tests have confirmed that sites and services are completely operational.
On November 25th between 17:00 – 19:00 Eastern OwnWebNow will be upgrading the OS version on its Value bandwidth router. Estimated period of impact is no more than fifteen minutes. During this time, the core router will require a reboot, interrupting service momentarily
We are currently attempting to isolate a large scale DDoS on our 220.127.116.11/24 network. This ticket will be updated as soon as we have more information.
At 9:40 we experienced a power event in the Dallas DC affecting ExchangeDefender internal infrastructure. This affected performance and availability of admin.exchangedefender.com, this web site, our support portals and Exchange hosting infrastructure.
By 9:50 everything appears to be back to normal but as these are highly utilized servers the performance may be affected for another 10-15 minutes as we run our tests to make sure everything is back to normal.
We apologize for the inconvenience and will update this ticket shortly.
In preparation for the release of ExchangeDefender 5.0 we’ve installed 4 new servers to process outbound mail for ExchangeDefender clients.
This transition was seamless and shouldn’t require any work from our partners, however any clients who are using SPF records will need to add the following IP addresses
On Friday, September 4th 2009 at 10:00 AM Eastern we will be changing the IP address for our Australia Exchange server due to changes in the network layout in the data center. Before the change we will lower the TTL time on the DNS record (exchange.australia.exchangedefender.com) to minimize the chance of downtime/unavailability.
Users on the Australia server should not have to make any changes after the IP address change for service to continue working.
Update 12:10 PM Eastern: The IP change has been delayed until later today. We will update this post before the change occurs.
Thank you for joining us for the Monday network infrastructure upgrade.
It has been an interesting week at Own Web Now. Following the major hardware swaps during the Thanksgiving break we have found several new pieces of hardware that really did not live up to the standards or the tests we put them through. What can you do, when you work in IT for a living you work to swap out disappointing upgrades.
Now on to something serious. In the past two weeks we have found two critical issues that have already been addressed:
Offsite Backups – We have discontinued the use of 1.5 TB drives due to the high level of failure during production. We have also isolated a single point of failure in our offsite backup strategy with the ways we were storing our security database and our file storage. This resulted in having to move approximately 30 customers from one grid to another and restoring their backups in reverse (from replication partners to masters) which is a painstakingly slow process. Over the past week we have also worked to remove this limitation from our systems and our enterprise backup.
ExchangeDefender – Last Thursday we had several messages with extended delays during which we have discovered several important problems. First, ExchangeDefender should not be used to route inbound mail from your own network to our inbound network. Please rely on a local SMTP service because our inbound network is configured not to expect local mail appearing on an external interface. Seems like common sense but apparently some users have their SMTP servers set to route using DNS lookups instead of using local SMTP servers or connectors. ExchangeDefender inbound network is not designed to function as your SMTP server, in fact quite the opposite – mail for local domains appearing from outside the ExchangeDefender network is put through further scanning as a possible forgery. This is either a flaw or a designed security feature depending on who you ask.
This week we will be rolling out two additional Exchange 2007 networks to help address the demand for the service. We will also be provisioning a shared cluster hosting concept and expanding storage allotments for our FTP and file storage services.
We are conducting an automation out-of-band update between Midnight and 2 AM tonight, Friday, December 12th, 2008. During this time you will not be able to create, delete or modify any Exchange 2007, SharePoint 3 or Offsite Backup accounts.
We are conducting an update to Shockey Monkey to bring in multi-tenant and private branding for our Shockey Monkey customers who wish to offer the same control panels you use at OwnWebNow to your customers within your own Shockey Monkey portal. This was previously mentioned as a concept by Vlad Mazek in his private blog posting “7th Monkey of the Seventh Monkey”
Good morning and happy Monday! It’s that time of the week again, where we open up our NOC task list and sprinkle some pixy dist on the wishes that the entire network remains at 100% uptime.
Weekly maintenance was completed on Saturday without major news to report.
Issues we are currently working on:
ExchangeDefender XD Engine Updates: We are currently monitoring the new antispam engine that went online last Wednesday. So far the performance has been remarkable and false positive ratio went down as well. You may have read about a major SPAM organization bust that went down last week, knocking McColo portion of Hurricane Electric offline, and you also may have noticed that it only affected the SureSPAM counts. This is because we have had their network mapped out for a while, the parts we are really concerned and continue to work on are the SPAM counts. You should see a huge decrease in koi8/cyrillic based SPAM messages as well as the UPS/mail trojans thanks to the new AV definition update.
Web Hosting Infrastructure: Our web hosting infrastructure will undergo a major update to address stability issues encountered over the past two weeks. Between the firewall upgrade and the major storage firmware upgrade we’ve encountered an unusual conflict that at times spikes the load on the system and sites stop responding momentarily. This is unlikely something that was noticed by our user base but it should not be happening.
Outbound traffic shaping: ExchangeDefender outbound network is being upgraded to 4.0 this week so we can bring you more accessible outbound mail tracking, smart routing and policy controls including throttling. This is an internal matter that should not affect the production network at all.
Thanksgiving holiday in United States is when we make significant changes to the network, replace a lot of aging hardware and replace with new hardware. Expect the Thursday, Saturday and Sunday of (November 27, 29 and 30) to have additional latency as we make network modifications and swaps. Details of all changes will be announced on the corporate blog since it affects all users and all services. No downtime is expected, “Black Friday” will not be affected and will again have all caps removed on traffic and quotas as our show of support for the e-commerce organizations that reside on our network.
Thank you for reading and have a wonderful day.
Management Request: If you can think of anything that could improve this site please contact us.
Good morning and happy Monday! This posting is part of our effort to keep you more in tune with the network operations group at Own Web Now because you rely on our services and should be a part of the conversation.
Extended maintenance windows over the weekend have been completed successfully and have addressed numerous growing pains and performance issues. Biggest changes were applied to our Exchange 2007 network and all the systems are now performing remarkably well. We would like to thank Microsoft CSS Exchange team for all the assistance they have rendered over the past two weeks in isolating creeping and unusual issues with the product that we have never encountered before.
Issues we are currently working on:
ExchangeDefender Reports Replication – At times the replication between our master maillog servers and slave reporting servers tends to lag. Sometimes the replication lags beyond our tolerance and appears to nearly stall. We have not yet isolated the issue that is casing this problem but are monitoring it very closely and adjusting replication points.
Offsite Backup Scaling – We are preparing our offsite backup network for another product addition to the matrix. You may experience slight availability issues as the management consoles reload while changes are taking place. This process will not interfere with the execution of offsite backups because the backup agent software automatically reloads.
ExchangeDefender SPAM Load – New ExchangeDefender antispam engine went online last week and the detection has been improved dramatically. We still haven’t kicked it up all the way but expect to by Wednesday. We have had several complaints about the level of SPAM and in our investigation have found that almost all of them are related to end user stupidity – whitelisting null senders <> and whitelisting their own domain or email address. When you whitelist any address you suppress ALL SPAM detection mechanisms, so when you whitelist your own address you open yourself up to a ton of SPAM. We have a process that we are preparing that will automatically remove null sender and mirror trusts because those should never be programmed in.
Lastly, an apology to our Canadian clients. Last week we let the cat out of the bag about our Canadian offsite backup product. This site is not intended to serve as an announcement board for new products, we just want to keep you informed about what our group is working on. From time to time what we are working on may not match the business side of Own Web Now so if you have a product or business related question we are not the most authorative source to provide an answer.
Another in the series of Monday posts covering the network events at Own Web Now.
Currently there are no major problems on the deck and we have few minor going concerns related to the infrastructure. The network maintenance interval this weekend was largely uneventful.
Tuesday, November 4th: New self-service control panels for Microsoft Exchange 2007, Offsite Backups, Microsoft SharePoint. Invoices and service management will be launched at the same time.
Wednesday, November 5th: Introduction of the new ExchangeDefender Exchange 2007 hosting server.
Thursday, November 6th: Introduction of OWN Offsite Backup grid in Canada.
All prior issues and problems have been resolved.
We are continuing the “Case of Mondays” posts covering OWN infrastructure events we are working on so you can be more connected with Own Web Now. On behalf of the entire team thank you for the nice emails you’ve sent to us regarding this. As a result of such tremendous feedback will soon be opening a forum to continue the conversation throughout the week.
#1 DIY RBL: We are implementing new systems to help deal with the less and less responsive RBL department activity at major service providers. As a result of the SPAM problem many larger carriers have implemented their own SPAM RBL systems which have been proven less than reliable in managing SPAM. Due to the volume of messages sent by ExchangeDefender we tend to fall towards the top of the statistical profile for some of these providers and we notice issues even when there are none. At this point we are investigating complaints about Comcast. Hotmail and AT&T issuing random errors, though we are unable to successfully replicate the issue and their offices are either nonexistant (unattended web form) or don’t work during the weekends. We are working on this issue and hope the solution we are put in place is working.
#2 Too Much SPAM: ExchangeDefender SPAM levels are still higher than usual. Over the past two weeks we have shifted to our own honeypot system in addition to the external subscriptions we receive from others. Our Outlook 2007 addin allows you to submit SPAM, which virtually guarantees that you will never see that SPAM again. It took a lot of experimentation and management to get this into ExchangeDefender and we are still not at 100% on its deployment so you might see elevated SPAM levels though they should be dropping off significantly already. If you are tired of SPAM help us kill it, www.exchangedefender.com and click on New! Download ExchangeDefender Outlook 2007 Addin.
#3 Offsite Backup reports from Europe: there are still some inconsistencies successful email reports in our European offsite backup network as we roll out the new control panels and a new OSB product. Failed, missed and setting change reports from our Europe grid are coming through fine, but successful backup reports are still not being sent to 100% of the customer base. Solution to this issue is expected today and we are working on a workaround.
Issue with successful job Offsite Backup reports in USA has been resolved.
Issues related to Exchange 2007 back pressure has been resolved.
Issues with BT have been resolved.
We will adding an adaptive feature for outbound mail routing to address future #1 issues. This will be announced somewhere towards the end of the week and will require an adjustment to your SPF records if you use that technology. Because large ISPs do not use the same RBLs that rest of us use we have no ability to monitor them and proactively address issues. Going forward we will do manual monitoring and route mail through the IPs that do not appear to be affected by a single provider.
Tonight we will be rolling out master control panels for OWN services. This will give you full control over accounts and settings for our Microsoft hosting and Offsite backup services with centralized management. As with any software rollout there could be issues, we will be staffing extra support to address them all.
New Exchange grids coming online this week. We will be adding another Exchange 2007 cluster to the service which will require some documentation adjustments and modifications.
As a continued effort to keep you completely in the loop of all the events at Own Web Now we will start a new post series called “Case of Mondays” where we discus ongoing issues with network and any changes that are being made throughout the weekend Maintenance schedules to keep the systems running along as well as any issues that we are working on that might come up.
We are still working on British Telecom which is rejecting messages with Relaying Denied. In the meantime please create a separate SMTP connector and attempt to deliver messages to them directly. We are still working with them on daily basis to resolve the issue but their response so far has not led to a solution.
ExchangeDefender Outlook 2007 Addin is being improved towards 1.1 and fixes are being rolled up as well. Currently most issues reported have to do with 64bit Vista. Other issues are noted in the documentation (for example, do not install out of a zip file, do not install from a UNC/network share).
Shockey Monkey bugfix update is scheduled for this Saturday, 10/18/2008, with changes to the reports and email notification upgrades.
Offsite Backups reports are now functioning 100% across the client base. The issue of some reports (backup pass/fail) have been resolved by AhSay with upgrade to 5.5.1. You do not need to upgrade your agents to take advantage of this fix, the issue was on the servers not agents.
ExchangeDefender email reports issue has been fixed. There was a brief sync issue between the report servers and admin servers which resulted in delayed updates to the configuration changes. This has been addressed but we are also working on a completely different algorithm that can compensate for SQL replication issues.
Exchange 2007 backpressure problems have been solved by removing the feature on all servers.
Tuesday/Wednesday nightmares continue this week with our Microsoft network. We are expecting 11 updates this week that we will be rolling out within 24 hours of successful testing. Although none of the updates affect the server-side of the network (all are related to Internet Explorer and MBSA) we have a policy of keeping our servers up-to-date and will be applying the patches silently.
We have still not completed our evaluation of Exchange 2007 SP1 Update Rollup #4. The problems that we have reported still exist in the new code and at this point we do not see a reason to upgrade. Here are the outstanding issues we are tracking with Exchange 2007:
- iPhone sync crashes store
- Mailbox database dismounts without event log notification, service remains in running state.
- Store service hangs.
- Backpressure reports invalid data and does not respect treshhold definitions.
Generally this blog is used to document any and all activity related to something that is broken. Since things have been working quite well lately and we’re mostly on new projects and performance related tasks I wanted to update you on the status of the entire network.
ExchangeDefender – Solid. No issues, no latencies, no delays and overall flawless performance. Over the past week we have not had any delay reports that turned out to be on this end, we have had no DDoS issues to report, we have our lowest latency ever (between 5-10 seconds, network-to-network) and fewest false positives in a while.
Hosting – Solid. No issues, working on Exchange 2007 SP1 upgrades and 2003-2007 migrations as well as WSS2 to WSS3 migrations. We are working on integrating our new global points of presence in Australia and Europe and they are coming together well.
Offsite backups – Solid. No major issues, one patch and hardware enhancement scheduled for this weekend to help with large file backups.
VoIP – We are adding another IAX2 provider to our network, should become available shortly.
Training – Mostly scenario and interface tests for ExchangeDefender 4.0. Bringing on new materials, including Shockey Monkey 2.0. Web site undergoing a major update.
That is all. It’s so boring it already feels like the summer.
Over the weekend our primary VoIP partner shifted IP addresses but apparently had trouble moving their DNS as well. So, for a few hours this morning we were pointing at a proxy that had 0 channels available. We have sorted out the issue with the provider and all channels are now ringing properly. Unfortunately, there is no such thing as redundancy on the toll free number, it points to a single DID which resolves to a single provider which is very redundant but does not support ring-down to the failover provider. So while we could make calls out and all direct calls made it in, outbound calls failed.
For what it’s worth, we realize that this is getting ridiculous. Unfortunately, this one is out of our control.
The Own Web Now Corp Network Ops team has a new home on the web at www.ownwebnow.com/noc
We have been planning this page for you, our customers and partners, for quite some time but decided to rush it due to the amount of feedback we received from you. So, what can you expect from this site? Constant, constant, constant network updates on availability, maintenance, emergencies and up-to-the-minute advisories on everything we do.
I want to ask for a favor. Subscribe to the global feed if you can but since we don’t want to overwhelm you with the amount of stuff going on at Own Web Now, I would like you to separately subscribe only to the services that you actually purchase from us. I have tried to boldface the important categories to the left, I hope they prove worthwhile to you.
This site also means the pending death of Own Web Now Corp email alerts. With the exception of the most important service notices, everything will be posted here instead of via email. If you are an “email person” and don’t like RSS feeds please check out Feedblitz, which converts RSS feeds to emails! While we cannot guarantee the availability or timeliness of Feedblitz I still wanted you to be aware of it in case it makes it more convenient for you to follow this site.
So, that is all. Thank you for looking around. Thank you for relying on our services. As I said, we did have to get this out ahead of schedule so you will see us bring more and more information to this blog as well as full integration into OWN Support portals, ExchangeDefender Administrator & SP Control Panels, etc. While this will be the central storage point of all alerts, they will be syndicated to the above mentioned sites.
Have a great day!
OWN NOC Team