September 2007

Earlier today we completed an upgrade of our Offsite Backup software to AhSay 5.2.2.5 which provides new functionality, flexibility and further breakfix upgrades. This feature has been in beta testing internally for the past three weeks and has shown a lot of promise towards overcoming limitations the previous edition had. The update has been completed for our entire network, affecting all frontend, backend and replication servers.

Furthermore, this edition allows us to provide a more managed service surrounding the offsite backup suite because we have more control over the system and can proactively address issues like quota upgrades, failed backup causes and troubleshooting, etc.

These are the latest, generic clients but are not required, previous agents will work just fine.

Update: 6:25 EST; We have been working with AhSay on some performance tuning fixes for about two hours now. We have decided to shut down a part of the frontend network to optimize the network efficiency. Some of you have also complained about the display of AhSay logos on the frontpage as well as the other identifying marks that we are currently trying to remove. We expect this maintenance to take until roughly noon EST and affect all our customers.

Everyone deserves a break after a hard week of work. Last weekend was the most intense maintenance cycle for us ever and after a few all-nighters the 7/24/365 needs a bit of a break. Our offices will be closed this weekend, Friday, Saturday and Sunday, September 28, 29 and 30th and all project, maintenance and most low tier support will be suspended. Regular activity will resume on Monday, October 1st.

All urgent and high priority cases will still be handled under the promised SLA.

This notice does not apply to our enterprise-grade client base.

Nothing gets by our user base; We may have tipped our hat a little with the SQL upgrade and a test of the message previews showing up in the control panels this morning and… yes.. yes, we are bringing full message previews to the ExchangeDefender web interface this fall. This upgrade is scheduled to launch along with our AJAX interface upgrade and will give you the ability to highlight a message and click to view the body directly from the web site. You’ll then have an option to reply right there without waiting to deliver or trust the sender, or do any combination of the three.

The interface preview you may have seen overnight was just that – a preview – and will not officially become a feature until November/December timeline. We have another SQL maintenance interval scheduled for tonight and I figured I owed you an explanation for why there seems to be so much “maintenance” as of late.

4:35 AM EST Update – Network shutdown proceeded as planned. The network recovery process is beginning. Please stand by as all the services are restored. Please be aware that we will not be able to work any Urgent or High priority cases while the maintenance cycle is active. We will resume regular SLA at noon EST. Please keep an eye on the blog as we update it throughout the morning.

10:51 AM EST Update – Powergrid maintenance complete, network maintenance complete, cage “shuffles” complete. Moving on to application layer testing.

1:21 AM EST, The day after update – All services, networks, systems and operations back to normal. We will provide additional details later, we’re also lobbying for a day off

During our preparation for ExchangeDefender Administrator and Service Provider daily reports we discovered a big performance issue in our reporting engine, big enough to cause us to rewrite the affected code immediately and alter our database schema. During the new engine rollout the reporting functionality was not at 100% and not all users had their daily reports generated. If your daily reports are set to be generated between 3:30 AM EST and 5:30 AM EST, your report may not have been generated. The reporting engine optimization / bugfix did not affect intraday reports.

We are in process of slowly resending the daily reports, they will arrive today. If your users need a report and it has not been delivered to them yet please remember that you now have multiple options:

– You can resend their daily report through the control panel on demand (report sent out within 60 seconds)

– You can tell the users how to access the quarantines directly (easiest and most efficient way) via https://admin.exchangedefender.com

– You can let their SPAM czar go through their mail for them.

We are currently considering not offering SPAM digest reports to end users anymore. The amount of information we can offer via email is very limited and inefficient. We can provide far more information (and make the user far more productive) if they looked at the reports over the web and are considering generating daily and intraday reports with summary data only and a link to the web interface – if there is enough junk mail to warrant a review, the user will click and see all the email that is waiting for them. If it is not, at least we will spare the user the trouble of their Outlook hanging for 8 minutes while they try to open an email thats more than 2 Kb in size. You will of course be given an option of sending users daily/intraday email summaries or full digest reports, with the summary of course being the default. We figure this may not be the most popular decision with some users to please let us know if you have an opinion.

We’re on the eve of the 3rd Annual ConnectWise Partner Summit and I’m proud to announce that ExchangeDefender fully integrates into ConnectWise’s Managed Mail sync as well as Management Report documents. Want to get it done tonight? It takes 5–10 minutes tops, just download this document and follow the directions:

Whitepaper: ConnectWise sync with ExchangeDefender How-To

Thats right, in 5–10 minutes you can have your ConnectWise deployment syncing up with ExchangeDefender and becoming a regular resident on your Management Reports.

Enjoy the conference, enjoy Tampa and enjoy the integration we now offer into your ConnectWise deployment.

Three times a year Own Web Now Corp conducts a global network maintenance cycle. These maintenance cycles are meant to double-check the equipment, swap out aging infrastructure, improve cable management as execute a disaster recovery procedures. In plain terms, we take the network down at an announced time and work on it during off-peak hours so it doesn’t crash unannounced in the middle of the day.

Our global network maintenance cycle is scheduled for this Sunday, September 23, 2 AM – 5 AM Central (GMT -5).

All networks, all services, all customers will be affected. We will literally be shutting the NOC down and restarting it from scratch.

We will also have a minor ExchangeDefender Policy Engine upgrade during Saturday afternoon, the services should not be affected beyond perhaps a few minutes without control panels while we swap out the switches and nodes.

New ExchangeDefender Reports are out! We have worked hard to bring the new reporting functionality to you and we hope you like the new information we are providing as well as what we are allowing you to remove from the report:

Xdreport

If you pardon the eraser tool you will notice that the layout of the reports has changed slightly and you will also notice some more information showing up in an unobtrusive way. So let’s look at the improvements:

  • Summary Guide – Top right blue box “You are reading an email summary…” has been the most demanded feature by our customers and partners alike because  it gives you single-click access to the ExchangeDefender portal. Click on the link to access your settings, searchable quarantines and more.
  • Email Activity Stats – Directly above the SPAM quarantines. This new summary field is showing you the total activity during this reporting period and helps address those “we are not receiving any email” complaints that users tend to feel once ExchangeDefender comes to life. 
  • Warnings & Caveats –  Bottom of the message. There are two warnings so let’s look at them carefully. “You chose to be notified of all SPAM quarantines.” is printed when you chose to receive full/complete SPAM reports. If you have a ton of email addresses and aliases it can get annoying to scroll down rows and rows of email addresses that never receive email just to see “No messages in this quarantine” so we gave you an option to suppress empty quarantine reports. Second warning is to let the user know that these reports do not include viruses, address book attacks, NDR storms, mailbox floods, mailbombs and other causual Internet annoyances. On a daily basis we only accept and process up to 80% of the inbound mail, most of the 20% being part of multiple RBLs or confirmed SPAM content. Majority of that 20% is dismissed anyhow! 
  • Branding – Background, messages and logo are now brandable. If you are a service provider you can make this your own!

That’s all for the user facing problems! Stay tuned for the Administrator reports tomorrow!

Note: We addressed a bug in the reports that did not print a header message for the SureSPAM category. If you received a report prior to 8 AM EST on Monday, September 17th, you would have noticed your SPAM and SureSPAM bundled together.. After 8 AM EST you will see both quarantine contents broken down individually.

We have received some reports of certain users emails bouncing on receipt. We are currently looking into the problem and will update the advisory shortly.

Update: 1:19 PM EST: Problem solved, 100% of the accounts are now online.

Update: 1:53 PM EST: ExchangeDefender has been reloaded and refreshed to assure absolutely everything is working perfectly. We have taken off administrative console access offline for the moment to determine the cause of network configuration failure. As mentioned in the 1:19 PM update, everything should be working perfectly fine and there should be no bounces.

Update: 2:16 PM EST: Notified ExchangeDefender administrators, updated trouble tickets and the recovery effort to deliver 800+/20+ inbound/outbound messages continues. This was a minor (albeit catastrophic) error in the ExchangeDefender network configuration that affected a small portion of our customer base but due to the distributed nature of the system it may have affected just about everyone. As a precaution, we have temporarily taken administrative interfaces offline to determine how this happened in the first place. Again, network, performance and system are at 100% at the moment with no known issues.

Update: 2:28 PM EST: Cause of network failure identified, fixed. Moving to the testing phase, control panels are still offline. 1/4 of the bounced messages have been recovered and delivered to the end users.

Update: 3:19 PM EST: Everything is still working perfectly fine. Our team is decrypting messages from the standby spool and dropping the messages by hand into the delivery queue. All “bounced” messages will still be delivered. Thanks to the technology behind LiveArchive, we are able to cache delivery so in case of bounces, like today, we can still manually drop the message into your mail server.

Update: 4:46 PM EST: Everything is back to normal, all external bounced mail has been delivered, 100% service restored and administrative control panels are restored as well.

The following message was sent as a part of the daily and intraday reports ExchangeDefender provides to users that choose the quarantine their mail:

You may have noticed that more SPAM messages than usual have gotten to your inbox between Thursday and Saturday of last week. This was an isolated case related to our software updates which have been designed to prevent future issues.

We are seeing a change in the way threats are delivered. Spammers used to rely on small text messages and links to their sites in the past, today they are using attachments, images, PDF files and other dangerous content to get your attention. We have kept up with them and minimized your exposure but as the SPAM problem evolves and becomes more threatening we found it neccessary to both increase the size of our network and the way we process messages.

Everything should be back to normal and you should be seeing less SPAM than you ever have before. Our network and software improvements have been training for a few days and were put in full effect at roughly midnight GMT.

Thank you for your patience and we’re sorry for any inconvenience the increased amount of SPAM may have caused you. During the software upgrade we still filtered out over 99.7% of all inbound mail but with the increasing number of SPAM operations even that small 0.3% of non-filtered mail can result in a dozen or more messages that got through.

We’re seeing a significant change in the way SPAM is being designed, delivered and spread. Where in the past we could simply rely on virus scanners and RBLs the future of threats and SPAM has gone to the new level.

Over the past month we have quadrupled the size of ExchangeDefender both in physical assets and bandwidth and have rewritten major parts of the system in expectation of worse SPAM problems in the future. The transition to the new engine, new systems, new networks and new software has been tough on us and on our customers but it has prepared us all for whats coming.

On behalf of the whole team thank you for your patience with the transition and problems that came up during August. If it helps in any way, we will be refunding all the ExchangeDefender fees for August. Even though our performance was well within the SLA (service level agreement) we believe in providing excellent service and you should expect no less from us.

Sincerely,
Vladimir Mazek
CEO, Own Web Now Corp