We are currently dealing with an exponential increase in SPAM throughout the network, mostly originating from Asia. At this point the overall utilization of the network is at 61% (daily peak) and we are looking to isolate the cause of the behavior. It appears to either be a new worm or the restart of the spambot network we noted earlier this year.
Of note, Rogers Canada has started blocking port 25 traffic on the major part of their residential network. ExchangeDefender is not supposed to be implemented on dynamic / residential services but if you have done so and are experiencing issues, please be advised that your port 25 may be blocked.
Several of our ranges have been placed on PBL blacklist although there seems to be no evidence that our servers have relayed junk mail. As of 5:21 PM EST it appears our address spaces have been removed from the RBL but we will continue to monitor the issue. So far no abuse has been reported to our network aliases and there have been no issues on the network as far as outbound SPAM is concerned.
We will keep you updated if we get any further information.
For approximately one hour between 2PM and 3PM EST our phone service was knocked out. Due to growth we ordered a block of additional DIDs (direct dial in numbers) and our VoIP IAX provider inadvertently reset all our DIDs to a different proxy server. Unfortunately, we were registering our PBX to the wrong VoIP proxy server and as a result of it, majority of our direct dialin numbers failed to connect. Among major numbers affected were our primary toll free number (877) 546-0316 and our primary local number for international calls (407) 465-6800.
We were able to quickly identify and resolve the problem but suffered approximately an hour of outage without a record of phone calls missed. We are sorry about any inconvenience this may have caused, if you missed us please keep in mind that you can reach most of us via the support portal, email, instant messaging and of course, direct sip dial if you are on VoIP as well we do accept connections at sip.ownwebnow.com
We are currently power cycling the entire web hosting cluster powered by Windows 2003. A critical security issue has been identified in one of the ASP.NET component libraries our customers rely on heavily and a reboot was required.
The system is currently being cycled and should be back in a moment.
At approximately 5:48 EST several of our large customer systems started a systematic DDoS against our outbound network. Because these are trusted systems, they are allowed access to our network without many checks. Unfortunately, these customer systems have been compromised, leading to a DDoS launch of close to 50,000 messages per second.
We have removed offending systems from our grid and mail is flowing normally. Unfortunately, outbound messages relayed during this time period of 5:48 – 6:20 EST (13:48 – 14:20 GMT) were destroyed or mangled. If you have sent messages during this time period, please resend.
We are sorry about the inconvenience and we have acted in as fast of a response as we could to what was a rather massive DDoS against our outbound network.
During early morning (2-5 AM) some customers on archive.exchangedefender.com were affected by a permission rebuilding script that failed to complete permission resets on the mail folders. The bounceback looked like this:
Diagnostic-Code: X-Postfix; maildir delivery failed: create maildir file /home/user/tmp/maildir: Permission denied
We have since fixed the issue and appologize for the inconvenience this may have caused.
We have noticed a number of offsite backup reports included weird formatting today, for backup reports generated since midnight. We are currently researching what may have caused these template problems and we will be updating this ticket when we have it resolved.
Update: 5:17 PM EST (GMT -5): We believe the issue has been resolved but are still watching it. We will be powering down the grid tomorrow and applying another patch to improve performance. We will update later.
Our VoIP provider (IAX-PSTN termination service) is performing routine maintenance on our accounts in order to port some of our numbers from third party services and telcos. We have been advised to expect outages throughout the day as their proxy servers are updated to route new numbers to us, after which we will be programming the proper DID routes in our system.
As a result, our phone systems may not be the most reliable way to reach us today, January 2nd, 2008. If you have an urgent issue, or even an issue that you would like human followup on, please post a support request at https://support.ownwebnow.com.