We have received a number of support tickets inquiring about the stabiblity of our San Jose data center (MAE West) following the 5.6 magnitude earthquake last night. While the 5.6 magnitude earthquake is significant, it has posed no issues to our data center or any infrastructure located there. All our west coast (Los Angeles, San Jose and Seattle data centers) equipment is rack mounted in four-post closed racks and even a significant quake would not pose any immediate danger to any equipment inside the building.
Thank you for your concerns and your well wishes to our staff, everyone is safe and sound and the network is as well. Your sympathies are appreciated nonetheless.
We are currently waiting the delisting interval for mail1.ownwebnow.com (shared virtual hosting email) off SpamCop XBL (CBL). The delist was put in place around 10 AM EST and will be removed within the hour, so by 11 AM EST there should be no issues.
This issue only affected certain virtual hosting customers, as always, we advise all our customers to use local SMTP servers provided by their ISPs.
Several minutes after midnight on October 30th, 2007 we completed all primary burnins of new ExchangeDefender inbound systems and went live. The DNS updates can take (at times several days) to be recognized by some name servers so the impact should be negligible. The network and analytics are running at the fraction of network capacity but with constant product feature evolution comes contstant upgrade.
As mentioned earlier, the reason for the upgrade is simply to increase the brute force of the ExchangeDefender security perimeter. With the changing scope of junk mail and threatening content being sent to compromise or damage systems, we are arming up for the problems we see on the horizon.
In a few moments we will be taking inbound38 and inbound33 ExchangeDefender networks offline in order to clone them to five new grids that will be processing ExchangeDefender network. System utilization is currently at 24% and we anticipate removal of inbound33 and inbound38 to increase the load to about 30% worldwide. We anticipate the three new ExchangeDefender grids to bring the peak network loads down several percentage points immediately and more as the burnin proceeds and routing takes them into account.
We are not deploying any new features nor will there be a significant topology or configuration change to our network. As a matter of fact, you will likely not even notice it. This action is merely to address the growth in the service popularity and further expansion.
From 10:00 AM to 12:30 PM EST we upgraded the outbound.exchangedefender.com DDoS network gear to protect from the recent surge in worm distribution. During the upgrade we went one node at a time which degraded the performance a little and introduced negligible (few minutes) unexpected latency on the outbound mail flow.
This was a routine hardware task, process complete and systems are back to normal.
We are currently tracking the issue on the virtual hosting email server related to an AT&T blacklist. Sample error:
—– Transcript of session follows —– … while talking to aln-mailrelay.att.net.:
>>> MAIL From: SIZE=22519
<<< 550-220.127.116.11 blocked by ldap:ou=rblmx,dc=worldnet,dc=att,dc=net
<<< 550 Blocked for abuse. See
554 5.0.0 Service unavailable
… Deferred: Connection timed out with xyz
We have reported the issue to AT&T and are awaiting their response. According to their postmaster the response will be received within two days:
Thank you for submitting your inquiry. A response will be sent within two business days to the e-mail address you specified.
We’re sorry about the inconvenience and will update the ticket when the issue is resolved. Currently this is having a minimal impact to a single address on our virtual mail hosting network (if you absolutely must get the email through just try sending it again and it might hit a different gateway).
Update from AT&T at 4:30 PM EST:
Thank you for contacting the Postmaster/Abuse team at ATT.NET and BELLSOUTH.NET.
The mail-server IP address(es) associated with your request are NOT CURRENTLY BLOCKED from sending mail to any of our servers. You may have received this message for any of the following reasons: the IP address you submitted has already been removed due to another request; the IP address you submitted is incorrect and the actual blocked IP address was not removed; the IP address you submitted has never been blocked at ATT.NET and BELLSOUTH.NET. ATT.NET and BELLSOUTH.NET, its affiliates, and network services customers do NOT intentionally block legitimate mail in the course of our anti-spam initiatives. We regret any inconvenience this may have caused.
At approximately 6 AM EST. we started investigating an issue on a shared mail server network, with one of the storage arrays in critical condition. New hard drives have been placed in the system and the array has been rebuilt, service has been restored as of 7:10 AM EST.
We are commencing the regularly scheduled network maintenance on the ExchangeDefender admin network at 9PM EST on Saturday, October 20, 2007. We expect the maintenance cycle to complete by 5 AM on Sunday, October 21, 2007.
We will update this ticket when the maintenance has been completed.
12:44 AM EST – Hardware upgrades completed, proceeding with stress testing and load simulatious. Expect the system to respond slowly as we simulate a DDoS against our administrative infrastructure. JCD, PHE
2:22 AM EST – Completed ahead of schedule, all maintenance work completed. System notes under ticket 3emj1ba for further review. All systems normal, no latency or abnormalities during test. JCD
Maintenance cycle completed, JCD.
We have officially frozen all external development tasks until November 10th, 2007. All timelines will be adjusted two weeks from today to accommodate internal development and work on projects related exclusively to Own Web Now Corp. You should not see any new features nor the changes to the default system behavior of any of our products until November 10th, 2007.
We are currently tracking an increase in SPAM messages that contain mp3 attachments. At the moment these are being discarded along with the regular SPAM because they are just random attachments to the SPAM messages which we can sort out pretty well. If you are concerned about this we advise you to block mp3 file access using your Exchange attachment filtering system.
The mp3 attachments are not dangerous (yet!) and appear to be of a teenager pushing pharmaceuticals and stocks.
Threat Level: Low.
The Own Web Now Corp Network Ops team has a new home on the web at www.ownwebnow.com/noc
We have been planning this page for you, our customers and partners, for quite some time but decided to rush it due to the amount of feedback we received from you. So, what can you expect from this site? Constant, constant, constant network updates on availability, maintenance, emergencies and up-to-the-minute advisories on everything we do.
I want to ask for a favor. Subscribe to the global feed if you can but since we don’t want to overwhelm you with the amount of stuff going on at Own Web Now, I would like you to separately subscribe only to the services that you actually purchase from us. I have tried to boldface the important categories to the left, I hope they prove worthwhile to you.
This site also means the pending death of Own Web Now Corp email alerts. With the exception of the most important service notices, everything will be posted here instead of via email. If you are an “email person” and don’t like RSS feeds please check out Feedblitz, which converts RSS feeds to emails! While we cannot guarantee the availability or timeliness of Feedblitz I still wanted you to be aware of it in case it makes it more convenient for you to follow this site.
So, that is all. Thank you for looking around. Thank you for relying on our services. As I said, we did have to get this out ahead of schedule so you will see us bring more and more information to this blog as well as full integration into OWN Support portals, ExchangeDefender Administrator & SP Control Panels, etc. While this will be the central storage point of all alerts, they will be syndicated to the above mentioned sites.
Have a great day!
OWN NOC Team