Mon Tue Wed Thu Fri
ExchangeDefender Inbound
ExchangeDefender Outbound
ExchangeDefender Apps Web
Exchange 2010 Hosting
Exchange 2007 Hosting
Offsite Backups
Web Hosting
Blackberry Enterprise Server
SharePoint 2007
SharePoint 2010
Exchange 2013 Hosting
SharePoint 2013
Virtual Servers
Virtual Hosting – ExchangeDefender Network Operations

July 24, 2011

Upgrading to PHP 5.3

Filed under: Virtual Hosting — admin @ 9:29 pm

At 10 PM EST tonight we will begin the upgrade of our network to PHP 5.3. This security upgrade is required for many modern PHP applications such as Wordpress, phpMySQL and Joomla. In order to be able to provide the latest applications and the security patches they require, PHP upgrade is required.

We will begin the upgrade process at 10 PM EST tonight and expect it to take roughly an hour. Between 10PM and 11 PM EST there may be intermittent outages on our web servers while the upgrades are being performed and services restarted.

We will upgrade this NOC post once all the work has been completed.

Update: 10:45 PM EST – Upgrade to PHP 5.3 has failed. We will attempt again shortly and update this advisory.


January 27, 2011

MAIL1

Filed under: Virtual Hosting — travis @ 4:49 pm

We are receiving reports from partners on our POP3 server, MAIL1 about incoming messages being rejected as “Proper Authentication Required”. This appears to be an issue with the configuration of MAIL1 and our engineers are currently working to diagnose the issue.

Update 3:50 PM Eastern: The issue has been resolved and was related to a routing file for configuration.


May 19, 2010

Mail1 HTTP Unavailable

Filed under: Virtual Hosting — travis @ 10:07 am

We are currently addressing an issue where the webmail site for mail1.ownwebnow.com (Webhosting mail server). We do not anticipate the outage lasting more than 20 minutes.

Update 11:23 AM Eastern: We’ve detected a hardware issue on mail1. In the interest of uptime during peek hours for our customers, we’ve decided to hold off on fully fixing the issue until tonight. Throughout the day, customers may experience brief periods of inaccessibility as the server soft-recovers itself.


September 16, 2009

Mail1 Delays

Filed under: Virtual Hosting — travis @ 12:38 pm

Our shared POP3 mailbox server mail1 is currently under an extreme load and slowly processing messages. We are working to alleviate the queue load and restore mail routing.

Update 1:11 PM: The load on the mail1 server has been reduced and new mail is being processed successfully. We anticipate issues for some users connecting as all users are concurrently connecting to download mail.


March 13, 2009

Issues with www1

Filed under: Virtual Hosting — admin @ 10:42 am

We are currently addressing a hardware issue on the Linux hosting cluster, affecting web, ftp and sql services. We will update the ticket when significant progress has been made, at this time we do not have an ETA as the file systems are being checked for errors.

Update: 6 PM EST: At this point over 90% of the service has been restored. We are moving on to more complex, database driven systems at this point.

Update: Saturday 8 PM EST: At this point all sites, even complex SQL driven ones, are online. Service upgraded to new versions of SQL and PHP and a more robust flexible storage array.


March 10, 2009

mail1 issues with DSBL resolved

Filed under: Virtual Hosting — admin @ 2:58 pm

Earlier today we encountered several tempfail mail delay scenarios due to a piece of software relying on DSBL realtime blacklist that was no longer actively maintained. We have taken action to remove DSBL from our web hosting mail servers which has resolved any delays and is currently processing the mail backlog.

Note: The issue only applied to mail1.ownwebnow.com. It did not affect any other services, such as Exchange hosting or ExchangeDefender. The issue was resolved and there should be no delays on the network at this time.


January 15, 2009

POP3 service for Web Hosting Clients

Filed under: Virtual Hosting — admin @ 11:20 am

We are currently investigating an issue on one of our pop3 hosting servers. It appears to be a complex issue affecting a number of accounts. We believe we are near bringing this component back to full functionality.

Symptom: Cluster is up and responsive but certain users are not receiving mail. (related to a corrupted journal on one of the SANs)

11 AM EST: Service restored. We are still researching the issue and will provide an update after the weekly maintenance this Saturday. This appears to have been caused by a DDoS that did not get properly picked up.


November 20, 2008

Security patch maintenance for mail1.ownwebnow.com

Filed under: Virtual Hosting — admin @ 5:49 pm

We are currently conducting an out-of-band security patch on mail1.ownwebnow.com network. We regret to make this modification during business hours but the scope of security issue was wide enough to warrant immediate action.

Service has been restored to 100% at 4:48 PM EST.


June 22, 2008

Maintenance on the Virtual and Web hosting Windows Network

Filed under: Virtual Hosting, Virtual Servers — admin @ 11:10 pm

We are conducting some routine maintenance on our Windows portion of the Virtual Hosting and Web Hosting network. We are applying patches, installing new hardware, general system maintenance task. All systems should be affected by a brief outage, and will be back within an hour at most.

Update: Maintenance cycle completed.


May 8, 2008

Tracking issue on shared mail server mail1

Filed under: Virtual Hosting — admin @ 1:13 pm

We are currently tracking accessibility problems on mail1.ownwebnow.com. Please stand by while we research the issue, the server appears up but several customers are reporting access issues, we are trying to resolve them right now.

We will update this site as soon as we have more information. The cluster is currently undergoing a reboot.

Update: All issues have been resolved, a scheduler service hung on the load balancer.


April 23, 2008

Investigating problems with roadrunner network

Filed under: ExchangeDefender, Virtual Hosting — admin @ 1:40 pm

We are currently working with RoadRunner (formerly Time Warner, AOL) service provider in United States, they are experiencing issues with their SMTP servers and randomly rejecting SMTP traffic. Currently mail is flowing through but some is bouncing back from them due to a reason they are still trying to narrow down. We will update when we have further information or a resolution.

This issue affects our entire global network, and some external sites we have tested.

Update: 6:34 PM EST: Even though we have not been officially updated, the problems with RoadRunner appear to have been resolved. 


March 5, 2008

Shared mail DDoS

Filed under: Virtual Hosting — admin @ 5:58 pm

From approximately 10 AM – 2 PM EST, our shared mail hosting platform (mail1.ownwebnow.com) suffered a large scale distributed denial of service attack (DDoS). Everything is under full control now and we have been able to filter out the offending systems. Unfortunately, there is little that can be done in terms of scale and protection against a DDoS as we already have both Cisco and Tipping Point in place.

DDoS attacks tend to be flared up by the regular user activity, as the system slows down end users keep clicking Send/Receive and effectively flood the connection until it times out. Systems are back to normal and messages are starting to arrive in regular sequence.


February 19, 2008

Virtual Mail server in maintenance mode

Filed under: Virtual Hosting — admin @ 7:51 pm

Our virtual hosting network (mail1.ownwebnow.com) is experiencing performance issues at the moment. We are troubleshooting the problem but a performance profile has been unusually low since about 4 PM EST (GMT -5).

We will update this item as soon as the issue has been identified. In the meantime, we have added some extra horsepower to the cluster to keep it moving.

The issue is classified as critical due to the potential mail loss that can occur when the system loads across the cluster spike beyond a certain level and shut down the inbound SMTP channel. We do not expect this to happen but are working under the assumption that the further performance degredadation will shut the network down.


February 7, 2008

mail1.ownwebnow.com mailbomb

Filed under: Virtual Hosting — admin @ 6:30 pm

We are working on isolating and eliminating the mailbomb that is currently targeting mail1.ownwebnow.com customers. Mail deliveries are currently delayed but we expect to have this issue filtered out shortly.


January 16, 2008

Cycling the Windows Web Hosting Cluster

Filed under: Virtual Hosting — admin @ 10:12 am

We are currently power cycling the entire web hosting cluster powered by Windows 2003. A critical security issue has been identified in one of the ASP.NET component libraries our customers rely on heavily and a reboot was required.

The system is currently being cycled and should be back in a moment.


October 30, 2007

mail1.ownwebnow.com RBL processing

Filed under: Virtual Hosting — admin @ 10:47 am

We are currently waiting the delisting interval for mail1.ownwebnow.com (shared virtual hosting email) off SpamCop XBL (CBL). The delist was put in place around 10 AM EST and will be removed within the hour, so by 11 AM EST there should be no issues.

This issue only affected certain virtual hosting customers, as always, we advise all our customers to use local SMTP servers provided by their ISPs.


October 22, 2007

Tracking AT&T Blacklist Removal

Filed under: Virtual Hosting — admin @ 2:53 pm

We are currently tracking the issue on the virtual hosting email server related to an AT&T blacklist. Sample error:

—– Transcript of session follows —– … while talking to aln-mailrelay.att.net.:
>>> MAIL From: SIZE=22519
<<< 550-65.99.192.3 blocked by ldap:ou=rblmx,dc=worldnet,dc=att,dc=net
<<< 550 Blocked for abuse. See
http://www.att.net/general-info/rblinquiry.html
554 5.0.0 Service unavailable
… Deferred: Connection timed out with xyz

We have reported the issue to AT&T and are awaiting their response. According to their postmaster the response will be received within two days:

Thank you for submitting your inquiry. A response will be sent within two business days to the e-mail address you specified.

We’re sorry about the inconvenience and will update the ticket when the issue is resolved. Currently this is having a minimal impact to a single address on our virtual mail hosting network (if you absolutely must get the email through just try sending it again and it might hit a different gateway).

Update from AT&T at 4:30 PM EST:

Thank you for contacting the Postmaster/Abuse team at ATT.NET and BELLSOUTH.NET.

The mail-server IP address(es) associated with your request are NOT CURRENTLY BLOCKED from sending mail to any of our servers.  You may have received this message for any of the following reasons: the IP address you submitted has already been removed due to another request; the IP address you submitted is incorrect and the actual blocked IP address was not removed; the IP address you submitted has never been blocked at ATT.NET and BELLSOUTH.NET.  ATT.NET and BELLSOUTH.NET, its affiliates, and network services customers do NOT intentionally block legitimate mail in the course of our anti-spam initiatives.  We regret any inconvenience this may have caused.


Shared Mail Server Issues

Filed under: Virtual Hosting — admin @ 7:24 am

At approximately 6 AM EST. we started investigating an issue on a shared mail server network, with one of the storage arrays in critical condition. New hard drives have been placed in the system and the array has been rebuilt, service has been restored as of 7:10 AM EST.


Powered by WordPress