Tue Wed Thu Fri Sat
ExchangeDefender Inbound
ExchangeDefender Outbound
ExchangeDefender Applications
Exchange 2010 Hosting
Exchange 2007 Hosting
Offsite Backups
Web Hosting
Blackberry Enterprise Server
SharePoint 2007
SharePoint 2010
August – 2010 – ExchangeDefender Network Operations

August 31, 2010

Scrooge database move

Filed under: Exchange Hosting — travis @ 5:32 pm

Tonight we will be completing the migration efforts from last week. At 9 PM Eastern we will be dismounting the mailbox databases and move them to the new RAID array on the new server. The move is expected to take about 3 hours to complete.

Update 9:30 PM Eastern: We are beginning the database move on Scrooge. Service is expected to be restored by 12:30 AM Eastern.

Update 1:40 AM Eastern: The database move is still in progress. Currently there is no ETA on a restore of service, however service will be restored before the start of business.

August 30, 2010

DEWEY SMTP Change

Filed under: Exchange Hosting — travis @ 2:30 pm

Last Friday we installed a new HUB server to the DEWEY network to alleviate pressure noticed by users. Unfortunately, the quickest and largest yielding result we could perform was to kill the transport services on DEWEY’s main hub, and offload it to a SMTP array of servers.

Because of the changes required, users can no longer submit messages via SMTP to 65.99.255.12 and must change their outgoing SMTP server to hub.dewey.exchangedefender.com

This change should only affect users connecting to DEWEY through non MAPI or non RPC/HTTPs connections, such as Mac clients, IMAP4 and POP3 users.

August 27, 2010

DEWEY Upgrades

Filed under: Uncategorized — travis @ 4:02 pm

Tonight starting at 8:30 PM Eastern we will begin upgrading the DEWEY Exchange 2007 network by installing additional domain controllers and additional hub transport servers. During the upgrades, we will have to reboot the mailbox and primary DC for DEWEY. Users may experience brief periods of disconnection from the Exchange server during the upgrades.

Update 8:35 PM Eastern: We are about to reboot the primary mailbox server for DEWEY. Connectivity is expected to be degraded for up to 20 minutes.

August 23, 2010

Continuing HUEY/Scrooge upgrades

Filed under: Uncategorized — travis @ 9:59 pm

We are beginning the process of continuing the upgrades to HUEY and Scrooge. We are beginning again with HUEY and will move on to Scrooge. Just as yesterday, access to mailboxes through Exchange will be interrupted until the upgrade is complete. Each server is scheduled to take about 4 hours.

Ref: http://www.ownwebnow.com/noc/2010/08/23/huey-and-scrooge-upgrades/

Update 1:37 AM Eastern: The imaging of the OS on HUEY was successful. We are in the process of moving the database and beginning work on Scrooge

Update 3:18 AM Eastern: The DB copy on HUEY is still in progress and is about half way completed. Since the information is being written to a new RAID volume, the copy process must write to all disks in the array. Scrooge has about 35 minutes on the OS image.

Update 4:42 AM Eastern: The DB copy has begun on HUEY and windows estimates 2.5 hours for completion. We are about to begin the image of Scrooge

Update 5:00 AM Eastern: The DB copy completed on HUEY and service has been restored.  Scrooge is committing the backup image to the new server and should be ready for the database copy in 30 minutes.

Update 7:44 AM Eastern: All upgrades were a stunning success. The upgrade to HUEY is complete and the databases were moved. Scrooge is on the new hardware, however the databases copy was taking longer than our allowed threshold. We’ve moved the old database drives to the new server and will complete the DB move either tonight or over the weekend.

Service is expected to be operational by 8AM Eastern, but may be slightly reduced until 8:30 as both servers process pending mail.

Update 8:08 AM Eastern: As a precautionary action we are running a chkdsk on the Scrooge DB drives to ensure they will be able to serve clients without issue. The scan is already 30 minutes in and should be completing by 9AM.

Update 8:55 AM Eastern: We are going to let the chkdsk run until 9:30 AM Eastern before we look into switching to the backup drive. We appreciate our partners patience as this upgrade will provide clients with a smoother Exchange experience.

Update 9:25 AM Eastern: Running a quick reboot in HUEY to correct the system time clock

Update 9:32 AM Eastern: The reboot on HUEY completed and the information store is coming online. If any clients experience mail “disappearing” after it shows up as new, please check the mail from Yesterday  in Outlook as the time on the server was one day behind.

Update 10:00 AM Eastern: The chkdsk on the drive is on the final stage, currently at 30%. The server should be 100% operational by 10:30 AM. Canceling the chkdsk was not optional as the other drive presented the same issue as the original.

Update 10:22 AM Eastern: The chkdsk is currently at 50%. This changes the final time to a more realistic 11AM. Customers needing instant access to their mail should utilize LiveArchive until the upgrades are complete. The upgrades were not scheduled to last past 9AM, however we needed to complete the physical migration today.

Update 10:58 AM Eastern: chkdsk is currently at 80% and looks like the slow down was due to a couple logical corrupted free-space allocation blocks.

Update 11:50 AM Eastern: The physical upgrade is complete and service is 100% operational.

DEWEY users experiencing issues

Filed under: Exchange Hosting — admin @ 6:04 pm

We received a couple reports of inhibited mail flow on the DEWEY server, with some clients reporting that they were disconnected from the server. Upon investigation, the transport service had a poisoned message in queue that locked up mail flow. After removing the message, clients reported they were able to connect.

The mail flow issues occurred between 5:30 PM Eastern and 6:00 PM Eastern

HUEY and Scrooge Upgrades

Filed under: Exchange Hosting — travis @ 12:37 pm

On Sunday August 22nd we will be migrating the HUEY and SCOOGE Exchange 2007 Servers to new Dell servers. During the upgrade clients will not be able to access their mailboxes via outlook, active sync or OWA. Customers needing constant access to their mailboxes should become familiar with using LiveArchive before the outage occurs.

Maintenance is scheduled to begin at 5PM Eastern and is likely to continue until the early AM on Monday. Service will be restored prior to 9AM on Monday August 23rd.

Update 12:35 AM Eastern Monday: We are beginning the upgrade process on HUEY tonight. Access will be unavailable to users on the HUEY server during the upgrades.

Update 7:54 AM Eastern Monday: The upgrade process took significantly longer than expected and was terminated to meet the 9AM deadline. The installation will continue today at 830pm Eastern

August 17, 2010

DEWEY Performance Issues

Filed under: Exchange Hosting — travis @ 3:46 pm

The DEWEY Exchange 2007 server was reported to have delays in processing requests from clients. We are restarting the information store to resolve the issue.

Update 1:00 PM: The information store has finished restarting and we’ve confirmed access to mailboxes on each DB.

Update 3:46 PM: Multiple parties have reported issues in delivery speed. We are rebooting the DEWEY server in order to clear out the cached memory.

Update 4:08 PM: We’ve been monitoring the server reboot via KVM. The server is in the process of installing windows updates before it completes the reboot.

Update 4:21 PM: The installation of the windows update has completed and the server is now stopping the Exchange services. The server should complete the reboot in the next 15 minutes.

Update 4:49 PM: The reboot has completed. We’ve verified that mail is being routed and confirmed that our test users could login.

Update 5:30 PM: After the reboot, the DEWEY transport started routing mail through one of it’s alias IPs instead of the primary IP, causing outgoing messages to be rejected as “Relaying denied, proper authentication required” The issue has been resolved and mail is routing out.

August 15, 2010

Huey and Scrooge Exchange Upgrades

Filed under: Uncategorized — travis @ 12:52 am

On Sunday August 22nd we will be migrating the HUEY and SCOOGE Exchange 2007 Servers to new Dell servers. During the upgrade clients will not be able to access their mailboxes via outlook, active sync or OWA. Customers needing constant access to their mailboxes should become familiar with using LiveArchive before the outage occurs.

Maintenance is scheduled to begin at 5PM Eastern and is likely to continue until the early AM on Monday. Service will be restored prior to 9AM on Monday August 23rd.

Update 12:35 AM Eastern Monday: We are beginning the upgrade process on HUEY tonight. Access will be unavailable to users on the HUEY server during the upgrades.

August 13, 2010

OWN Network

Filed under: Uncategorized — travis @ 4:18 pm

Around 4:02 PM Eastern we started receiving alerts from our monitoring system about packet loss on the network. Upon investigation, we found an IP that was flooding with fragmented packets..the ICS disabled the talker, but the backlog of traffic caused some unavailability to clients. 

DEWEY Exchange 2007 Issue

Filed under: Exchange Hosting — travis @ 11:25 am

The outage looks like an LDAP query was null routing, but not dying and eating up resources — We killed the query and was able to establish connection to our test account. Users should be able to reconnect in the next few minutes as the databases reconnect. The outage looks to have started around 11:10 AM and was resolved by 11:20 AM.

August 12, 2010

Livearchive service

Filed under: ExchangeDefender — travis @ 8:52 am

Access to livearchive is currently deprecated. Yesterday we noticed mail submission would randomly drop, so we increased our monitoring. Today, we’re noticing the same behavior, but it’s pointing more to the mailbox databases. We are currently taking multiple databases off at a time and running integrity checks.

We highly apologize for the inconvenience and any partner who has a down server that needs access to livearchive, please open a case with support and we will work out an alternative solution.

August 11, 2010

Random relaying denied on outbound.exchangedefender.com

Filed under: ExchangeDefender — travis @ 11:42 am

We received alerts from our monitoring software that messages on outbound1 were being rejected as relaying denied. We killed the sendmail process and resolved the issue. The bounce backs occurred between 11:20 AM Eastern and 11:35 AM Eastern.

Update 6:44 PM Eastern: The issue reappeared around 6:20 PM and began rejecting messages. We located the root cause, which was a faulty update from an AV vendor. We applied the previous definitions and await an update from the vendor.

Livearchive access

Filed under: Uncategorized — travis @ 11:13 am

Access to livearchive will be temporarily unavailable for some users as we dismount databases to make schema level changes for our new monitoring. Service is expected to be fully operational by noon Eastern.

Update 12:00 PM Eastern: Service to livearchive is back to normal operation

Update 1:54 PM Eastern: We’re receiving reports that some clients cannot access livearchive. We are actively investigating the issue.

Update 2:48 PM Eastern: As part of our investigation, we have to disable mail routing to livearchive. Customers should expect to see a gap in mail in their livearchive mailboxes.

Update 3:09 PM Eastern: We are installing more memory modules into the primary mbox server for livearchive. There is no ETA as of yet.

August 5, 2010

Exchange 2007 DEWEY

Filed under: Exchange Hosting — travis @ 2:07 pm

Our Exchange 2007 network DEWEY is currently registering as offline in our monitoring software. We are in the process of investigating this issue now.

Update 2:09 PM Eastern: The issue is related to one of the core networks used by DEWEY. We detected and thwarted a DDoS attack. Service is back to 100% operation.

Powered by WordPress