We are currently performing a hardware upgrade to Shockey Monkey infrastructure in order to prepare for Monkey Remote beta launch this Friday. While we do not expect services to be interrupted to all users, please accept our sincerest apologies for any inconvenience this may cause you as we roll the upgrades across the SQL backend.
November – 2010 – ExchangeDefender Network Operations
November 29, 2010ExchangeDefender DDoSOur ExchangeDefender network on 65.99.255.0/24 network sustained an extended DDoS attack since approximately 7 AM EST. While the attack was addressed nearly immediately, due to the failover routing in ExchangeDefender, this network was removed from the scanning grid and deactivated. At this point, we are reactivating it and all traffic patterns should be back to normal. Users should not have have been able to notice any interruption as the network is load balanced across other data centers. There may have been some limited delays in email delivery for any messages that were in-scanning when the DDoS started.
Comments Off
November 24, 2010Maintenance LOUIE Dec 3 and 4To fully resolve the OWA Search issues on LOUIEMBOX1, we will have to reinstall Exchange on MBOX1. To minimize the impact on clients the following schedule has been established: Friday December 3rd: 11:00 PM Eastern: Blackberry Enterprise Server will be shutdown and will remain off until the repairs are complete. 11:15 PM Eastern: Mailbox Databases will be moved from MBOX1 to MBOX2. Due to the state of MBOX1, we cannot utilize the DAG for the cutover, so during the DB moves users will be unable to access their mailboxes.
Saturday December 4th: *Estimated* 2:00 AM Eastern: DB1 should be moved over and access to the mailboxes for users on DB1 will be restored. We will then start the move on DB2 *Estimated* 4:00 AM Eastern: DB2 should be moved and access to all user mailboxes should be restored. 4:30 AM Eastern: The HUB role will be transferred to MBOX2 as the primary HUB server and the IP address for LOUIEMBOX1 HUB service will be moved to MBOX2. 5:00 AM Eastern: The reinstallation of OS on MBOX1 will begin. 7:00 AM Eastern: The installation of Exchange 2010 will begin on MBOX1 The roadmap after the installation is complete will vary based on the time of completion. We will provide updates on the post that will be opened when maintenance begins.
Comments Off
November 19, 2010Router Firmware UpgradeOn November 25th between 17:00 – 19:00 Eastern OwnWebNow will be upgrading the OS version on its Value bandwidth router. Estimated period of impact is no more than fifteen minutes. During this time, the core router will require a reboot, interrupting service momentarily
Comments Off
November 18, 2010LOUIE DAGThroughout the night we prepared LOUIE for the new DAG network. As we finalize the changes, users may notice brief disconnection notices. We anticipate all the work being completed by 8AM Eastern.
Comments Off
November 17, 2010DEWEY Database MoveTonight at 9:30 PM Eastern we will be moving DEWEYDB3 to a new RAID array. During the move, users on this database will be unable to connect to their mailbox. After the move is completed, any mail that arrived during the transition will be delivered to the user’s mailbox. The move is estimated to take an hour and a half to complete.
Comments Off
November 15, 2010LOUIE OWA SearchDue to the massive adjustments over the weekend to LOUIE, we’ve noticed random errors with the previous index database for OWA searches. In an attempt to resolve search results returning “nothing” we’ve reset the index database. Over the next day the index databases will repopulate with searchable results. Update 1:15 PM Eastern: We will be rebooting the mailbox servers to reapply changes to the Exchange Schema that were lost over the weekend upgrade. The reboot will only take 15 minutes and will conducted at 00:00 -5 GMT (Eastern) 11/17/2010 11/17/10 Update 12:30 AM: The mailbox servers are now rebooting.
Comments Off
November 14, 2010LOUIE UpdatesWe are in the midst of updating our monitoring software on the LOUIE network and installing driver updates from DELL. There will be a brief period of service interruption as the mailbox servers are rebooted. 10/14/2010 9:06 EST – We are working on the upgrade on LOUIE. Right now we are just making software checks after the upgrade to ensure all of the nodes are connected to the cluster correctly and that all the services are restarting correctly. Please stand by for an update in the NOC blog for work completion. 10/14/2010 6:22 EST – All LOUIE services have been fully restored with a successful upgrade.
Comments Off
November 10, 2010Outbound CongestionWe are seeing an unusually high load of outbound mail along with delays at major free email destinations like Yahoo, Google, Hotmail, etc. Please advise your users that outbound mail tends to have less of it’s usual real time feel around the holidays as most users take to shopping, opening up new computers that become immediately infected and turned into SPAM bots, time online increases and we see more and more overloading of free email providers.
Comments Off
November 3, 2010Outbound Message Recompiling IssueA recent software update on one of the outbound nodes ended up being corrupted. The corrupted update affected roughly 400 messages and the issue on that node was addressed. It was a performance issue related to a technology we had discontinued and we caught it almost as soon as it started but due to the volume of mail processed by the server at a given moment a small percentage of mail was affected.
Comments Off
Powered by WordPress |
