Server and Network Announcements

[Completed] Demeter Server Reboot - Friday, May 11th, 2012

Sat, 12 May 2012 03:29:43 +0000

Hello,

Due to a system kernel issue we're scheduling a reboot for the Demeter server on Friday, May 11th, 2012 between 10 PM and 2 AM EST. We expect the downtime to be no longer than 8 minutes.

[Completed] Cypress Seed Backup

Tue, 17 Apr 2012 23:37:55 +0000

Hello,

Cypress is currently undergoing a *full* backup versus the standard incremental backup which is much more intensive and takes quite a bit longer. We started this process yesterday as Sunday is an off-peak time for such processes, however we did not anticipate the process running into today as well. Due to it having already spent nearly 16 hours running, and 4 hours remaining we are going to allow the process to continue as not to go a complete week without recent backups. If we were to cancel this backup process now, not only would the last 16 hours be in vein, but we would not be able to keep regular daily backups until next weekend when we could run the process all over again.

I do apologize for any issues this may cause you, as the performance is definitely going to be a bit lower than you are used to until the process completes. I'll try and update this thread when the process is done, however it does not alert us when it's completed and I'll just have to check on it periodically.

You may also notice the server load is reporting from 20 to 30, and this is primarily due to a bug in the current version of the CloudLinux System Kernel. You can read more information on this in another thread on our forum at http://forums.mddhosting.com/topic/715-notice-cypress-and-hermes-server-load-averages/

For any curious, here is a screen-shot from the backup process so far as of this thread:

[Notice] Cypress and Hermes Server Load Averages

Tue, 17 Apr 2012 02:47:19 +0000

Hello,

As you likely already know, we run CloudLinux which is a little different than standard Linux. There is a wealth of information on their site at http://www.cloudlinux.com/ if you're interested in learing more.

CloudLinux, being different, calculates load a little differently than a standard Linux build. For example, we upgraded Cypress and Hermes from CloudLinux (2.6.18-338.19.1.el5.lve0.8.36) to CloudLinux (2.6.18-408.el5.lve0.8.58) tonight and the load numbers have changed drastically. The only thing that *really* changed was the way that CloudLinux calculates the server load. We just wanted to make our client base aware of this change as we do have a lot of customers who watch the load numbers as an indication of performance when in reality there are other much better and accurate resources to monitor. A post in the CloudLinux knowledgebase itself addresses the load changes which you can see at https://helpdesk.cloudlinux.com/index.php?/Knowledgebase/Article/View/30/0/after-i-started-to-use-cloudlinux-my-load-average-is-lowhigh

Cypress was previously reporting loads between 5 and 9, which is 100% acceptable for a 16 core system and is now reporting anywhere from 15 to 20 which ordinarily would be very bad. In this instance, it'a mis-calculation of the load and has no bearing on the real performance of the server.

If you have any questions about this, by all means, post a response on this thread and I'll be happy to address it.

[Resolved] Cypress System Issues - Stuck Processes

Sun, 15 Apr 2012 03:19:00 +0000

Due to some stuck processes that are causing major performance issues on the Cypress server, we're going to be issuing an immediate emergency reboot. We do not anticipate this to take any longer than 10 minutes, however, we will update you if there are any unforeseen issues that arise.

[Resolved] Cypress DDoS Attack

Sat, 07 Apr 2012 18:46:51 +0000

An IP on the Cypress server has come under attack, and we're working to move clients off of the IP. If your site is currently offline, it's likely that you are on the affected IP. Feel free to open a ticket with us if you have any questions and your site is offline.

Keep in mind this attack is not against *us*, one of our clients is under attack and we're currently working to identify the target of the attack so we can isolate them. Due to the nature of most of these attacks, they use malformed packets that don't actually make requests for any specific domain or URL which makes identifying the target more difficult.

We'll update this thread when we have more information to provide.

[Resolved] Boreas VPS Node Outage

Fri, 23 Mar 2012 03:41:09 +0000

Hello,

A client's VPS crashed the Boreas node a short bit ago, and locked the server up to the point that we were forced to issue a reset. We tried to avoid the reset, but we were not even able to log into the system directly using a keyboard and mouse at the physical system itself. Due to the unclean reset, a file system check is being forced by the OS which can take anywhere from 10 minutes to 120 minutes. We will update this thread with progress as we work on this.

[Completed] Network Distribution Switch Reboot

Thu, 22 Mar 2012 06:54:35 +0000

Hello,

We have to perform a reboot on the distribution switch that handles the public networking inside of our cabinet tonight. The reboot should take no longer than 2 to 3 minutes. All servers will remain powered online and operational, they will simply lose connectivity for a couple of minutes.

If you have any questions at all, let us know.

[Completed] Echo Server Ram Upgrade (12gb -> 24gb)

Sat, 18 Feb 2012 03:25:41 +0000

We ordered new ram for the Echo server to upgrade it to 24 GB from its current 12 GB. This will allow much more cache for MySQL and disk I/O resulting in an overall improvement in performance across the board for all customers on these servers. This server is our last that is on 12 GB, and were upgrading it as a part of our continual process to improve our services. This upgrade will be performed Friday (February 17th) between 9 PM and 12 AM EST and the estimated downtime is approximately 20 minutes.

If you have any questions at all about this process, by all means, do let us know.

[Completed] Gemini RAM Upgrade

Sat, 04 Feb 2012 04:42:13 +0000

We ordered new ram for the Echo and Gemini servers to upgrade them to 24 GB from their current 12 GB. This will allow much more cache for MySQL and disk I/O resulting in an overall improvement in performance across the board for all customers on these servers. These two servers are our last that are on 12 GB, and were upgrading them as a part of our continual process to improve our services. This upgrade will be performed tomorrow (February 3rd) between 9 PM and 12 AM EST and the estimated downtime is approximately 15 minutes for each server.

If you have any questions at all about this process, by all means, do let us know.

[Resolved] Demeter Server - Web Server Crashes

Sun, 22 Jan 2012 21:14:29 +0000

On Demeter this morning LiteSpeed has crashed twice and failed to auto-restart (which it normally does an exceptionally good job of doing). When it's crashing it's not sending a report nor is it writing anything useful to the log files which is making the issue extremely difficult to track down. We've contacted LiteSpeed support for them to look into the issue and we've also rolled back a version to hopefully avoid any further unexpected crashes.

We sincerely apologize for any inconvenience these crashes are causing and we will have this resolved as soon as possible.

If you have any questions, feel free to post them.

Thank you,

[Resolved] DDoS Attack - Demeter Server - 01/19/2012 at 7:40 AM EST

Thu, 19 Jan 2012 13:39:50 +0000

Our network experienced an extremely large inbound DDoS attack that affected our entire network. The networking team quickly blocked the attack to restore network integrity but there are still some accounts that are offline due to being on the targeted/attacked IP. We've worked hard to bring all affected clients back online and have moved all affected accounts to new IP addresses. If you're using external DNS and you need to know the new IP address for your account you can open a support ticket and we'll be happy to tell you the IP (just let us know the primary domain name on the account).

If you are using our name servers (ns1~3.mddservices.com) things should come back online for you automatically anywhere from 5 minutes to 2 hours from now as DNS propagates.

[Resolved] Google Public DNS Down

Sun, 08 Jan 2012 01:48:41 +0000

Hello,

It has come to her attention that some or all of Google services or off-line at this point. One popular service is Google Public DNS, many Web servers around the world, including ours, used their services to change domain names into IP addresses. With Google Public DNS down, services such as pain and him services such as Pingdom will report downtime that does not actually exist.

All of our services are online and fully operational, if you are unable to reach your site � you are likely using Google Public DNS. I can only assume at this point, that Google is aware of the issue, and will resolve it very quickly.

If you have any questions about this, please post them here. All support tickets related to this issue, will be directed to this thread.

[Completed] Seed Backups for All Servers

Mon, 26 Dec 2011 20:00:23 +0000

We've upgraded our backup systems which usually doesn't require a fresh backup, but in this case it does. We will be performing full seed backups of all servers over the next 72 hours - if you need a file or database restored from a time before today - you need to open a support ticket within the next 12 hours with the restoration request.

If you have any questions at all, let us know.

[Resolved] Packet Loss across Network

Mon, 12 Dec 2011 17:21:06 +0000

We are aware of an issue that is causing substantial packet loss across our network and we're actively working to resolve this issue. When I have more details to provide, I will post them here.

[Completed] Gemini Server Reboot - 11/30/11 at 4:30 AM EST

Tue, 29 Nov 2011 17:35:21 +0000

We will be rebooting the Gemini server to resolve some lingering issues from the failed backup process referenced in this thread:
http://forums.mddhosting.com/topic/646-resolved-gemini-server-backups-r1soft/

The process should take no longer than 5 minutes.

[Resolved] Gemini Server Backups - R1Soft

Tue, 29 Nov 2011 17:29:52 +0000

The R1Soft 3.0 software filled all empty disk space (over 100 GB) on the Gemini server tonight during a routine backup. This is abnormal and resulted in the server exhausting all free disk space. Gemini was rebooted recently and, as such, is running a slightly newer backup module (4.2.1 build 15715) than the rest of the servers.

We've sent an email to R1Soft regarding this issue and will update this thread in regards to this issue. We have also learned that other providers (some much larger than us) have also experienced this issue and as of yet, there is no known fix short of making sure there are several hundred GB of space available to the backup system on each server (unrealistic) or simply not backing the server(s) up.

At this point all R1Soft backups on the Gemini server will be paused due to this issue, and we will update this thread once the backups are resumed.

[Closed] Unexpected Network Outage

Sun, 13 Nov 2011 22:33:12 +0000

At 07:26:57PM on 11/11/2011 our external monitoring software detected a network wide outage. The outage lasted approximately 9 minutes with about 5 more minutes for our servers to catch up with the sudden flood of requests that came in when the network was restored. To our knowledge, there were no plans for scheduled maintenance or similar. We are still awaiting details from our data center's networking team and will share these details with you when they become available.

At this time, all services appear to be operating normally.

If you have any questions, feel free to post here or open a support ticket.

----
Update - 10:37 PM EDT
Here is the latest update from our networking team:

Quote

At approximately 5:31PM to 5:40PM CDT, we had a network wide outage. The issue returned a short while ago (8:15PM). We are actively working on resolving the root condition that is causing these problems. Additional service interruptions may unfortunately occur as a result of our troubleshooting.

Stay updated about server and network issues and events!

Sat, 12 Nov 2011 01:13:03 +0000

One of the best ways to stay up to date when it comes to scheduled and unscheduled server and network issues is to subscribe to this forum section. Any time there are server or network issues or scheduled maintenance we will post in this section.

Subscribing to the forum is easy:
1. Register on the Forums here.
2. "Watch" this section of the forum:

[Completed] Network Maintenance

Thu, 10 Nov 2011 08:20:04 +0000

For those who noticed, our network was offline for about 5 minutes at 3:03AM ET on Nov. 10, 2011. The brief interruption was to install critical updates to our networking hardware. All services are now back online. If you have any questions, feel free to ask here, or open a support ticket in our help desk.

[Resolved] 2 GBPS+ DDoS on Fresco Server - Attack affecting entire network intermittenly.

Wed, 09 Nov 2011 19:55:46 +0000

An IP address on the Fresco server has come under a very large attack (2 GBPS+ and 24 million+ packets per second) and we were forced to null-route the IP to preserve our network and speed for everybody else not on the affected IP address. We are currently investigating to try and identify the target of the attack so that we can safely bring everybody else affected by this null-route back online as soon as possible.

If you have any questions, feel free to ask them, however we may not be able to reveal certain details of the attack publicly and I may respond to you via PM with specifics after addressing your question generally here in this thread.