Server and Network Announcements

[Completed] Echo Server Ram Upgrade (12gb -> 24gb)

Sat, 18 Feb 2012 03:25:41 +0000

We ordered new ram for the Echo server to upgrade it to 24 GB from its current 12 GB. This will allow much more cache for MySQL and disk I/O resulting in an overall improvement in performance across the board for all customers on these servers. This server is our last that is on 12 GB, and were upgrading it as a part of our continual process to improve our services. This upgrade will be performed Friday (February 17th) between 9 PM and 12 AM EST and the estimated downtime is approximately 20 minutes.

If you have any questions at all about this process, by all means, do let us know.

[Completed] Gemini RAM Upgrade

Sat, 04 Feb 2012 04:42:13 +0000

We ordered new ram for the Echo and Gemini servers to upgrade them to 24 GB from their current 12 GB. This will allow much more cache for MySQL and disk I/O resulting in an overall improvement in performance across the board for all customers on these servers. These two servers are our last that are on 12 GB, and were upgrading them as a part of our continual process to improve our services. This upgrade will be performed tomorrow (February 3rd) between 9 PM and 12 AM EST and the estimated downtime is approximately 15 minutes for each server.

If you have any questions at all about this process, by all means, do let us know.

[Resolved] Demeter Server - Web Server Crashes

Sun, 22 Jan 2012 21:14:29 +0000

On Demeter this morning LiteSpeed has crashed twice and failed to auto-restart (which it normally does an exceptionally good job of doing). When it's crashing it's not sending a report nor is it writing anything useful to the log files which is making the issue extremely difficult to track down. We've contacted LiteSpeed support for them to look into the issue and we've also rolled back a version to hopefully avoid any further unexpected crashes.

We sincerely apologize for any inconvenience these crashes are causing and we will have this resolved as soon as possible.

If you have any questions, feel free to post them.

Thank you,

[Resolved] DDoS Attack - Demeter Server - 01/19/2012 at 7:40 AM EST

Thu, 19 Jan 2012 13:39:50 +0000

Our network experienced an extremely large inbound DDoS attack that affected our entire network. The networking team quickly blocked the attack to restore network integrity but there are still some accounts that are offline due to being on the targeted/attacked IP. We've worked hard to bring all affected clients back online and have moved all affected accounts to new IP addresses. If you're using external DNS and you need to know the new IP address for your account you can open a support ticket and we'll be happy to tell you the IP (just let us know the primary domain name on the account).

If you are using our name servers (ns1~3.mddservices.com) things should come back online for you automatically anywhere from 5 minutes to 2 hours from now as DNS propagates.

[Resolved] Google Public DNS Down

Sun, 08 Jan 2012 01:48:41 +0000

Hello,

It has come to her attention that some or all of Google services or off-line at this point. One popular service is Google Public DNS, many Web servers around the world, including ours, used their services to change domain names into IP addresses. With Google Public DNS down, services such as pain and him services such as Pingdom will report downtime that does not actually exist.

All of our services are online and fully operational, if you are unable to reach your site � you are likely using Google Public DNS. I can only assume at this point, that Google is aware of the issue, and will resolve it very quickly.

If you have any questions about this, please post them here. All support tickets related to this issue, will be directed to this thread.

[Completed] Seed Backups for All Servers

Mon, 26 Dec 2011 20:00:23 +0000

We've upgraded our backup systems which usually doesn't require a fresh backup, but in this case it does. We will be performing full seed backups of all servers over the next 72 hours - if you need a file or database restored from a time before today - you need to open a support ticket within the next 12 hours with the restoration request.

If you have any questions at all, let us know.

[Resolved] Packet Loss across Network

Mon, 12 Dec 2011 17:21:06 +0000

We are aware of an issue that is causing substantial packet loss across our network and we're actively working to resolve this issue. When I have more details to provide, I will post them here.

[Completed] Gemini Server Reboot - 11/30/11 at 4:30 AM EST

Tue, 29 Nov 2011 17:35:21 +0000

We will be rebooting the Gemini server to resolve some lingering issues from the failed backup process referenced in this thread:
http://forums.mddhosting.com/topic/646-resolved-gemini-server-backups-r1soft/

The process should take no longer than 5 minutes.

[Resolved] Gemini Server Backups - R1Soft

Tue, 29 Nov 2011 17:29:52 +0000

The R1Soft 3.0 software filled all empty disk space (over 100 GB) on the Gemini server tonight during a routine backup. This is abnormal and resulted in the server exhausting all free disk space. Gemini was rebooted recently and, as such, is running a slightly newer backup module (4.2.1 build 15715) than the rest of the servers.

We've sent an email to R1Soft regarding this issue and will update this thread in regards to this issue. We have also learned that other providers (some much larger than us) have also experienced this issue and as of yet, there is no known fix short of making sure there are several hundred GB of space available to the backup system on each server (unrealistic) or simply not backing the server(s) up.

At this point all R1Soft backups on the Gemini server will be paused due to this issue, and we will update this thread once the backups are resumed.

[Closed] Unexpected Network Outage

Sun, 13 Nov 2011 22:33:12 +0000

At 07:26:57PM on 11/11/2011 our external monitoring software detected a network wide outage. The outage lasted approximately 9 minutes with about 5 more minutes for our servers to catch up with the sudden flood of requests that came in when the network was restored. To our knowledge, there were no plans for scheduled maintenance or similar. We are still awaiting details from our data center's networking team and will share these details with you when they become available.

At this time, all services appear to be operating normally.

If you have any questions, feel free to post here or open a support ticket.

----
Update - 10:37 PM EDT
Here is the latest update from our networking team:

Quote

At approximately 5:31PM to 5:40PM CDT, we had a network wide outage. The issue returned a short while ago (8:15PM). We are actively working on resolving the root condition that is causing these problems. Additional service interruptions may unfortunately occur as a result of our troubleshooting.

Stay updated about server and network issues and events!

Sat, 12 Nov 2011 01:13:03 +0000

One of the best ways to stay up to date when it comes to scheduled and unscheduled server and network issues is to subscribe to this forum section. Any time there are server or network issues or scheduled maintenance we will post in this section.

Subscribing to the forum is easy:
1. Register on the Forums here.
2. "Watch" this section of the forum:

[Completed] Network Maintenance

Thu, 10 Nov 2011 08:20:04 +0000

For those who noticed, our network was offline for about 5 minutes at 3:03AM ET on Nov. 10, 2011. The brief interruption was to install critical updates to our networking hardware. All services are now back online. If you have any questions, feel free to ask here, or open a support ticket in our help desk.

[Resolved] 2 GBPS+ DDoS on Fresco Server - Attack affecting entire network intermittenly.

Wed, 09 Nov 2011 19:55:46 +0000

An IP address on the Fresco server has come under a very large attack (2 GBPS+ and 24 million+ packets per second) and we were forced to null-route the IP to preserve our network and speed for everybody else not on the affected IP address. We are currently investigating to try and identify the target of the attack so that we can safely bring everybody else affected by this null-route back online as soon as possible.

If you have any questions, feel free to ask them, however we may not be able to reveal certain details of the attack publicly and I may respond to you via PM with specifics after addressing your question generally here in this thread.

[Completed] Fresco RAM Replacement (And Upgrade!)

Sun, 23 Oct 2011 04:33:02 +0000

We have been having some intermittent issues with the Fresco server which we believe to be related to the system memory (RAM). Due to the potential for damage we're scheduling an emergency maintenance window to replace the RAM (and also, upgrading the ram in the process) within the hour. The estimated downtime for this process is only 5 to 10 minutes and we do apologize for the short notice.

We do believe that this short maintenance window will improve the stability of the server and help to make sure that your data does not become damaged.

If you have any questions at all, let us know.

[Resolved] Fresco experienced a momentary issue, we fixed it.

Tue, 11 Oct 2011 16:51:12 +0000

We'll update you here, when we know more.

[Completed] Hermes - MySQL Governor/Stats Installation

Thu, 06 Oct 2011 14:49:21 +0000

We will be installing a MySQL Governor and Statistics system so that we can more easily track which MySQL databases are causing issues for the server for us to ensure overall stability and performance. This installation should be seamless with the only MySQL downtime being up to 5 minutes while the MySQL server is restarted. We will update this thread once the installation is completed or if any unexpected issues arise.

[Completed] Hermes - File System Check

Sun, 25 Sep 2011 08:21:41 +0000

The file system on our Hermes server needs to be checked as there are some inconsistencies being reported in the system logs. These inconsistencies do not mean that we have to bring the server down immediately (i.e. it's not an emergency at this point), however, if the server is rebooted for any reason the server will force a file system check that we will not be able to skip. We do not anticipate this happening, however, we will email all affected clients if the server does have to be restarted for any reason resulting in an unscheduled file system check.

At this point we are scheduling the file system check for Sunday, September 25th, at 3 AM EDT and we expect the process to take no longer than 3 hours. Actual downtime should be in the 30 to 60 minute range, however we have scheduled a larger window in case more work is required than is expected.

If you have any questions at all about this scheduled maintenance, please do let us know.

[Completed] Fresco - MySQL Statistics Installation

Sun, 25 Sep 2011 06:20:46 +0000

We will be installing a MySQL Statistics system so that we can more easily track which MySQL databases are causing issues for the server for us to ensure overall stability and performance. This installation should be seamless with the only MySQL downtime being up to 5 minutes while the MySQL server is restarted. We have included a larger maintenance window should any unexpected issues arise during this maintenance. We will update this thread once the installation is completed or if any unexpected issues arise.

[Completed] Fresco - Account Moves

Sat, 17 Sep 2011 23:02:41 +0000

Hello,

In order to increase long-term stability we are moving numerous accounts off of the Fresco server. This will likely cause some slowness this morning as the accounts transfer, however, this will only last as long as the transfers are running which shouldn't be more than a couple of hours. The end result will be a server with less overall load meaning increased stability.

The issue with the Fresco server is not the sheer number of accounts or their CPU usage, but the amount of disk reads and writes those accounts are doing. The hard drives in the Fresco server are simply having a hard time keeping up which is why there are intermittent slowdowns periodically and then when the backups run at night it really causes some slowness.

If you have any questions, let us know. We will update this thread once the process is completed and will be notifying customers as their accounts are moved.

[Notice] Hermes Server MySQL Restart

Sun, 11 Sep 2011 20:40:00 +0000

We are adjusting some settings for the MySQL server for the Hermes server and to make the changes live we have to restart the MySQL server. We do so gracefully so that no damage is done to any databases and it can take upwards of 5 minutes to flush the RAM contents to the disk and restart. The restart is being issued within the next 60 seconds after this post.