Server and Network Announcements

Multiple Servers - Emergency Maintenance Reboots to close Operating System Security Hole

Thu, 16 May 2013 20:42:33 +0000

Unfortunately the internet is a hostile place and there are a lot of individuals out there. There has been what is called a 'root escalation exploit' and in layman's terms it would permit any user to become an administrator and perform whatever commands they want [such as reading your data, wiping out the server, etc...]. This is an operating-system level exploit and not an issue we could have prevented ourselves and, as such, we have to rely upon our software vendors for patched versions of the operating systems.

Due to the nature of this issue, as soon as a patch or updated operating system is available we will be rebooting all affected systems into the new kernel. While we don't anticipate the process to take more than 5 to 10 minutes per server, there is the possibility of unexpected issues such as a file system check on reboot that may take substantially longer.

Currently the servers slated for reboots as soon as patches are available are as follows: Kobold, Jasmine, Icarus, Atlantis, and Boreas.

When we are rebooting each server we will update this thread and if any issues arise, they will be posted here as well.

We're also going to be sending out an email to all customers on the affected servers directing them to this thread as well.

If you have any questions, by all means feel free to ask.

Jasmine Emergency Kernel Update Status

Wed, 15 May 2013 14:05:09 +0000

During emergency maintenance to our jasmine server, we updated to a newer kernel that is not vulnerable to the current zero-day exploit. Unfortunately Jasmine did not reboot as expected and we are still investigating why. We will update this thread with information specific to Jasmine as we have new information for you.

Update:
10:00pm - Marking this topic resolved.

Emergency Atlantis kernel update status

Wed, 15 May 2013 03:51:39 +0000

During emergency maintenance to our atlantis server, we updated to a newer kernel that is not vulnerable to the current zero-day exploit. Unfortunately we found a seperate hardware issue affecting atlantis that prevented us from bringing it back online. We are currently testing and replacing affected hardware in this server.

We will post any updates regarding this hardware issue with atlantis in this forum thread.

Update:
9:56pm - This issue is now resolved.

Server Upgrades - Echo and Fresco servers

Fri, 10 May 2013 20:31:23 +0000

Hello!

We have ordered, received, and assembled new hardware to replace the aging Echo and Fresco servers. Echo and Fresco are our two oldest servers that are still online and, as such, they are the two that are getting upgraded first. This upgrade, while substantial, will require account transfers from the old hardware to the new. For those of you using our DNS the impact should be minimal resulting in a short period of propagation and possibly requiring you to update your email clients to reflect the new server information. The transfer process will begin no earlier than Friday, April 26th, 2013 at 8 PM ET.

We will be leaving the old servers online for at least 7 days after all transfers are completed to ensure DNS has time to update/propagate. Anybody not using our DNS will need to manually update their DNS provider with the new IP address information once the transfer process is completed. We do not plan on purging data from the old servers immediately after taking them offline just in case this post and our emails are missed and DNS isn't updated for a site.

The new hardware is running dual hex-core processors with Hyper Threading, 64GB DDR3 RAM, and RAID10 SAS storage that is Read/Write cached by a RAID1 SSD array. We are also evaluating moving to BetterLinux from CloudLinux which should allow all accounts to perform tasks more quickly while still balancing usage for all users to keep things fast and stable.

We are going to do our best to ensure the impact from this move is as small as possible, however, if you have any questions feel free to let us know.

Updates:

Monday, April 29: Transfers to the new server have not started yet. An additional email notice will be sent before we begin.
Friday, May 3: Transfers will likely commence this evening, and continue this weekend if reqired. An email notice will be sent to you before your account is moved.
Friday, May 3: Transfers from echo and fresco have begun and emails have been sent to affected accounts.
All transfers have been completed.
The echo and fresco servers will be retired no later than Wednesday, May 15, 2013.

Fresco SSL certificate expired

Fri, 03 May 2013 21:40:17 +0000

Hello Everyone,

The SSL certificate for our fresco server has expired and you will see a warning when connecting to cPanel, WHM, email, and other encrypted services on this server. As a temporary measure, you can accept or bypass any warning. This should be corrected shortly.

DDoS Attack on Echo Server - IP 173.248.191.214

Fri, 26 Apr 2013 23:41:33 +0000

Hello,

Our upstream provider alerted us to an inbound DDoS attack on the IP address 173.248.191.214 on the Echo server. To protect our entire network and all customers not on this IP, this IP address has been null-routed. We are going to do what we can to move accounts off of this IP in small groups to restore service and/or to hopefully identify the target of this attack but we cannot offer any sort of ETA at this time.

Attached Thumbnails

Icarus Server - Inbound DDoS on 173.248.188.124

Sun, 21 Apr 2013 11:44:47 +0000

The IP "173.248.188.124" on the Icarus server has been being hit off and on by decently sized DDoS attacks as of 6:53 AM ET.

The attacks started at around 200 MBPS and have worked their way up to 400 MBPS and they're a combination of TCP SYN floods [HTTP Requests] as well as a DNS Amplification attack on port 53 as well.

Our facility, at this point, has null-routed the IP - we're going to do what we can to restore service to anyone affected.

Extremely Large WordPress WP-Admin Brute Force Attacks

Tue, 16 Apr 2013 23:51:19 +0000

There is an ongoing WordPress brute-force attack that is affecting a large number of providers. CloudFlare has made a blog post about the issue and has reported that the attack is coming from upwards of 100,000 individual IP addresses or systems. Many providers have had entire servers taken offline and accounts compromised as a result of this ongoing attack.

While we haven't had any servers go offline as a result of these attacks the larger issue comes as a result of any compromised WordPress installations that may result from this attack. Should your WordPress installation be brute-forced successfully, the attacker could upload malicious files to your account to include your account in this attack, future attacks, or worse. They could view all of your data, delete your data, modify your data, etc.

One basic step that can be taken to protect your WordPress installation and your account with us, is to make sure that you are not using the default username of 'admin' for your WordPress administration. This is the default username for a new WordPress installation. WordPress does suggest changing this username as a method of security through obscurity.

As a result of this attack we've chosen to take a step that we would not ordinarily take, and that is to change the log-in username of any WordPress installation where it is currently 'admin'. This not only will help keep your account secure from this attack, but also from all possible future brute-force attacks on your WordPress installations while still allowing you full access to your WordPress administration via the new username. Keep in mind that if you've already changed your administration username to something other than 'admin' or you use an alternate username to log-in to your WordPress administration - this change will not affect you.

In the interest of keeping your WordPress installation secure, and keeping the username obscure from potential attackers we are not going to include the new usernames into this post. We are going to be sending out a mass mail to all of our customers advising you of the change and what the new usernames are. We are making this post on our forums as well as cross-posting it on our FaceBook Page, and our Twitter Feed to help ensure everybody is aware of this change.

If you have any questions about this, you are welcome to ask them here if they are generic in nature. If your questions are specific to your account, do please open a support ticket and a member of the senior staff will answer your questions concerning this.

Update regarding WordPress Network/MultiSite:
An issue has come to our attention regarding WordPress Network Installations (formerly WordPress MultiSite). If you were previously using the default 'admin' log in to access all of your WordPress network sites and can no longer access other sites in your WordPress network with the new username, please open a support ticket so can verify the issue and correct it for you. When opening the support ticket, please be sure to include the URL to access WordPress, and if possible, the WordPress database name.

Do not reply on our forums to request assistance with this issue. A support ticket is required to protect the privacy of your information.

Fresco Server - DDoS on 173.248.187.18

Tue, 26 Mar 2013 14:52:05 +0000

The ip "173.248.187.18" came under a rather large DDoS attack a short time ago. The attack was only about 300 megabit, however, it was comprised of over 500,000 packets per second which is enough to effectively take the entire server offline as the kernel can't keep up with that much. We've been forced to null-route (i.e. de-route) the affected IP to keep the service online for everybody else not on that IP.

As usual, we're going to work to mitigate this including, but not limited to, changing the IP address of affected accounts. There is a possibility that the attack will shift IPs when this happens and, if that's the case, we'll have to keep moving and splitting accounts until we are able to identify the target.

It's important to keep in mind that we aren't under attack, but one of our customers is. The attack is simply a packet flood to port 53 (standard DNS port) so there is no way to identify who the target is based upon the traffic / packets / data alone.

Emergency Maintenance - Hermes Server MySQL

Fri, 15 Mar 2013 19:07:23 +0000

Hello,

The Hermes server has a small issue with MySQL that we need to address in order to apply some critical updates to the system. We should be able to perform the update without more than a minute or two of MySQL downtime, however, there is the possibility it could end up taking 30 to 60 minutes if there are any unforeseen issues.

We'll update this thread if we run into any issues that will cause MySQL to be down for more than a few minutes, and we'll update this thread when the maintenance is completed.

We apologize for any trouble this may cause, we try not to do such short-term maintenance but this issue is preventing some important updates from being applied and the risks of waiting outweigh the risks of not waiting in this case.

Icarus Server Instability

Thu, 21 Feb 2013 02:39:40 +0000

Update - 02/20/2013 - 9:40 PM ET
This issue is now resolved.

============

On Monday, February 18th at approximately 6:20 AM ET the Icarus server crashed with a Kernel Panic (i.e. when something goes wrong with the operating system and it 'gives up' and crashes). This sort of situation is out of our control beyond which version of the system kernel we boot into. One would think the latest 'stable' version would be stable but more often than not we've resolved one problem and found two new ones during kernel upgrades.

Unfortunately kernel upgrades are required not only for security purposes but also to resolve issues that can cause the server to lock up unexpectedly as it did on this morning. During the process of troubleshooting the panic we decided to upgrade to a newer version of the kernel.

This morning at approximately 2:50 AM ET all was well until our R1Soft backups decided to run at which point the server hang up and quit responding forcing a reset. We believe this to be a Kernel issue, but possibly it's a R1Soft issue. We're going to be working with both software vendors to try and find the source of the issue.

If you have any questions at all, just let us know.

Outbound UDP flood FROM Fresco

Wed, 20 Feb 2013 14:23:33 +0000

This morning we were forced to block some traffic from our fresco server due to an outbound UDP flood. We are still investigating this issue and will post more details when we have them.

Jasmine Server - DDoS on 173.248.188.150

Wed, 20 Feb 2013 13:37:19 +0000

Update 8:52 PM ET
All accounts have been shifted to alternate IP addresses.
============

We saw a rather large DDoS attack hit the IP 173.248.188.150 on our Jasmine server just now. Our networking team acted quickly to drop this IP to keep everybody else online.

We're going to work to shift everybody on this IP to a small subset of new IPs to bring people back online as well as to hopefully narrow down who is actually under attack.

It's unfortunate that the internet is such a hostile place these days.

Outbound Spam Issue with Jasmine and Gemini

Sat, 16 Feb 2013 04:30:59 +0000

We have, at least temporarily, had to disable all outbound email (port 25) on the Jasmine and Gemini servers. The servers have been blasting spam and upon investigation we were initially unable to identify the source due to the nature of the attack.

We called in a third party security company that has expertise in dealing with this sort of issue and will keep you updated. In the meantime all web sites will function as normal, but outbound mail will not.

We do apologize for any problems this may cause you, however, until we have this issue resolved we cannot allow the server to blast out spam as it has been doing and the only way we've been able to stop it, thus far, is by closing the outbound port.

While we do our absolute best to keep everything rock solid and secure, we are humans working in the real world and unfortunately bad things can and inevitably will happen and we're going to do our best to get this resolved as quickly as humanly possible.

We'll keep this thread updated as we learn more.

Degraded Raid Array - Drive Failure - Demeter Server

Thu, 07 Feb 2013 07:19:40 +0000

The Demeter server has had one of it's 8 drives fail and, as such, the array is degraded. We run RAID10 which allows for drive failure, removal, and replacement. While the drive is missing performance may be a little lower than expected and during the rebuild period as well. It should barely be noticeable if at all, but we wanted to let you know either way.

The drive has already been pulled and we're going to be inserting a new drive and rebuilding the array very shortly.

Echo Server - DDoS on IP 173.248.187.73

Thu, 31 Jan 2013 09:29:12 +0000

Hello,

The Echo server came under a rather large DDoS attack tonight on the IP 173.248.187.73 and, as such, we've had to null-route this IP to restore service for all others on the server. We will be doing our best to restore service for those who were on that IP, however, there is no ETA at this time.

We will update this thread when we have more information to provide.

The graphs drop off quickly due to us placing a null-route quickly to eliminate this attack traffic. It peaked at 448,000 packets per second and 549.73 megbits per second before it was null-routed.

Jasmine Server - Kernel Panic resulting in Corrupted Data

Sat, 05 Jan 2013 06:48:57 +0000

The Jasmine server has experienced a Kernel panic, forcing a reset. We'll update this thread with more details when we have them.

If you wish to see the panic: http://www.screen-sh...02_22-46-32.png

Icarus Scheduled Maintenance - Battery Backup Unit Replacement

Thu, 20 Dec 2012 23:05:10 +0000

Update - 12/19/2012
The maintenance window has been scheduled for December 20th at 3 AM ET (GMT-5) [i.e. tonight] and should take no longer than 15 minutes. We'll update this thread should any complications arise, however, we expect no issues.

============
Hello,

Our Icarus server is reporting that the battery back up unit for the raid controller has failed, resulting in the card going to Write-Through to protect data integrity in the unlikely event of a total power loss. Denver's power is historically extremely solid and our facility does have UPS and Generator power in the event of a grid loss, so we've forced Write-Back on with this failed battery back up.

We do need to schedule maintenance as soon as possible to replace this battery to help ensure data integrity in the event of a major power issue. We will also be updating the firmware on the controller to the latest version which will offer better failure prediction in the future so we can hopefully plan the replacement a bit further out.

Battery back up units are warrantied for a period of 12 months and this BBU has gone 24 months. All of our newer/future servers run capacitor backups (similar to BBU, however, they do not degrade over time like a regular battery back up does). We're working on moving everybody over to the capacitor backups, however, it's a process that will take time.

We'll update this thread as soon as we have a time scheduled. If you have any questions, let us know. It will definitely be during off-peak (likely somewhere between 1 AM and 5 AM ET.

Unexpected Outage due to Stuck Process - All servers

Thu, 13 Dec 2012 23:28:14 +0000

Today we were doing some network maintenance, and in the process we were identifying our private networking ports on our servers to ensure that we didn't mistakenly change the wrong thing, better to be sure than sorry. In this process we ran a tool called "ethtool" to identify the networking ports (causes them to flash to identify) and when we ran this tool, which we've run hundreds if not thousands of times before, the process got stuck on all servers and caused load to skyrocket to a point that we could no longer access them.

We could immediately reset each server to bring it back online, but doing so risks data corruption and lengthy file system repairs (2 to 8 hours) so, instead, we're going to be logging into each server locally and killing the stuck processes manually.

It will take longer than a reset, but doesn't incur the same risk of corruption and lengthy file system repairs.

It's really an odd issue to have, the servers are 'online' in that they're powered up, operating, and have networking - they're just too busy with this stuck process to do anything else.

We'll update this thread as we have more information.

Emergency VPS node (Atlantis) maintenance

Mon, 10 Dec 2012 23:25:04 +0000

12/10/2012 6:30 PM ET
All customers are now back onto the normal high-end hardware at this time and this issue is considered resolved.
-------------------------------------------------

We have determined that emergency maintenance is required for the atlantis VPS node. We are temporarily relocating all VPS servers on this node to another server and will migrate them back when maintenance is complete. You may notice your VPS is offline for 5 to 10 minutes while it shuts down, is moved to the temporary server, and is brought back online. No other significant changes will be made, your IP addresses will not change, and overall this should have a very minimal impact on all VPS servers on this node.

If you have any questions, feel free to ask here, or to contact our support department.