27 replies to this topic

[MikeDVB]

This issue is now marked as resolved.

 

You may have seen this thread regarding file systems errors on the Echo Server: http://forums.mddhos...dow-is-2-hours/ [If not, I suggest giving it a quick read].

The system is not reporting the file system as damaged, however, the file system is obviously damaged from the standpoint that it's getting to where new folders cannot be created in specific locations. While it is theoretically possible that this is fallout from the previous damage - we suspect there to be a silent corruption issue on the server.

 

Due to this we are going to be migrating accounts off of the server as soon as possible to avoid/prevent further damage to customer data.  We are going to do our absolute best to keep IP addresses the same so that this transition is as seamless as possible.  There may be a brief period [up to an hour] where you are unable to connect to email or FTP at the server hostname, however, we will also be re-routing this IP to the new temporary server as well to restore this without DNS propagation as well - you may experience SSL Certificate Warnings which you can safely ignore/accept.

 

While we apologize for any trouble this may cause you we do appreciate your understanding while we take immediate action in order to protect your data integrity and your services with us.  Without taking this action the server could, at any point in time, go read-only or complely crash/lose data and we want to avoid this if at all possible.  We do have backups of the server and will have those on-hand just in case.

 

We are going to be sending out an email to all affected clients regarding this issue.

 

If you have any generalized questions about this issue do feel free to ask them here in this thread as others will likely have the same questions.  If you have any account-specific questions by all means please feel free to open a technical support ticket by respond to the email that you will receive shortly or simply by opening a new ticket manually.

[MikeDVB]

Unfortunately we're going to have to resort to moving accounts using our backups taken last night at around 3 AM to keep everybody online and to be as seamless as possible.  We will then reboot the Echo server, file system check it, and perform a backup of it [if possible] to snag a copy of any data changed since the backup last night.

[MikeDVB]

We're about 35% done converting our incremental nightly backups into files that can be restored by cPanel.

[MikeDVB]

We're still in the process of restoring backups to an alternative server before re-routing the IPs assigned to the Echo server.

 

This move will be temporary - only until the Echo server issue is diagnosed and resolved permanently.

 

This process should be as seamless as possible as we're not changing any IPs on any accounts.

[MikeDVB]

Account backup restorations are approximately 72% completed.

[MikeDVB]

Restorations are at approximately 91%.

[MikeDVB]

It looks as though Echo has now gone offline due to this issue.  Thankfully we have roughly 91% of our customers on the Echo server active on another server.

 

Due to the increased strain of the transfers in addition to the additional load of the Echo accounts [we were hoping not to re-point the IP to Kobold until the transfers were done] your sites will be sluggish until the transfer is completed.

 

We apologize for any trouble this may cause - we've tried to get this done as quickly as possible to avoid this but unfortunately it looks like the Echo server went offline faster than anticipated.

[gplurn]

Hello, greatly appreciate all the updates.

 

worldofbenbo.com seems to be one of the 9%, plus email appears to be down on all accounts although I can't log in to webmail to verify whether it's just my mail client.

 

The cpanel login does redirect to Kobold.

 

I would appreciate any ETA on being able to login to webmail/cpanel etc.

 

Cheers

[MikeDVB]

Hello, greatly appreciate all the updates.

Doing my best.
 

worldofbenbo.com seems to be one of the 9%, plus email appears to be down on all accounts although I can't log in to webmail to verify whether it's just my mail client.

You are part of the 9% - the server is working on accounts that start with "u" at this time.
 

The cpanel login does redirect to Kobold.

Aye - the IP routed to Kobold when Echo went offline.

We did just now get Echo back online [admittedly after extensive file system repairs - there is no knowing how long it will be online before it goes back offline or how much data is damaged]. I would suggest snagging a copy of your databases and any recently uploaded/modified files just in case as our last backup is just prior to this incident beginning which is roughly 24 hours ago.
 

I would appreciate any ETA on being able to login to webmail/cpanel etc.

As quickly as possible is about as good as I can do. Some accounts are very tiny and some are huge - right now it appears to be working on an account with a LOT of small files which makes it take substantially longer.

You should be able to access webmail/cPanel on Echo for the time being but I cannot promise it will not go back offline unexpectedly.

[MikeDVB]

If you have made significant changes to your account that would not be easy to replicate I would suggest logging in now at http://echo.supportedns.com/cpanel -> Backups -> Full Backups -> Download and Generate a Full Backup.

I can't guarantee how long the Echo server will be online before it crashes again and I cannot guarantee the accuracy of the data on the Echo server but, that said, it's better to have some of it than none of it if you need to recreate some of it by hand.

I apologize for any trouble this may cause you but we're doing our best to preserve data but due to the nature of how we have to go about this process we're not able to pull over the newest data from the Echo server just yet - we will try to do so once all accounts are restored on alternate hardware.

[gplurn]

thanks for the updates Mike

[MikeDVB]

We have initiated full cPanel backups to be generated to a standby backup server to preserve as much up-to-date account data as possible before the server goes offline, however, please do not rely upon this. This sytem goes alphabetically so, if for example, your username starts with a 'z' there's a significant chance it won't get to your account in time.

[MikeDVB]

All of our backups from the other day are restored and active on Kobold.  We are working on pulling over updated data from Echo now to Kobold.

[gplurn]

thanks for your efforts!

[Cuga]

Are the issues with Echo the reason Softaculous doesn't seem to be available on Echo's cpanel for me?

[MikeDVB]

Are the issues with Echo the reason Softaculous doesn't seem to be available on Echo's cpanel for me?

I am seeing Softaculous listed when I check.

I believe we found the issue with the Echo server - our testing indicated the issue was resolved, however, we're still going to take an extra set of backups just to be sure.

[Cuga]

Thanks for the quick response

 

When I click on the Softaculous link it goes to a dead link (404):

https://echo.support.../index.live.php

 

The bottom panel of panel of cpanel that normally displayed all the apps that can be installed is empty (just a single link to the dead page above as well).

[MikeDVB]

This should be fixed.

[Cuga]

This should be fixed.

 

Confirmed, Softaculous seems to be working again.  Thank you again for the rapid response and fix

[afkn]

I believe we found the issue with the Echo server - our testing indicated the issue was resolved, however, we're still going to take an extra set of backups just to be sure.

Is it safe to make changes to our files now?