Jump to content
MDDHosting Forums

Echo Server - Silent Data Corruption - Migrations Pending


Recommended Posts

This issue is now marked as resolved.

 

You may have seen this thread regarding file systems errors on the Echo Server: http://forums.mddhosting.com/topic/967-echo-server-file-system-check-required-scheduled-for-sunday-jan-12-2014-at-2-am-et-estimated-maintenance-window-is-2-hours/ [if not, I suggest giving it a quick read].

The system is not reporting the file system as damaged, however, the file system is obviously damaged from the standpoint that it's getting to where new folders cannot be created in specific locations. While it is theoretically possible that this is fallout from the previous damage - we suspect there to be a silent corruption issue on the server.

 

Due to this we are going to be migrating accounts off of the server as soon as possible to avoid/prevent further damage to customer data. We are going to do our absolute best to keep IP addresses the same so that this transition is as seamless as possible. There may be a brief period [up to an hour] where you are unable to connect to email or FTP at the server hostname, however, we will also be re-routing this IP to the new temporary server as well to restore this without DNS propagation as well - you may experience SSL Certificate Warnings which you can safely ignore/accept.

 

While we apologize for any trouble this may cause you we do appreciate your understanding while we take immediate action in order to protect your data integrity and your services with us. Without taking this action the server could, at any point in time, go read-only or complely crash/lose data and we want to avoid this if at all possible. We do have backups of the server and will have those on-hand just in case.

 

We are going to be sending out an email to all affected clients regarding this issue.

 

If you have any generalized questions about this issue do feel free to ask them here in this thread as others will likely have the same questions. If you have any account-specific questions by all means please feel free to open a technical support ticket by respond to the email that you will receive shortly or simply by opening a new ticket manually.

Link to comment
Share on other sites

Unfortunately we're going to have to resort to moving accounts using our backups taken last night at around 3 AM to keep everybody online and to be as seamless as possible. We will then reboot the Echo server, file system check it, and perform a backup of it [if possible] to snag a copy of any data changed since the backup last night.

Link to comment
Share on other sites

We're still in the process of restoring backups to an alternative server before re-routing the IPs assigned to the Echo server.

 

This move will be temporary - only until the Echo server issue is diagnosed and resolved permanently.

 

This process should be as seamless as possible as we're not changing any IPs on any accounts.

Link to comment
Share on other sites

It looks as though Echo has now gone offline due to this issue. Thankfully we have roughly 91% of our customers on the Echo server active on another server.

 

Due to the increased strain of the transfers in addition to the additional load of the Echo accounts [we were hoping not to re-point the IP to Kobold until the transfers were done] your sites will be sluggish until the transfer is completed.

 

We apologize for any trouble this may cause - we've tried to get this done as quickly as possible to avoid this but unfortunately it looks like the Echo server went offline faster than anticipated.

Link to comment
Share on other sites

Hello, greatly appreciate all the updates.

 

worldofbenbo.com seems to be one of the 9%, plus email appears to be down on all accounts although I can't log in to webmail to verify whether it's just my mail client.

 

The cpanel login does redirect to Kobold.

 

I would appreciate any ETA on being able to login to webmail/cpanel etc.

 

Cheers

Link to comment
Share on other sites

Hello, greatly appreciate all the updates.

Doing my best.

 

worldofbenbo.com seems to be one of the 9%, plus email appears to be down on all accounts although I can't log in to webmail to verify whether it's just my mail client.

You are part of the 9% - the server is working on accounts that start with "u" at this time.

 

The cpanel login does redirect to Kobold.

Aye - the IP routed to Kobold when Echo went offline.

 

We did just now get Echo back online [admittedly after extensive file system repairs - there is no knowing how long it will be online before it goes back offline or how much data is damaged]. I would suggest snagging a copy of your databases and any recently uploaded/modified files just in case as our last backup is just prior to this incident beginning which is roughly 24 hours ago.

 

I would appreciate any ETA on being able to login to webmail/cpanel etc.

As quickly as possible is about as good as I can do. Some accounts are very tiny and some are huge - right now it appears to be working on an account with a LOT of small files which makes it take substantially longer.

 

You should be able to access webmail/cPanel on Echo for the time being but I cannot promise it will not go back offline unexpectedly.

Link to comment
Share on other sites

If you have made significant changes to your account that would not be easy to replicate I would suggest logging in now at http://echo.supportedns.com/cpanel -> Backups -> Full Backups -> Download and Generate a Full Backup.

 

I can't guarantee how long the Echo server will be online before it crashes again and I cannot guarantee the accuracy of the data on the Echo server but, that said, it's better to have some of it than none of it if you need to recreate some of it by hand.

 

I apologize for any trouble this may cause you but we're doing our best to preserve data but due to the nature of how we have to go about this process we're not able to pull over the newest data from the Echo server just yet - we will try to do so once all accounts are restored on alternate hardware.

Link to comment
Share on other sites

We have initiated full cPanel backups to be generated to a standby backup server to preserve as much up-to-date account data as possible before the server goes offline, however, please do not rely upon this. This sytem goes alphabetically so, if for example, your username starts with a 'z' there's a significant chance it won't get to your account in time.
Link to comment
Share on other sites

Are the issues with Echo the reason Softaculous doesn't seem to be available on Echo's cpanel for me?

I am seeing Softaculous listed when I check.

 

I believe we found the issue with the Echo server - our testing indicated the issue was resolved, however, we're still going to take an extra set of backups just to be sure.

Link to comment
Share on other sites

backups downloaded ok. still waiting for a reply for confirmation if server will change from ECHO to KOBOLD

 

Please advise

 

You should remain on the ECHO server now. It appears the issue is resolved. We will update this thread if anything changes.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...