Michael D. Posted January 18, 2014 Report Share Posted January 18, 2014 This issue is now marked as resolved. You may have seen this thread regarding file systems errors on the Echo Server: http://forums.mddhosting.com/topic/967-echo-server-file-system-check-required-scheduled-for-sunday-jan-12-2014-at-2-am-et-estimated-maintenance-window-is-2-hours/ [if not, I suggest giving it a quick read].The system is not reporting the file system as damaged, however, the file system is obviously damaged from the standpoint that it's getting to where new folders cannot be created in specific locations. While it is theoretically possible that this is fallout from the previous damage - we suspect there to be a silent corruption issue on the server. Due to this we are going to be migrating accounts off of the server as soon as possible to avoid/prevent further damage to customer data. We are going to do our absolute best to keep IP addresses the same so that this transition is as seamless as possible. There may be a brief period [up to an hour] where you are unable to connect to email or FTP at the server hostname, however, we will also be re-routing this IP to the new temporary server as well to restore this without DNS propagation as well - you may experience SSL Certificate Warnings which you can safely ignore/accept. While we apologize for any trouble this may cause you we do appreciate your understanding while we take immediate action in order to protect your data integrity and your services with us. Without taking this action the server could, at any point in time, go read-only or complely crash/lose data and we want to avoid this if at all possible. We do have backups of the server and will have those on-hand just in case. We are going to be sending out an email to all affected clients regarding this issue. If you have any generalized questions about this issue do feel free to ask them here in this thread as others will likely have the same questions. If you have any account-specific questions by all means please feel free to open a technical support ticket by respond to the email that you will receive shortly or simply by opening a new ticket manually. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted January 19, 2014 Author Report Share Posted January 19, 2014 Unfortunately we're going to have to resort to moving accounts using our backups taken last night at around 3 AM to keep everybody online and to be as seamless as possible. We will then reboot the Echo server, file system check it, and perform a backup of it [if possible] to snag a copy of any data changed since the backup last night. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted January 19, 2014 Author Report Share Posted January 19, 2014 We're about 35% done converting our incremental nightly backups into files that can be restored by cPanel. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted January 19, 2014 Author Report Share Posted January 19, 2014 We're still in the process of restoring backups to an alternative server before re-routing the IPs assigned to the Echo server. This move will be temporary - only until the Echo server issue is diagnosed and resolved permanently. This process should be as seamless as possible as we're not changing any IPs on any accounts. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted January 19, 2014 Author Report Share Posted January 19, 2014 Account backup restorations are approximately 72% completed. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted January 19, 2014 Author Report Share Posted January 19, 2014 Restorations are at approximately 91%. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted January 19, 2014 Author Report Share Posted January 19, 2014 It looks as though Echo has now gone offline due to this issue. Thankfully we have roughly 91% of our customers on the Echo server active on another server. Due to the increased strain of the transfers in addition to the additional load of the Echo accounts [we were hoping not to re-point the IP to Kobold until the transfers were done] your sites will be sluggish until the transfer is completed. We apologize for any trouble this may cause - we've tried to get this done as quickly as possible to avoid this but unfortunately it looks like the Echo server went offline faster than anticipated. Quote Link to comment Share on other sites More sharing options...
gplurn Posted January 19, 2014 Report Share Posted January 19, 2014 Hello, greatly appreciate all the updates. worldofbenbo.com seems to be one of the 9%, plus email appears to be down on all accounts although I can't log in to webmail to verify whether it's just my mail client. The cpanel login does redirect to Kobold. I would appreciate any ETA on being able to login to webmail/cpanel etc. Cheers Quote Link to comment Share on other sites More sharing options...
Michael D. Posted January 19, 2014 Author Report Share Posted January 19, 2014 Hello, greatly appreciate all the updates.Doing my best. worldofbenbo.com seems to be one of the 9%, plus email appears to be down on all accounts although I can't log in to webmail to verify whether it's just my mail client.You are part of the 9% - the server is working on accounts that start with "u" at this time. The cpanel login does redirect to Kobold.Aye - the IP routed to Kobold when Echo went offline. We did just now get Echo back online [admittedly after extensive file system repairs - there is no knowing how long it will be online before it goes back offline or how much data is damaged]. I would suggest snagging a copy of your databases and any recently uploaded/modified files just in case as our last backup is just prior to this incident beginning which is roughly 24 hours ago. I would appreciate any ETA on being able to login to webmail/cpanel etc.As quickly as possible is about as good as I can do. Some accounts are very tiny and some are huge - right now it appears to be working on an account with a LOT of small files which makes it take substantially longer. You should be able to access webmail/cPanel on Echo for the time being but I cannot promise it will not go back offline unexpectedly. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted January 19, 2014 Author Report Share Posted January 19, 2014 If you have made significant changes to your account that would not be easy to replicate I would suggest logging in now at http://echo.supportedns.com/cpanel -> Backups -> Full Backups -> Download and Generate a Full Backup. I can't guarantee how long the Echo server will be online before it crashes again and I cannot guarantee the accuracy of the data on the Echo server but, that said, it's better to have some of it than none of it if you need to recreate some of it by hand. I apologize for any trouble this may cause you but we're doing our best to preserve data but due to the nature of how we have to go about this process we're not able to pull over the newest data from the Echo server just yet - we will try to do so once all accounts are restored on alternate hardware. Quote Link to comment Share on other sites More sharing options...
gplurn Posted January 19, 2014 Report Share Posted January 19, 2014 thanks for the updates Mike Quote Link to comment Share on other sites More sharing options...
Michael D. Posted January 19, 2014 Author Report Share Posted January 19, 2014 We have initiated full cPanel backups to be generated to a standby backup server to preserve as much up-to-date account data as possible before the server goes offline, however, please do not rely upon this. This sytem goes alphabetically so, if for example, your username starts with a 'z' there's a significant chance it won't get to your account in time. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted January 19, 2014 Author Report Share Posted January 19, 2014 All of our backups from the other day are restored and active on Kobold. We are working on pulling over updated data from Echo now to Kobold. Quote Link to comment Share on other sites More sharing options...
gplurn Posted January 19, 2014 Report Share Posted January 19, 2014 thanks for your efforts! Quote Link to comment Share on other sites More sharing options...
Cuga Posted January 20, 2014 Report Share Posted January 20, 2014 Are the issues with Echo the reason Softaculous doesn't seem to be available on Echo's cpanel for me? Quote Link to comment Share on other sites More sharing options...
Michael D. Posted January 20, 2014 Author Report Share Posted January 20, 2014 Are the issues with Echo the reason Softaculous doesn't seem to be available on Echo's cpanel for me?I am seeing Softaculous listed when I check. I believe we found the issue with the Echo server - our testing indicated the issue was resolved, however, we're still going to take an extra set of backups just to be sure. Quote Link to comment Share on other sites More sharing options...
Cuga Posted January 20, 2014 Report Share Posted January 20, 2014 Thanks for the quick response When I click on the Softaculous link it goes to a dead link (404):https://echo.supportedns.com:2083/cpsess2741204711/frontend/x3/softaculous/index.live.php The bottom panel of panel of cpanel that normally displayed all the apps that can be installed is empty (just a single link to the dead page above as well). Quote Link to comment Share on other sites More sharing options...
Michael D. Posted January 20, 2014 Author Report Share Posted January 20, 2014 This should be fixed. Quote Link to comment Share on other sites More sharing options...
Cuga Posted January 20, 2014 Report Share Posted January 20, 2014 This should be fixed. Confirmed, Softaculous seems to be working again. Thank you again for the rapid response and fix Quote Link to comment Share on other sites More sharing options...
afkn Posted January 20, 2014 Report Share Posted January 20, 2014 I believe we found the issue with the Echo server - our testing indicated the issue was resolved, however, we're still going to take an extra set of backups just to be sure.Is it safe to make changes to our files now? Quote Link to comment Share on other sites More sharing options...
kix766 Posted January 20, 2014 Report Share Posted January 20, 2014 hi mike, finding various pages empty on atomonline.com (these are the ones that have been updated lately) downloading backup as we type re emails, can you confirm all will be changed to kobold or will everything remain on echo ? Quote Link to comment Share on other sites More sharing options...
kix766 Posted January 20, 2014 Report Share Posted January 20, 2014 first attempt to download full backup failed... trying it again Quote Link to comment Share on other sites More sharing options...
kix766 Posted January 20, 2014 Report Share Posted January 20, 2014 backups downloaded ok. still waiting for a reply for confirmation if server will change from ECHO to KOBOLD Please advise Quote Link to comment Share on other sites More sharing options...
Tim Posted January 20, 2014 Report Share Posted January 20, 2014 backups downloaded ok. still waiting for a reply for confirmation if server will change from ECHO to KOBOLD Please advise You should remain on the ECHO server now. It appears the issue is resolved. We will update this thread if anything changes. Quote Link to comment Share on other sites More sharing options...
Tim Posted January 20, 2014 Report Share Posted January 20, 2014 Is it safe to make changes to our files now?Yes, it should be fine to make changes now. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.