
Echo Server - Silent Data Corruption - Migrations Pending
#1
Posted 18 January 2014 - 06:37 PM
This issue is now marked as resolved.
You may have seen this thread regarding file systems errors on the Echo Server: http://forums.mddhos...dow-is-2-hours/ [If not, I suggest giving it a quick read].
The system is not reporting the file system as damaged, however, the file system is obviously damaged from the standpoint that it's getting to where new folders cannot be created in specific locations. While it is theoretically possible that this is fallout from the previous damage - we suspect there to be a silent corruption issue on the server.
Due to this we are going to be migrating accounts off of the server as soon as possible to avoid/prevent further damage to customer data. We are going to do our absolute best to keep IP addresses the same so that this transition is as seamless as possible. There may be a brief period [up to an hour] where you are unable to connect to email or FTP at the server hostname, however, we will also be re-routing this IP to the new temporary server as well to restore this without DNS propagation as well - you may experience SSL Certificate Warnings which you can safely ignore/accept.
While we apologize for any trouble this may cause you we do appreciate your understanding while we take immediate action in order to protect your data integrity and your services with us. Without taking this action the server could, at any point in time, go read-only or complely crash/lose data and we want to avoid this if at all possible. We do have backups of the server and will have those on-hand just in case.
We are going to be sending out an email to all affected clients regarding this issue.
If you have any generalized questions about this issue do feel free to ask them here in this thread as others will likely have the same questions. If you have any account-specific questions by all means please feel free to open a technical support ticket by respond to the email that you will receive shortly or simply by opening a new ticket manually.
█ Scalable shared hosting plans in the cloud! Check them out!
█ Highly Available Cloud Shared, Reseller, and VPS
█ http://www.mddhosting.com/
#2
Posted 18 January 2014 - 07:05 PM
Unfortunately we're going to have to resort to moving accounts using our backups taken last night at around 3 AM to keep everybody online and to be as seamless as possible. We will then reboot the Echo server, file system check it, and perform a backup of it [if possible] to snag a copy of any data changed since the backup last night.
█ Scalable shared hosting plans in the cloud! Check them out!
█ Highly Available Cloud Shared, Reseller, and VPS
█ http://www.mddhosting.com/
#3
Posted 18 January 2014 - 07:51 PM
We're about 35% done converting our incremental nightly backups into files that can be restored by cPanel.
█ Scalable shared hosting plans in the cloud! Check them out!
█ Highly Available Cloud Shared, Reseller, and VPS
█ http://www.mddhosting.com/
#4
Posted 18 January 2014 - 10:54 PM
We're still in the process of restoring backups to an alternative server before re-routing the IPs assigned to the Echo server.
This move will be temporary - only until the Echo server issue is diagnosed and resolved permanently.
This process should be as seamless as possible as we're not changing any IPs on any accounts.
█ Scalable shared hosting plans in the cloud! Check them out!
█ Highly Available Cloud Shared, Reseller, and VPS
█ http://www.mddhosting.com/
#5
Posted 19 January 2014 - 10:35 AM
Account backup restorations are approximately 72% completed.
█ Scalable shared hosting plans in the cloud! Check them out!
█ Highly Available Cloud Shared, Reseller, and VPS
█ http://www.mddhosting.com/
#6
Posted 19 January 2014 - 01:33 PM
Restorations are at approximately 91%.
█ Scalable shared hosting plans in the cloud! Check them out!
█ Highly Available Cloud Shared, Reseller, and VPS
█ http://www.mddhosting.com/
#7
Posted 19 January 2014 - 01:37 PM
It looks as though Echo has now gone offline due to this issue. Thankfully we have roughly 91% of our customers on the Echo server active on another server.
Due to the increased strain of the transfers in addition to the additional load of the Echo accounts [we were hoping not to re-point the IP to Kobold until the transfers were done] your sites will be sluggish until the transfer is completed.
We apologize for any trouble this may cause - we've tried to get this done as quickly as possible to avoid this but unfortunately it looks like the Echo server went offline faster than anticipated.
█ Scalable shared hosting plans in the cloud! Check them out!
█ Highly Available Cloud Shared, Reseller, and VPS
█ http://www.mddhosting.com/
#8
Posted 19 January 2014 - 02:12 PM
Hello, greatly appreciate all the updates.
worldofbenbo.com seems to be one of the 9%, plus email appears to be down on all accounts although I can't log in to webmail to verify whether it's just my mail client.
The cpanel login does redirect to Kobold.
I would appreciate any ETA on being able to login to webmail/cpanel etc.
Cheers
#9
Posted 19 January 2014 - 02:32 PM
Doing my best.Hello, greatly appreciate all the updates.
You are part of the 9% - the server is working on accounts that start with "u" at this time.worldofbenbo.com seems to be one of the 9%, plus email appears to be down on all accounts although I can't log in to webmail to verify whether it's just my mail client.
Aye - the IP routed to Kobold when Echo went offline.The cpanel login does redirect to Kobold.
We did just now get Echo back online [admittedly after extensive file system repairs - there is no knowing how long it will be online before it goes back offline or how much data is damaged]. I would suggest snagging a copy of your databases and any recently uploaded/modified files just in case as our last backup is just prior to this incident beginning which is roughly 24 hours ago.
As quickly as possible is about as good as I can do. Some accounts are very tiny and some are huge - right now it appears to be working on an account with a LOT of small files which makes it take substantially longer.I would appreciate any ETA on being able to login to webmail/cpanel etc.
You should be able to access webmail/cPanel on Echo for the time being but I cannot promise it will not go back offline unexpectedly.
█ Scalable shared hosting plans in the cloud! Check them out!
█ Highly Available Cloud Shared, Reseller, and VPS
█ http://www.mddhosting.com/
#10
Posted 19 January 2014 - 02:42 PM
I can't guarantee how long the Echo server will be online before it crashes again and I cannot guarantee the accuracy of the data on the Echo server but, that said, it's better to have some of it than none of it if you need to recreate some of it by hand.
I apologize for any trouble this may cause you but we're doing our best to preserve data but due to the nature of how we have to go about this process we're not able to pull over the newest data from the Echo server just yet - we will try to do so once all accounts are restored on alternate hardware.
█ Scalable shared hosting plans in the cloud! Check them out!
█ Highly Available Cloud Shared, Reseller, and VPS
█ http://www.mddhosting.com/
#11
Posted 19 January 2014 - 02:49 PM
thanks for the updates Mike
#12
Posted 19 January 2014 - 02:51 PM
█ Scalable shared hosting plans in the cloud! Check them out!
█ Highly Available Cloud Shared, Reseller, and VPS
█ http://www.mddhosting.com/
#13
Posted 19 January 2014 - 05:17 PM
All of our backups from the other day are restored and active on Kobold. We are working on pulling over updated data from Echo now to Kobold.
█ Scalable shared hosting plans in the cloud! Check them out!
█ Highly Available Cloud Shared, Reseller, and VPS
█ http://www.mddhosting.com/
#14
Posted 19 January 2014 - 05:40 PM
thanks for your efforts!
#15
Posted 19 January 2014 - 08:07 PM
Are the issues with Echo the reason Softaculous doesn't seem to be available on Echo's cpanel for me?
#16
Posted 19 January 2014 - 08:19 PM
I am seeing Softaculous listed when I check.Are the issues with Echo the reason Softaculous doesn't seem to be available on Echo's cpanel for me?
I believe we found the issue with the Echo server - our testing indicated the issue was resolved, however, we're still going to take an extra set of backups just to be sure.
█ Scalable shared hosting plans in the cloud! Check them out!
█ Highly Available Cloud Shared, Reseller, and VPS
█ http://www.mddhosting.com/
#17
Posted 19 January 2014 - 08:26 PM
Thanks for the quick response
When I click on the Softaculous link it goes to a dead link (404):
https://echo.support.../index.live.php
The bottom panel of panel of cpanel that normally displayed all the apps that can be installed is empty (just a single link to the dead page above as well).
#18
Posted 19 January 2014 - 08:56 PM
█ Scalable shared hosting plans in the cloud! Check them out!
█ Highly Available Cloud Shared, Reseller, and VPS
█ http://www.mddhosting.com/
#19
Posted 19 January 2014 - 09:10 PM
This should be fixed.
Confirmed, Softaculous seems to be working again. Thank you again for the rapid response and fix
#20
Posted 20 January 2014 - 01:46 AM
I believe we found the issue with the Echo server - our testing indicated the issue was resolved, however, we're still going to take an extra set of backups just to be sure.
Is it safe to make changes to our files now?
Also tagged with one or more of these keywords: Resolved
![]() |
Resolved
MDDHosting Announcements →
Server and Network Announcements →
P1 - Remote Inbound MySQL Connections DisabledStarted by Scott, 05 Aug 2018 ![]() |
|
![]() |
|
Resolved
MDDHosting Announcements →
Server and Network Announcements →
S4 Server - Out of Memory ConditionStarted by MikeDVB, 23 Nov 2017 ![]() |
|
![]() |
||
Resolved
MDDHosting Announcements →
Server and Network Announcements →
Degraded performance - P1, S1, S4Started by Scott, 19 Nov 2017 ![]() |
|
![]() |
||
Resolved
MDDHosting Announcements →
General Announcements →
MDDHosting - Phone System OutageStarted by Scott, 05 Nov 2017 ![]() |
|
![]() |
||
Resolved
MDDHosting Announcements →
Server and Network Announcements →
[Zayo Transit Outage] Intermittent Connectivity for Some Customers Tonight - 6:30 PM to 7:45 PM ET.Started by MikeDVB, 30 Oct 2016 ![]() |
|
![]() |
1 user(s) are reading this topic
0 members, 1 guests, 0 anonymous users