fjpoblam Posted September 22, 2018 Report Share Posted September 22, 2018 @Leah2: Thanks! Quote Link to comment Share on other sites More sharing options...
interworld Posted September 23, 2018 Report Share Posted September 23, 2018 Mike... I haven't bothered to open a support ticket in the spirit of that it would only serve to help slow down the restoration process for everyone. That being said, would opening up a ticket help to get a client's site up any sooner (the squeaky wheel gets the grease theory)? My sites can wait. 1 Quote Link to comment Share on other sites More sharing options...
hippy Posted September 23, 2018 Report Share Posted September 23, 2018 While i am not happy about my sites being down and losing out on sales and quotes because of it. The transparency from you, of what has happened and the updates to the status of progress being made to restore sites has kept me from blowing my top over it. I been with yall for several years now and to this date, you have been the best hosting company i have ever had in my 17 years of hosting companies used. I am completely confident you will continue to be after this disaster has been fixed. One of the few complaints i have over this though is the landing page on my site now. Would be nice if it didnt make me look like a dumbass who didnt know how to configure a website. It was a SHTF event and overall i am satisfied with the way it has been handled. 1 Quote Link to comment Share on other sites More sharing options...
glennk Posted September 23, 2018 Report Share Posted September 23, 2018 Would it be possible to speak to the percentage of sites that are complete vs those that are left? And what servers are being worked on? 1 Quote Link to comment Share on other sites More sharing options...
npad69 Posted September 23, 2018 Report Share Posted September 23, 2018 I have about 4 accounts being hosted at MDD but I'm only worrying about 2 of them. The first one is a government website I am maintaining and good thing the downtime happened on a weekend since the account is being used primarily as their internal system to move things around in their workplace so I hope it gets restored completely before 8am monday (+800). The second one is used primarily to process hundreds of thousands of profile data, this is probably the one my client is going to lose a significant amount of money with since they had to pay a team of data encoders their daily wage and couldn't do anything due to the downtime. I just hope the data backup are as latest as possible since my client would have to pay the team again to redo the stuff they have already done. I know this isn't helping the MDD team restore accounts faster and I'm aware they are doing the best and as honestly as they can, Sorry I just needed to vent out some stuff to ease out my worries. I know we'll just have to wait this out and hope for the best. Well at least nobody died right? Quote Link to comment Share on other sites More sharing options...
Brad Posted September 23, 2018 Report Share Posted September 23, 2018 I'm finding it so hard to be chill about this down time. After 24 hours I was still relatively relaxed about the whole ordeal but now this is leaving me with a bad feeling. The worst is that the error page guests are seeing make us webmasters look bad. MDD should be investing some time into maintaining it's clients reputations in the face of potential traffic being lost. Which is a small ask. Instead of a giant:"SORRY!""If you are the owner of this website, please contact your hosting provider"error page it would be great if it instead briefly explained the situation and excused us as webmasters from the RFO. MDD should take a bit of a hit for this rather than us. Quote Link to comment Share on other sites More sharing options...
elatum Posted September 23, 2018 Report Share Posted September 23, 2018 Thank you for the update. Quote Link to comment Share on other sites More sharing options...
Mooball Posted September 23, 2018 Report Share Posted September 23, 2018 My site is great. Redirects to some random other staging site of another domain. That domain when I type it in works perfectly and doesn't redirect to mine... Pretty embarrassing. Quote Link to comment Share on other sites More sharing options...
cziv Posted September 23, 2018 Report Share Posted September 23, 2018 Can we have please some update with the % of accounts restored ? What worries me more is the emails lost. If this continues for an other 24 hours, and tomorrow (Monday), i will be in akward situation since all my email is based in one domain here. Quote Link to comment Share on other sites More sharing options...
bobptz Posted September 23, 2018 Report Share Posted September 23, 2018 Is it possible (and easy) to get a notification email when our account is restored? After we reveive this email we can make thorow examinations on our sites and report any problems to you. Otherwise, if I see some pages working and some errors elswhere, I may assume that the restoration or the final setup is not finished yet by you. Quote Link to comment Share on other sites More sharing options...
BuckoA51 Posted September 23, 2018 Report Share Posted September 23, 2018 I had hoped to wake up and find my site up again today but no luck... My websites are my only source of income, I'm losing serious money while this outage continues Quote Link to comment Share on other sites More sharing options...
Michael D. Posted September 23, 2018 Report Share Posted September 23, 2018 Is it possible (and easy) to get a notification email when our account is restored? After we reveive this email we can make thorow examinations on our sites and report any problems to you. Otherwise, if I see some pages working and some errors elswhere, I may assume that the restoration or the final setup is not finished yet by you.We are trying to notify everybody but we are moving thousands of accounts and silly enough when one is finished restoring it simply disappears from the list. We have installed a set of solid state drives into the backup server and are in the process of doing a direct in-server copy of the S1 Server backups to these solid state drives. Once this copy is completed these drives will be pulled out and moved into another server for restoration. This should allow us to cut the restoration time of the S1 server by up to 75%. Once the S1 server is done copying to these SSDs - we will be doing the same with the next server. Meanwhile we are continuing to do account-level restorations to bring as many people online as quickly as we can. Sorry for the lack of updates - I did manage to get a few hours of sleep. Quote Link to comment Share on other sites More sharing options...
cziv Posted September 23, 2018 Report Share Posted September 23, 2018 This is a simple question. After one day and a half of outage, how many % of the accounts or data has been restored ? Quote Link to comment Share on other sites More sharing options...
Decio Posted September 23, 2018 Report Share Posted September 23, 2018 Can we have please some update with the % of accounts restored ? What worries me more is the emails lost. If this continues for an other 24 hours, and tomorrow (Monday), i will be in akward situation since all my email is based in one domain here. Yes, a rough % would be nice. About emails, just an idea: Redirect emails to a trial google apps account, for the time being. It you´re using cloudflare it should be very easy, if not, could make this extra step of configuring cloudflare for your domain. Quote Link to comment Share on other sites More sharing options...
Laimonas Posted September 23, 2018 Report Share Posted September 23, 2018 Sorry guys, I can't understand one thing : why can't MDD place info: site i down due to MDD Hosting failure. All people think my site was hacked or my webmaster made something wrong and my phone is ringing non stop. Quote Link to comment Share on other sites More sharing options...
cziv Posted September 23, 2018 Report Share Posted September 23, 2018 Actually it is NOT showing Cpanel but "Your Connection is NOT SAFE, the owner has not ... bla bla bla". Cpanel shows IF and After you select to move on accepting a few things. Firefox that is. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted September 23, 2018 Report Share Posted September 23, 2018 Actually it is NOT showing Cpanel but "Your Connection is NOT SAFE, the owner has not ... bla bla bla". Cpanel shows IF and After you select to move on accepting a few things. Firefox that is.Unfortunately there's no way to avoid the SSL error short of installing an SSL on the server for every client domain - and without the client domains being restored - we have no way of having a full list of those domains. Furthermore if we did try to do this - when we do go to restore accounts it's going to create a lot of problems and break things. I wish there was a good solution just like I wish this incident never happened and I wish we were back online already 100%. Unfortunately I can wish all I want and that doesn't change reality - but we are working to restore services as quickly as we can. We're unfortunately limited by the hardware that holds the data presently and we're working on overcoming that. Quote Link to comment Share on other sites More sharing options...
SarisIsop Posted September 23, 2018 Report Share Posted September 23, 2018 Thank you for the Email Michael. I knew you would be working flat out to get this sorted, it's gone wrong but I feel comforted to know that you are still on the ball. So pleased also to read that you will never sell up and that hopefully MDD will keep it's values as a family business when your Sons are old enough. Yes I think you are right, you will undoubtedly lose some customers over this but If it helps, I will be staying with MDD. You have always looked after me and my forum members, even though we are just small fish. As for the money you will lose, I have only just renewed for a year but if you was to offer one of your existing customer discounts I would be happy to add enough for another year if it helps. I still have faith in MDD. Quote Link to comment Share on other sites More sharing options...
cziv Posted September 23, 2018 Report Share Posted September 23, 2018 Michael, i have been with you for 4 years now. I have nothing but good words for you all. Having said that, what i do not like is your backup strategy. I do remember an other instance with you during those 4 years that i was NOT able to actually restore a backup from your backup pool (for some reason you said it was not working. I believe it is still in my old tickets with you). So if your current backup for some reason was corrupted, you would be actually out of business. Do you realize that ? Personally i do a manual Cpanel full backup every Monday (usually) for my two plans and i have automated an ftp transfer to my pc. From my PC there is also a full image to an external drive (using Macrium Reflect, a free for home AND business diamond-software). So i'm covered and many times i restored a full site (after a bad Wordpress update). I suggest your customers do the very same. Anyway, no reason to feel bad and apologize to us so much, just be sure you spend all your time restoring the sites. Tickets can get an automated reply. Good luck !!!! Quote Link to comment Share on other sites More sharing options...
Michael D. Posted September 23, 2018 Report Share Posted September 23, 2018 This backup system is new and was to replace the old system you had issues with. It has worked well and to be honest it's not the software that is the issue this time, but the hardware powering it. The old backup system - it was all software issues. I am aware that if this current backup were to be damaged we'd have nothing to restore. It's not something that hasn't occurred to me but it was not intended to be the primary form of data protection. I do also suggest our customers - and any customer of any hosting provider everywhere - take their own periodic backups. If any clients have a cPanel backup of their account they want restored - we can restore that and get them back online extremely quickly. As it stands right now - I'm just watching a copy bar progress - there's nothing at this moment I can do to assist with or speed up the process of restoring services or I would be doing just that. I will be posting status updates here on this very thread. Quote Link to comment Share on other sites More sharing options...
Gregory Wells Posted September 23, 2018 Report Share Posted September 23, 2018 Ditto for me in all of the previous poster's comments. I'll be a customer as long as I have a need to be so. Quote Link to comment Share on other sites More sharing options...
bobptz Posted September 23, 2018 Report Share Posted September 23, 2018 I am not an admin, but here are some thoughts. Please don;t laugh if they are techically impossible. I initially thought it might be a good idea to physically transfer the backup server from Phoenix to Denver. Then I saw that they are 943 km flying distance, or 1321.36 km by car. What if you had created servers in Phoenix, where the backup server is located and restore to them with realy high speed? Would this be possible? Quote Link to comment Share on other sites More sharing options...
Michael D. Posted September 23, 2018 Report Share Posted September 23, 2018 I am not an admin, but here are some thoughts. Please don;t laugh if they are techically impossible. I initially thought it might be a good idea to physically transfer the backup server from Phoenix to Denver. Then I saw that they are 943 km flying distance, or 1321.36 km by car. What if you had created servers in Phoenix, where the backup server is located and restore to them with realy high speed? Would this be possible?The issue wasn't that the distance slowed things down. We can transfer data at the rate of gigabytes per second from Phoenix to Denver. It's the backup hardware itself that is causing problems. We're copying the data off of this backup server in the fastest way possible and moving that new secondary copy of the data to a server with the capability to handle the data at the speed we need to handle it for maximum restoration speed. This step would have to happen whether or not both servers were in the same location to begin with or not. We're planning for 50% of services to be restored within 24 hours and 100% by 48. Based upon our estimations it should take a fair bit less than this but we don't want to promise a short time and end up taking longer if we can help it. Quote Link to comment Share on other sites More sharing options...
Kraken Posted September 23, 2018 Report Share Posted September 23, 2018 Is there any advantage whatsoever to opening a support ticket? I'm losing money while my store is offline so it would be nice to get into a priority queue, if you have one. I think I'm on server NS1, but my info might be outdated. I don't want to make extra work for anybody if a ticket would be irrelevant at this point. I appreciate your open communication, I've been a satisfied customer for many years, and I won't hold this against you. But I sure would like to get my store back ASAP, and I'll need to know how old the database is. Quote Link to comment Share on other sites More sharing options...
Laimonas Posted September 23, 2018 Report Share Posted September 23, 2018 I am on S1 and my site is still down. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.