Jump to content
MDDHosting Forums

Major Outage - 09/21/18+ - Client Discussion


KevinD872

Recommended Posts

P1 is back, thank you MDD! I know you're working ****** this and I appreciate you guys and all the great service over the years.

 

I wanted to share this just so that everyone knows that in the world of hosting and IT every provider has issues.

 

So, for my P1 domains I went into GoDaddy to update the new IP in DNS and guess what? There DNS servers are down. That's right, even the mighty GoDaddy has issues!!

http://thearagonagency.com/2018-09-24_0827.png​

http://thearagonagency.com/2018-09-24_0827_001.png

 

If you've been doing this for any amount of time you that this happens. I've been at it since 2000, for 18 years and I've been with many hosting providers. No other has even come close to MDD hosting in service and support. I know when they come back from this they will be even better. I'm sticking around and I hope you do too.

Link to comment
Share on other sites

Ok, in light of Interworld's comment, I guess there is/was a page like that, but I am not able to see it now for the reasons he mentioned. And maybe I never saw it before because of the root directory exception?

If I'm following you correctly, it sounds like you were expecting 404 error pages to be similarly worded to the default page. I believe they only changed the default page. There are 3 or 4 other pages that could be changed to be less scary sounding.

 

MDD: If this wouldn't take any time from performing the backups, would it be possible to replace all of the default pages (or also the 404 page at a minimum) to something closer to the default page you guys put in place of the original?

Link to comment
Share on other sites

I'm with giantjack. Customers are pressuring us to leave MDDHosting and we have 6 client sites still down and all on S2 with no good news to give them. That this will go on all day still is placing us in a very uncomfortable position and the restoration process is indeed painfully slow.

 

At the very least I would like more frequent updates on what is happening like what servers are being restored, etc. Frequent updates is really the least you can do at this point as most of us here are obviously defending you with our own angry clients because of your amazing track record in the past. But with some clients, that can only go so far and we need information to manage expectations. Especially since I had a crazy week planned already with no time to spare and I now have to manage this and field angry client calls when I should really be concentrating on other things...

  • Upvote 1
Link to comment
Share on other sites

My friend that has website + email with mdd has a backup of the website. I/he could open a ticket about a special restore effort for that; but what he really cares about is the email, not the website. I can't see how restoring the website will restore his email config at mdd. So what backup is it that people are talking about that would speed the return of email?

Link to comment
Share on other sites

KevinJones... for me there was a bit of a lag between cPanel and FTP working and the site itself coming up. Also the old 404 page is a good sign as it means your site is coming back up. Keep clearing things and if your site is bookmarked, make sure the http is typed in the browser. This was the scenario for me last night.

Interworld, how long of a lag was there for you? It has been an hour and a half, and I still have no HTTP access. I am wondering in case there might be some other issue, but I don't want to jump the gun and open a ticket if it is not necessary.

 

Again, I don't believe browser caching is an issue here (cleared cache, tried curl)

Link to comment
Share on other sites

I'm with giantjack. Customers are pressuring us to leave MDDHosting and we have 6 client sites still down and all on S2 with no good news to give them. That this will go on all day still is placing us in a very uncomfortable position and the restoration process is indeed painfully slow.

 

At the very least I would like more frequent updates on what is happening like what servers are being restored, etc. Frequent updates is really the least you can do at this point as most of us here are obviously defending you with our own angry clients because of your amazing track record in the past. But with some clients, that can only go so far and we need information to manage expectations. Especially since I had a crazy week planned already with no time to spare and I now have to manage this and field angry client calls when I should really be concentrating on other things...

 

Same here. 8 sites total, 2 mine, S2 and S3.

 

This is definitely not good. Very slow process. 4 days.

Link to comment
Share on other sites

 

 

At the very least I would like more frequent updates on what is happening like what servers are being restored, etc.

 

Updates unfortunately are getting lost amongst all the other posts in this thread.

 

Eric just posted this:

 

Current ETA based on the progress up till now(Eastern Time):

s1 Completed

p1 Completed

r1 Monday, September 24, 2018 at 11:00:00 PM

p2 Monday, September 24, 2018 at 3:00:00 PM

s2 Tuesday, September 25, 2018 at 1:00:00 AM

r2 Tuesday, September 25, 2018 at 9:00:00 AM

s3 Tuesday, September 25, 2018 at 4:00:00 PM

r3 Wednesday, September 26, 2018 at 12:00:00 AM

s4 Wednesday, September 26, 2018 at 4:00:00 AM

r4 Wednesday, September 26, 2018 at 6:00:00 AM

s5 Wednesday, September 26, 2018 at 12:00:00 PM

s0 Wednesday, September 26, 2018 at 12:00:00 PM

Link to comment
Share on other sites

My friend that has website + email with mdd has a backup of the website. I/he could open a ticket about a special restore effort for that; but what he really cares about is the email, not the website. I can't see how restoring the website will restore his email config at mdd. So what backup is it that people are talking about that would speed the return of email?

 

Someone from MDD can confirm, but I believe what they were saying is that you can open a ticket to have a new blank cPanel account created for you and once you do that, you could then configure your email settings. I believe they also said that they would not overwrite anything (i.e. your email stuff) unless you specifically tell them to do so in a ticket. Here is the problem with this: If you use IMAP email, it would likely cause some problems if all of those emails (or any IMAP sub-folders) are suddenly missing.

 

My entire family uses IMAP email through one of my domains and I have two others for business-related things. As much as I would like to restore my email, it is very complicated for me. I also use lots of filters.

 

Edit: The backup they were referring to in this scenario is that if YOU have a local backup of your entire website, you can have them create the cPanel account and then you could upload your own backup (via FTP, etc) to restore your website quicker yourself.

Edited by KevinD872
Link to comment
Share on other sites

 

Current ETA based on the progress up till now(Eastern Time):

s1 Completed

p1 Completed
r1 Monday, September 24, 2018 at 11:00:00 PM
p2 Monday, September 24, 2018 at 3:00:00 PM
s2 Tuesday, September 25, 2018 at 1:00:00 AM
r2 Tuesday, September 25, 2018 at 9:00:00 AM
s3 Tuesday, September 25, 2018 at 4:00:00 PM
r3 Wednesday, September 26, 2018 at 12:00:00 AM
s4 Wednesday, September 26, 2018 at 4:00:00 AM
r4 Wednesday, September 26, 2018 at 6:00:00 AM
s5 Wednesday, September 26, 2018 at 12:00:00 PM
s0 Wednesday, September 26, 2018 at 12:00:00 PM

 

This seems to be a sliding scale. So that one does not need to sift through pages and pages of posts to find your latest recovery time objective, can the current RTO be stickied?

 

A total of 4 to 5 days down - and that is still not certain - for me. Google will start to drop the sites from its index, email not getting through and most smtp servers stop trying after 24 - 48 hours. I've opened a hosting account with another provider to at least get emails up and a few main web pages, but dns propagation takes 24 - 48 hours. In hindsight, I should have done this earlier.

Link to comment
Share on other sites

For people that end up on the defaultwebpage.cgi page on P1, I figured out what was/is wrong (at-least for my site),the IP address of the server changed, and since my DNS records were set with a TTL of 4 hours some users will still have the wrong IP in their local dns cache (or their upstream dns servers) for a couple of more hours, and they end up on the the defaultwebpage.cgi page.

Link to comment
Share on other sites

Thanks Kevin that makes sense. We do have a backup of the website, but I correctly didn't think that was going to help with email. Your reply makes it clear that the half step for email would be made via a new cpanel. I will have to open a ticket with mdd to go over the risks/integration. They've pushed the restore times back and that has me worried.

Link to comment
Share on other sites

I've opened a hosting account with another provider to at least get emails up and a few main web pages, but dns propagation takes 24 - 48 hours. In hindsight, I should have done this earlier.

Hey There,

 

Wherever you host your DNS, lower the TTL as low as possible. Additionally, if you haven't already you should sign up to Cloudflare (it is free). Because your visitors will be going through Cloudflare from here on out, there won't be any secondary propagation for them later. They will keep visiting via Cloudflare, while you inform Cloudflare of where your website has moved to.

 

If I was in your shoes, that is exactly what I would do. :)

Link to comment
Share on other sites

This seems to be a sliding scale. So that one does not need to sift through pages and pages of posts to find your latest recovery time objective, can the current RTO be stickied?

 

A total of 4 to 5 days down - and that is still not certain - for me. Google will start to drop the sites from its index, email not getting through and most smtp servers stop trying after 24 - 48 hours. I've opened a hosting account with another provider to at least get emails up and a few main web pages, but dns propagation takes 24 - 48 hours. In hindsight, I should have done this earlier.

 

 

Since used to work on the IT, it seems to me that a lot of bad decisions were made about how to restore the servers. A lot of time was wasted. Apparently they had no plan about it and they were trying things and at some point they changed their strategy.

 

Anyway the google problem is with me also, started yesterday to get 404 increase notifications from google webmaster tool. And also a huge ammount of email is simply lost. Gone.

 

If i knew that the restore was gone be more and more delayed, i would have switched to something else with a week old backup. But didn't expected that it would take a week to restore everything.

 

And i'm pretty sure after the restore, there would be still some problems to resolve.

Link to comment
Share on other sites

Is there a reason why the restoration sequence was changed? S2 was moved to after P2 in contrast to the previous annoucement. It's not a major difference in time, but neverless that's a detail a bit irritating in a sensible situation like this where people are losing money and/or customers.

Link to comment
Share on other sites


The websites of two major Internet radio stations are waiting on S2 to come back up. They were in the process of being redesigned anyway, and the only real lament I have is not having any backups since the redesign began. But that's on me, and I'm fine with starting over if I have to. The streams of these two stations are handled through a different streaming media host, so they are still up and broadcasting, it's just that the primary portal where the world finds the stations are 404. I've posted on Facebook about it, and that's all I can do.


I've been with MDD for over 6 years now I think it must be and I do not plan on leaving. I think this is all just one big "The World According To Garp" story. The Cessna has just crashed into the bedroom, so I'm buying this house now for sure; i.e., what are the chances this will EVER happen again?


Personally, my biggest concern is how much I've relied on the email accounts attached to my various web properties. I hope I'm not losing mail on the 18-some-odd email accounts attached to all my properties. A few of those emails are receiving a total of about 1k spam emails per day. Maybe the undeliverable bounces back to the spammers will get me off their lists. HA! Keeping optimistic here.
  • Upvote 2
Link to comment
Share on other sites

Interworld, how long of a lag was there for you? It has been an hour and a half, and I still have no HTTP access. I am wondering in case there might be some other issue, but I don't want to jump the gun and open a ticket if it is not necessary.

 

Again, I don't believe browser caching is an issue here (cleared cache, tried curl)

 

Not sure if the lag time wasn't caused by me using my bookmark... it didn't have the https:// in it. When I typed my url in by hand, my site came right up. Could have been a coincidence.

Link to comment
Share on other sites

 

Updates unfortunately are getting lost amongst all the other posts in this thread.

 

Eric just posted this:

 

Current ETA based on the progress up till now(Eastern Time):

s1 Completed

p1 Completed

r1 Monday, September 24, 2018 at 11:00:00 PM

p2 Monday, September 24, 2018 at 3:00:00 PM

s2 Tuesday, September 25, 2018 at 1:00:00 AM

r2 Tuesday, September 25, 2018 at 9:00:00 AM

s3 Tuesday, September 25, 2018 at 4:00:00 PM

r3 Wednesday, September 26, 2018 at 12:00:00 AM

s4 Wednesday, September 26, 2018 at 4:00:00 AM

r4 Wednesday, September 26, 2018 at 6:00:00 AM

s5 Wednesday, September 26, 2018 at 12:00:00 PM

s0 Wednesday, September 26, 2018 at 12:00:00 PM

 

I see those, what I'd like to know is what is being done right now and the approximate progress % and frequent updates of the same. Our clients are requesting more info and I have none to give them.

Link to comment
Share on other sites

Is there a reason why the restoration sequence was changed? S2 was moved to after P2 in contrast to the previous annoucement. It's not a major difference in time, but neverless that's a detail a bit irritating in a sensible situation like this where people are losing money and/or customers.

We are simply doing things as quickly as we can. We had a short lapse in the ability to move the disks from Slow Backup to Fast Backup and in this lapse we went ahead and copied the smallest server which is S2. We did this in lieu of just sitting around doing nothing.

If P2 was the smallest server it still would have been copied at that point.

This is one of the reasons the ETAs are only estimates - because we are doing our best to predict how long it will take to copy and restore each servers' backups and this changes based upon the actual data being copied. For example a server with 1TB of data usage and 150,000,000 files is going to take a LOT longer to copy than a server with 4TB of data and 25,000,000 files.

If you have issues with your site or account after the restoration you will need to open a support ticket. While I wish we could keep up with individual issues here on these forums it's not feasible. We have extra staff working on the helpdesk and we're doing our absolute best to keep up considering the ticket load.

If your site is restored and you are seeing a cPanel error page there is a good chance your account is not on the same IP as it was and you're using third party DNS. If you log into your cPanel you can see your new site IP in the status bar or under 'Server Information'. You'll want to update this at your DNS. Originally we planned on trying to make sure everybody was put back on their original IPs but the work to do that would double the restoration time or more.

  • Upvote 1
Link to comment
Share on other sites

 

I see those, what I'd like to know is what is being done right now and the approximate progress % and frequent updates of the same. Our clients are requesting more info and I have none to give them.

We're not keeping up with support tickets as it is - if we were to start providing this level of detail on the restore it would mean pulling somebody off of tickets and slowing down our rate of support by a fair margin.

 

I will see if we can't provide some sort of periodic update but the ETA's are for full restoration of each server. If it's feasible to provide additional information we will do so.

Link to comment
Share on other sites

 

The websites of two major Internet radio stations are waiting on S2 to come back up. They were in the process of being redesigned anyway, and the only real lament I have is not having any backups since the redesign began. But that's on me, and I'm fine with starting over if I have to. The streams of these two stations are handled through a different streaming media host, so they are still up and broadcasting, it's just that the primary portal where the world finds the stations are 404. I've posted on Facebook about it, and that's all I can do.
I've been with MDD for over 6 years now I think it must be and I do not plan on leaving. I think this is all just one big "The World According To Garp" story. The Cessna has just crashed into the bedroom, so I'm buying this house now for sure; i.e., what are the chances this will EVER happen again?
Personally, my biggest concern is how much I've relied on the email accounts attached to my various web properties. I hope I'm not losing mail on the 18-some-odd email accounts attached to all my properties. A few of those emails are receiving a total of about 1k spam emails per day. Maybe the undeliverable bounces back to the spammers will get me off their lists. HA! Keeping optimistic here.

 

All I was concerned about was a few of my emails. They created the temp account, I re-created my emails which took about 5 mins and they're working great!

Link to comment
Share on other sites

Wanted to mention, after waiting 2.5 hours after P1 was restored and I had cpanel/ftp, my HTTP still wasn't working. I finally filed a ticket and found out I had to point my dns to a different ip.

 

So while it is good to be patient even after your account is partially restored, it may be well to file a ticket if it is taking a long time to access your webpage, as there may be some other issue at play.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...