Jump to content


Photo

Major Outage - 09/21/18+ - Client Discussion


  • Please log in to reply
419 replies to this topic

#181 KevinJones

KevinJones

    Newbie

  • Members
  • Pip
  • 12 posts

Posted 24 September 2018 - 10:13 AM

KevinJones... for me there was a bit of a lag between cPanel and FTP working and the site itself coming up. Also the old 404 page is a good sign as it means your site is coming back up. Keep clearing things and if your site is bookmarked, make sure the http is typed in the browser. This was the scenario for me last night.

Interworld, how long of a lag was there for you?  It has been an hour and a half, and I still have no HTTP access.  I am wondering in case there might be some other issue, but I don't want to jump the gun and open a ticket if it is not necessary.

 

Again, I don't believe browser caching is an issue here (cleared cache, tried curl)


  • 0

#182 cziv

cziv

    Member

  • Members
  • PipPip
  • 52 posts
  • Gender:Male

Posted 24 September 2018 - 10:16 AM

I'm with giantjack. Customers are pressuring us to leave MDDHosting and we have 6 client sites still down and all on S2 with no good news to give them. That this will go on all day still is placing us in a very uncomfortable position and the restoration process is indeed painfully slow.

 

At the very least I would like more frequent updates on what is happening like what servers are being restored, etc. Frequent updates is really the least you can do at this point as most of us here are obviously defending you with our own angry clients because of your amazing track record in the past. But with some clients, that can only go so far and we need information to manage expectations. Especially since I had a crazy week planned already with no time to spare and I now have to manage this and field angry client calls when I should really be concentrating on other things...

 

Same here. 8 sites total, 2 mine, S2 and S3.

 

This is definitely not good. Very slow process. 4 days.


  • 0

#183 SarisIsop

SarisIsop

    Advancing Member

  • Members
  • PipPipPip
  • 156 posts
  • Gender:Not Telling

Posted 24 September 2018 - 10:20 AM

 

 

At the very least I would like more frequent updates on what is happening like what servers are being restored, etc.

 

Updates unfortunately are getting lost amongst all the other posts in this thread.

 

Eric just posted this:

 

Current ETA based on the progress up till now(Eastern Time):
s1    Completed
p1    Completed
r1    Monday, September 24, 2018 at 11:00:00 PM
p2    Monday, September 24, 2018 at 3:00:00 PM
s2    Tuesday, September 25, 2018 at 1:00:00 AM
r2    Tuesday, September 25, 2018 at 9:00:00 AM
s3    Tuesday, September 25, 2018 at 4:00:00 PM
r3    Wednesday, September 26, 2018 at 12:00:00 AM
s4    Wednesday, September 26, 2018 at 4:00:00 AM
r4    Wednesday, September 26, 2018 at 6:00:00 AM
s5    Wednesday, September 26, 2018 at 12:00:00 PM
s0    Wednesday, September 26, 2018 at 12:00:00 PM


  • 0

#184 KevinD872

KevinD872

    Member

  • Members
  • PipPip
  • 25 posts
  • Gender:Male
  • Location:NW Indiana
  • Interests:Drums & music, motorcycles, computers, programming and gadgets.

Posted 24 September 2018 - 10:22 AM

My friend that has website + email with mdd has a backup of the website. I/he could open a ticket about a special restore effort for that; but what he really cares about is the email, not the website. I can't see how restoring the website will restore his email config at mdd. So what backup is it that people are talking about that would speed the return of email?

 

Someone from MDD can confirm, but I believe what they were saying is that you can open a ticket to have a new blank cPanel account created for you and once you do that, you could then configure your email settings.  I believe they also said that they would not overwrite anything (i.e. your email stuff) unless you specifically tell them to do so in a ticket.  Here is the problem with this:  If you use IMAP email, it would likely cause some problems if all of those emails (or any IMAP sub-folders) are suddenly missing.

 

My entire family uses IMAP email through one of my domains and I have two others for business-related things.  As much as I would like to restore my email, it is very complicated for me.  I also use lots of filters.

 

Edit:  The backup they were referring to in this scenario is that if YOU have a local backup of your entire website, you can have them create the cPanel account and then you could upload your own backup (via FTP, etc) to restore your website quicker yourself.


Edited by KevinD872, 24 September 2018 - 10:25 AM.

  • 0

#185 webeye

webeye

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 24 September 2018 - 10:27 AM

 

Current ETA based on the progress up till now(Eastern Time):
s1    Completed

p1    Completed
r1    Monday, September 24, 2018 at 11:00:00 PM
p2    Monday, September 24, 2018 at 3:00:00 PM
s2    Tuesday, September 25, 2018 at 1:00:00 AM
r2    Tuesday, September 25, 2018 at 9:00:00 AM
s3    Tuesday, September 25, 2018 at 4:00:00 PM
r3    Wednesday, September 26, 2018 at 12:00:00 AM
s4    Wednesday, September 26, 2018 at 4:00:00 AM
r4    Wednesday, September 26, 2018 at 6:00:00 AM
s5    Wednesday, September 26, 2018 at 12:00:00 PM
s0    Wednesday, September 26, 2018 at 12:00:00 PM

 

This seems to be a sliding scale. So that one does not need to sift through pages and pages of posts to find your latest recovery time objective, can the current RTO be stickied?

 

A total of 4 to 5 days down - and that is still not certain - for me. Google will start to drop the sites from its index, email not getting through and most smtp servers stop trying after 24 - 48 hours. I've opened a hosting account with another provider to at least get emails up and a few main web pages, but dns propagation takes 24 - 48 hours. In hindsight, I should have done this earlier.


  • 0

#186 LazyDodo

LazyDodo

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 24 September 2018 - 10:27 AM

For people that end up on the defaultwebpage.cgi page on P1, I figured out what was/is wrong (at-least for my site),the IP address of the server changed, and since my DNS records were set with a TTL of 4 hours some users will still have the wrong IP in their local dns cache (or their upstream dns servers) for a couple of more hours, and they end up on the the defaultwebpage.cgi page. 


  • 0

#187 spocklikesspores

spocklikesspores

    Newbie

  • Members
  • Pip
  • 5 posts

Posted 24 September 2018 - 10:33 AM

Thanks Kevin that makes sense. We do have a backup of the website, but I correctly didn't think that was going to help with email. Your reply makes it clear that the half step for email would be made via a new cpanel. I will have to open a ticket with mdd to go over the risks/integration. They've pushed the restore times back and that has me worried.


  • 0

#188 richardmichael

richardmichael

    Newbie

  • Members
  • Pip
  • 3 posts

Posted 24 September 2018 - 10:33 AM

I've opened a hosting account with another provider to at least get emails up and a few main web pages, but dns propagation takes 24 - 48 hours. In hindsight, I should have done this earlier.

Hey There,

 

Wherever you host your DNS, lower the TTL as low as possible.  Additionally, if you haven't already you should sign up to Cloudflare (it is free).  Because your visitors will be going through Cloudflare from here on out, there won't be any secondary propagation for them later.  They will keep visiting via Cloudflare, while you inform Cloudflare of where your website has moved to.

 

If I was in your shoes, that is exactly what I would do.  :)


  • 0

#189 richardmichael

richardmichael

    Newbie

  • Members
  • Pip
  • 3 posts

Posted 24 September 2018 - 10:34 AM

Double post. 


  • 0

#190 cziv

cziv

    Member

  • Members
  • PipPip
  • 52 posts
  • Gender:Male

Posted 24 September 2018 - 10:39 AM

This seems to be a sliding scale. So that one does not need to sift through pages and pages of posts to find your latest recovery time objective, can the current RTO be stickied?

 

A total of 4 to 5 days down - and that is still not certain - for me. Google will start to drop the sites from its index, email not getting through and most smtp servers stop trying after 24 - 48 hours. I've opened a hosting account with another provider to at least get emails up and a few main web pages, but dns propagation takes 24 - 48 hours. In hindsight, I should have done this earlier.

 

 

Since used to work on the IT, it seems to me that a lot of bad decisions were made about how to restore the servers. A lot of time was wasted. Apparently they had no plan about it and they were trying things and at some point they changed their strategy.

 

Anyway the google problem is with me also, started yesterday to get 404 increase notifications from google webmaster tool. And also a huge ammount of email is simply lost. Gone.

 

If i knew that the restore was gone be more and more delayed, i would have switched to something else with a week old backup. But didn't expected that it would take a week to restore everything.

 

And i'm pretty sure after the restore, there would be still some problems to resolve.


  • 0

#191 starki

starki

    Newbie

  • Members
  • Pip
  • 3 posts

Posted 24 September 2018 - 10:43 AM

Is there a reason why the restoration sequence was changed? S2 was moved to after P2 in contrast to the previous annoucement. It's not a major difference in time, but neverless that's a detail a bit irritating in a sensible situation like this where people are losing money and/or customers.


  • 0

#192 KevinJones

KevinJones

    Newbie

  • Members
  • Pip
  • 12 posts

Posted 24 September 2018 - 10:55 AM

FWIW, http://p1.temporary-access.com gives 404 error.


  • 0

#193 djMot

djMot

    Newbie

  • Members
  • Pip
  • 11 posts
  • Gender:Male

Posted 24 September 2018 - 11:09 AM

 
The websites of two major Internet radio stations are waiting on S2 to come back up.  They were in the process of being redesigned anyway, and the only real lament I have is not having any backups since the redesign began.  But that's on me, and I'm fine with starting over if I have to.  The streams of these two stations are handled through a different streaming media host, so they are still up and broadcasting, it's just that the primary portal where the world finds the stations are 404.  I've posted on Facebook about it, and that's all I can do.
 
I've been with MDD for over 6 years now I think it must be and I do not plan on leaving.  I think this is all just one big "The World According To Garp" story.  The Cessna has just crashed into the bedroom, so I'm buying this house now for sure; i.e., what are the chances this will EVER happen again?
 
Personally, my biggest concern is how much I've relied on the email accounts attached to my various web properties.  I hope I'm not losing mail on the 18-some-odd email accounts attached to all my properties.  A few of those emails are receiving a total of about 1k spam emails per day.  Maybe the undeliverable bounces back to the spammers will get me off their lists.  HA!  Keeping optimistic here.

  • 2

#194 interworld

interworld

    Newbie

  • Members
  • Pip
  • 5 posts

Posted 24 September 2018 - 11:09 AM

Interworld, how long of a lag was there for you?  It has been an hour and a half, and I still have no HTTP access.  I am wondering in case there might be some other issue, but I don't want to jump the gun and open a ticket if it is not necessary.

 

Again, I don't believe browser caching is an issue here (cleared cache, tried curl)

 

Not sure if the lag time wasn't caused by me using my bookmark... it didn't have the https:// in it. When I typed my url in by hand, my site came right up. Could have been a coincidence.


  • 0

#195 Stéphane Bergeron

Stéphane Bergeron

    Newbie

  • Members
  • Pip
  • 3 posts

Posted 24 September 2018 - 11:11 AM

 

Updates unfortunately are getting lost amongst all the other posts in this thread.

 

Eric just posted this:

 

Current ETA based on the progress up till now(Eastern Time):
s1    Completed
p1    Completed
r1    Monday, September 24, 2018 at 11:00:00 PM
p2    Monday, September 24, 2018 at 3:00:00 PM
s2    Tuesday, September 25, 2018 at 1:00:00 AM
r2    Tuesday, September 25, 2018 at 9:00:00 AM
s3    Tuesday, September 25, 2018 at 4:00:00 PM
r3    Wednesday, September 26, 2018 at 12:00:00 AM
s4    Wednesday, September 26, 2018 at 4:00:00 AM
r4    Wednesday, September 26, 2018 at 6:00:00 AM
s5    Wednesday, September 26, 2018 at 12:00:00 PM
s0    Wednesday, September 26, 2018 at 12:00:00 PM

 

I see those, what I'd like to know is what is being done right now and the approximate progress % and frequent updates of the same. Our clients are requesting more info and I have none to give them.


  • 0

#196 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 24 September 2018 - 11:16 AM


Is there a reason why the restoration sequence was changed? S2 was moved to after P2 in contrast to the previous annoucement. It's not a major difference in time, but neverless that's a detail a bit irritating in a sensible situation like this where people are losing money and/or customers.

We are simply doing things as quickly as we can.  We had a short lapse in the ability to move the disks from Slow Backup to Fast Backup and in this lapse we went ahead and copied the smallest server which is S2.   We did this in lieu of just sitting around doing nothing.

 

If P2 was the smallest server it still would have been copied at that point.

 

This is one of the reasons the ETAs are only estimates - because we are doing our best to predict how long it will take to copy and restore each servers' backups and this changes based upon the actual data being copied.  For example a server with 1TB of data usage and 150,000,000 files is going to take a LOT longer to copy than a server with 4TB of data and 25,000,000 files.

 

If you have issues with your site or account after the restoration you will need to open a support ticket.  While I wish we could keep up with individual issues here on these forums it's not feasible.  We have extra staff working on the helpdesk and we're doing our absolute best to keep up considering the ticket load.

 

If your site is restored and you are seeing a cPanel error page there is a good chance your account is not on the same IP as it was and you're using third party DNS.  If you log into your cPanel you can see your new site IP in the status bar or under 'Server Information'.  You'll want to update this at your DNS.  Originally we planned on trying to make sure everybody was put back on their original IPs but the work to do that would double the restoration time or more.


  • 1
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#197 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 24 September 2018 - 11:18 AM

 

I see those, what I'd like to know is what is being done right now and the approximate progress % and frequent updates of the same. Our clients are requesting more info and I have none to give them.

We're not keeping up with support tickets as it is - if we were to start providing this level of detail on the restore it would mean pulling somebody off of tickets and slowing down our rate of support by a fair margin.

 

I will see if we can't provide some sort of periodic update but the ETA's are for full restoration of each server.  If it's feasible to provide additional information we will do so.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#198 tannersj

tannersj

    Newbie

  • Members
  • Pip
  • 1 posts

Posted 24 September 2018 - 11:29 AM

 

 
The websites of two major Internet radio stations are waiting on S2 to come back up.  They were in the process of being redesigned anyway, and the only real lament I have is not having any backups since the redesign began.  But that's on me, and I'm fine with starting over if I have to.  The streams of these two stations are handled through a different streaming media host, so they are still up and broadcasting, it's just that the primary portal where the world finds the stations are 404.  I've posted on Facebook about it, and that's all I can do.
 
I've been with MDD for over 6 years now I think it must be and I do not plan on leaving.  I think this is all just one big "The World According To Garp" story.  The Cessna has just crashed into the bedroom, so I'm buying this house now for sure; i.e., what are the chances this will EVER happen again?
 
Personally, my biggest concern is how much I've relied on the email accounts attached to my various web properties.  I hope I'm not losing mail on the 18-some-odd email accounts attached to all my properties.  A few of those emails are receiving a total of about 1k spam emails per day.  Maybe the undeliverable bounces back to the spammers will get me off their lists.  HA!  Keeping optimistic here.

 

All I was concerned about was a few of my emails. They created the temp account, I re-created my emails which took about 5 mins and they're working great! 


  • 0

#199 KevinJones

KevinJones

    Newbie

  • Members
  • Pip
  • 12 posts

Posted 24 September 2018 - 11:35 AM

Wanted to mention, after waiting 2.5 hours after P1 was restored and I had cpanel/ftp, my HTTP still wasn't working.  I finally filed a ticket and found out I had to point my dns to a different ip.

 

So while it is good to be patient even after your account is partially restored, it may be well to file a ticket if it is taking a long time to access your webpage, as there may be some other issue at play.


  • 0

#200 Plippers

Plippers

    Newbie

  • Members
  • Pip
  • 16 posts
  • Gender:Male

Posted 24 September 2018 - 11:38 AM

Mike / Tim. Maybe you guys could mention the IP changes after each transfer completes (sticky the new IPs somewhere). Seems that's the issue catching the transfered people out at the moment. Especially for those using CloudFlare or other external DNS. Cheers!
  • 1




1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users