Jump to content


Photo

Major Outage - 09/21/18+ - Client Discussion


  • Please log in to reply
419 replies to this topic

#101 Leah2

Leah2

    Member

  • Members
  • PipPip
  • 29 posts
  • Gender:Not Telling

Posted 23 September 2018 - 06:48 PM

Hi Mike,

Just wanted to say I came to MDD back in 2010 & brought all my clients with me as I've been super happy with MDD's support. It's always a shock to the system when I have to deal with other hosts. I'm not super happy with the situation as it is a nail biter, but mistakes happen as we all know being in the tech business. Thank you so much for keeping everybody in the loop. & for changing the error message, it made me & everyone else look totally stupid.

 

I do have a question - would you recommend going forward we make sure we use the Cloudflare option in cpanel? Do you recommend cname or nameserver. I want to make a change to my system as I have frequent backups mandated but need to add a further layer of protection for my clients (some uncomfortable conversations about how I am going to protect them in the future - & here I am preaching to the choir...).

Thanks!


  • 0

Electronic Logic Concepts

 

“What is Your Digital Strategy? Websites Built With SEO First Practices”

 

www.ELC-SEO.com


#102 jsolo

jsolo

    Newbie

  • Members
  • Pip
  • 14 posts
  • Gender:Male

Posted 23 September 2018 - 06:49 PM

 

If we restore your backup - it's up to you if you want us to overwrite it or not.

 

 

Understood.  That's fine. Go ahead over write (or erase everything first then restore, which ever will bring it back to the same state when the backup was taken.)


  • 0

#103 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 September 2018 - 06:49 PM

The cname option should work even if your account's IP changes I believe - but it's been a while since I've messed with it as most use the nameservers.


  • 1
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#104 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 September 2018 - 06:50 PM

 

Understood.  That's fine. Go ahead over write (or erase everything first then restore, which ever will bring it back to the same state when the backup was taken.)

I wish I could do it directly from the forums but we'll need it in a ticket and we'll get to it as quickly as we can.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#105 jsolo

jsolo

    Newbie

  • Members
  • Pip
  • 14 posts
  • Gender:Male

Posted 23 September 2018 - 07:37 PM

I wish I could do it directly from the forums but we'll need it in a ticket and we'll get to it as quickly as we can.

 

Yes sir.  Ticket #657811.  Thanks


  • 0

#106 AMC4x4

AMC4x4

    Newbie

  • Members
  • Pip
  • 24 posts

Posted 23 September 2018 - 07:40 PM

At least you have a process now and can see the progress! Thanks for keeping us up to date. 


  • 0

#107 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 September 2018 - 07:48 PM

At least you have a process now and can see the progress! Thanks for keeping us up to date. 

The old backup server is so inconsistent every ETA we calculated varied wildly so I wasn't comfortable sharing that information and know it's quite likely inaccurate.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#108 nada

nada

    Newbie

  • Members
  • Pip
  • 1 posts

Posted 23 September 2018 - 08:10 PM

The old backup server is so inconsistent every ETA we calculated varied wildly so I wasn't comfortable sharing that information and know it's quite likely inaccurate.

 

Kick it to the curb! ;)

 

Hopefully, this is the reason you needed to upgrade equipment and your processes to make sure this NEVER happens again.


  • 0

#109 Leah2

Leah2

    Member

  • Members
  • PipPip
  • 29 posts
  • Gender:Not Telling

Posted 23 September 2018 - 09:07 PM

Latest ETA Calculations:

 

s1    Sunday, September 23, 2018 at 11:59:00 PM
p1    Monday, September 24, 2018 at 4:00:00 AM
r1    Monday, September 24, 2018 at 4:00:00 PM
s2    Monday, September 24, 2018 at 4:00:00 PM
p2    Monday, September 24, 2018 at 3:00:00 PM
r2    Tuesday, September 25, 2018 at 2:00:00 AM
s3    Tuesday, September 25, 2018 at 6:00:00 AM
r3    Tuesday, September 25, 2018 at 2:00:00 PM
s4    Tuesday, September 25, 2018 at 5:00:00 PM
r4    Tuesday, September 25, 2018 at 6:00:00 PM
s5    Tuesday, September 25, 2018 at 10:00:00 PM
s0    Tuesday, September 25, 2018 at 10:00:00 PM

 

Thank you! For now clients have been understanding but patience is wearing thin so this is super helpful.


  • 0

Electronic Logic Concepts

 

“What is Your Digital Strategy? Websites Built With SEO First Practices”

 

www.ELC-SEO.com


#110 LShoe

LShoe

    Newbie

  • Members
  • Pip
  • 13 posts

Posted 23 September 2018 - 09:12 PM

I have a request to everyone participating in this forum thread: Please keep in mind that if Mike is responding to you, he can't be addressing server issues or answering tickets, despite his valiant attempts to multi-task with little sleep. Some of the questions on here, like, "Are you going to give me a refund?", or "What do you recommend for my backups going forward?" IMO should wait. Even though you are justifiably anxious and participating here helps, your questions are not more urgent than the team getting to tickets (mine and others), helping people figure out how to restore their sites, and continuing to brainstorm with his team on how to improve ETAs. 

 

Seeing my server with an ETA of Thursday (even if that is a worst case) certainly does challenge my resolve to be empathetic and calm about all this - but I will stick with it even if it is with clammy hands and shortened breaths, since I know that Mike and team are doing the best they can and are being honorably transparent, and everyone makes mistakes, including me. (I'm just lucky that the weight of the online business world isn't on my shoulders when I have to clean up from mine.)

 

Thanks for hearing me out.


  • 0

#111 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 September 2018 - 09:17 PM

I have a request to everyone participating in this forum thread: Please keep in mind that if Mike is responding to you, he can't be addressing server issues or answering tickets, despite his valiant attempts to multi-task with little sleep. Some of the questions on here, like, "Are you going to give me a refund?", or "What do you recommend for my backups going forward?" IMO should wait. Even though you are justifiably anxious and participating here helps, your questions are not more urgent than the team getting to tickets (mine and others), helping people figure out how to restore their sites, and continuing to brainstorm with his team on how to improve ETAs. 
 
Seeing my server with an ETA of Thursday (even if that is a worst case) certainly does resolve to be empathetic and calm about all this - but I will stick with it even if it is with clammy hands and shortened breaths, since I know that Mike and team are doing the best they can and are being honorably transparent, and everyone makes mistakes, including me. (I'm just lucky that the weight of the online business world isn't on my shoulders when I have to clean up from mine.)
 
Thanks for hearing me out.

I appreciate what you've had to say.

That said ETA's were just updated and the latest 100% restoration point is presently calculated as Tuesday at 10PM. It's two posts before yours :).
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#112 LShoe

LShoe

    Newbie

  • Members
  • Pip
  • 13 posts

Posted 23 September 2018 - 09:21 PM

I appreciate what you've had to say.

That said ETA's were just updated and the latest 100% restoration point is presently calculated as Tuesday at 10PM. It's two posts before yours :).

 

Great to hear, Mike ... I was typing as you posted. Still folks, hold off on the non-urgent, if only for my sanity. ;-)


  • 0

#113 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 September 2018 - 09:22 PM

Great to hear, Mike ... I was typing as you posted. Still folks, hold off on the non-urgent, if only for my sanity. ;-)

Still doing our best to try and catch up with tickets. Today was a lot more busy - several thousand new tickets opened many of them getting 2 or 3 replies.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#114 Rhody401

Rhody401

    Newbie

  • Members
  • Pip
  • 12 posts
  • Gender:Male
  • Location:Providence, RI USA
  • Interests:Consultant, Contractor, Beta Tester, IT Director.

Posted 23 September 2018 - 09:39 PM

You're all doing a heck of a great job to recover. Your efforts and transparency are very appreciated!


  • 0

#115 fjpoblam

fjpoblam

    Member

  • Members
  • PipPip
  • 63 posts

Posted 23 September 2018 - 10:14 PM

Thanks for the estimates. (I don’t know your time zone: maybe the same as my MDT). I know you’re swamped with tickets, so I won’t open one. I’m on s1 and I see ftp is already available and one of my add-ons (oddly enough) is online but my main domain and the other add-on are getting IP errors. An oddity, TBS. I’ll wait it out…

Good luck, folks.
  • 0

#116 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 September 2018 - 10:49 PM

Our time zone is Eastern Time :).
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#117 KevinD872

KevinD872

    Member

  • Members
  • PipPip
  • 25 posts
  • Gender:Male
  • Location:NW Indiana
  • Interests:Drums & music, motorcycles, computers, programming and gadgets.

Posted 23 September 2018 - 10:49 PM

I have a request to everyone participating in this forum thread: Please keep in mind that if Mike is responding to you, he can't be addressing server issues or answering tickets, despite his valiant attempts to multi-task with little sleep. Some of the questions on here, like, "Are you going to give me a refund?", or "What do you recommend for my backups going forward?" IMO should wait. Even though you are justifiably anxious and participating here helps, your questions are not more urgent than the team getting to tickets (mine and others), helping people figure out how to restore their sites, and continuing to brainstorm with his team on how to improve ETAs.

 

While I'm sure they have been bombarded with support tickets, honestly I think this thread has been pretty quiet compared to what it could be.  The few complaints we have seen are completely understandable but the fact that most posts have been pretty supportive of the efforts to fix this mess speaks volumes for how great MDDHosting has been for many of us clients over the years; Especially when it would seem that human nature typically results in more people complaining than praising in online forums.


  • 1

#118 Plippers

Plippers

    Newbie

  • Members
  • Pip
  • 16 posts
  • Gender:Male

Posted 23 September 2018 - 10:49 PM

Why is S2 so slow? It was something like 4th in line for restore. Now dead last?

*edit* Mike, looks like you deleted the revised estimate with Wednesday mentioned?
  • 0

#119 jsolo

jsolo

    Newbie

  • Members
  • Pip
  • 14 posts
  • Gender:Male

Posted 23 September 2018 - 10:55 PM

How long does it take to do a restore after the backup file has been manually uploaded to the site?

 

Ticket notes from 21:09 EDT indicate restore is in progress.  As of now, 23:55 EDT, site is still inaccessible.

 

Ticket #657811


  • 0

#120 Plippers

Plippers

    Newbie

  • Members
  • Pip
  • 16 posts
  • Gender:Male

Posted 23 September 2018 - 10:58 PM

Thanks Eric.
  • 0




4 user(s) are reading this topic

0 members, 4 guests, 0 anonymous users