Jump to content


Photo

Major Outage - 09/21/18+ - Client Discussion


  • Please log in to reply
419 replies to this topic

#361 SarisIsop

SarisIsop

    Advancing Member

  • Members
  • PipPipPip
  • 155 posts
  • Gender:Not Telling

Posted 26 September 2018 - 08:14 AM

I'm guessing things will take a while to settle down. ?

 

Posts made on my forum since being back on-line have suddenly gone, I have closed the forum down for now.

 

Would anyone know when things will start getting saved again.

 

Thank you.


  • 0

#362 ADC

ADC

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 26 September 2018 - 08:17 AM

Not off hand - does it require any custom ports?

This was my mistake.  I forgot that Friday's orders were missing from the database, so new orders were assigned their order numbers.  Fixed.

 

 

 
@Mike, unless you're going to be offering some serious credits for the outage  I have a hard time justifying a reason to stay.

 

Well, good luck to you if you decide to leave. I've had nothing but trouble with every other host I've been with- hacks, outages... and they frankly didn't give a rip if my site was up or not.  I'll be more careful about my backups from now on, but I don't recall being down once prior since hosting with MDD. This kind of a thing was a yearly occurrence with others.


  • 0

#363 sf2099

sf2099

    Newbie

  • Members
  • Pip
  • 6 posts

Posted 26 September 2018 - 08:58 AM

For whatever is worth to people considering leaving MDD, I would like to say that I will be staying with MDD.

 

Although I don't share the fanboysim from few individuals here, I know that Mike and his team have been solid since becoming their customer in Jan 2013.

 

Yes, no doubt about it.  This was a big, self-inflicted f***up.  BUT, Mike and his team stepped up and owned it and I believe that deserves some credit.

 

And I like his plans on minimizing the risk of these types of catastrophic events from happening again in the future.  I would like to hear more about backup testings in future.

 

As a small business owner myself, I do like supporting other small businesses.  So as long as Mike delivers on his promises, he has my support.

 

Lastly, what happened here does not absolve myself from being more diligent either.  Here is a screenshot of what Mike told me back in 2013 before I became his customer.  A good advice, still in 2018!

 

 

TSNbakq.jpg


  • 2

#364 Dehenderson

Dehenderson

    Member

  • Members
  • PipPip
  • 26 posts
  • Gender:Male
  • Location:Arlington, VA

Posted 26 September 2018 - 09:58 AM

Removed by Request


Edited by MikeDVB, 26 September 2018 - 12:28 PM.
Removed at the request of Dehenderson

  • 3

#365 smikwily

smikwily

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 26 September 2018 - 10:20 AM

Can we get a status update on S4, please?


  • 0

#366 LShoe

LShoe

    Newbie

  • Members
  • Pip
  • 13 posts

Posted 26 September 2018 - 10:26 AM

I had restored from my own backups and my site was up, but I gave the go ahead to overwrite this with the S5 server backup. My site is currently down (with the terrible "Sorry" message". I don't know when it went down, but is this a very short outage while the restore process is actively writing to my site, or is this a long outage while you write the entire contents of the server and then afterwards gradually turn on the accounts?


  • 0

#367 ithelpme2

ithelpme2

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 26 September 2018 - 10:26 AM

@MDD Hosting.  Hello, I'm very familiar with Nimble snapshots.  I work with Nimble storage on a daily basis at work.  I'm the Systems Engineer at work, website hobbyists outside of work.  You mentioned that you keep up to 10 hours worth of snapshots?  Unless you're pressed for space, I think 10 hours is nothing.  Our file servers at work, we snapshot hourly and keep that for years.  It protects us from unknown vulnerabilities in which we may need to get data past the unexpected time frame.


  • 0

#368 digibread

digibread

    Newbie

  • Members
  • Pip
  • 9 posts

Posted 26 September 2018 - 10:31 AM

A few things I've learned in the 18 years of hosting websites:

 

1. Have your own backups stored offsite.

     - I back up databases daily and file system weekly for all sites.

2. Don't be a cheap hosting service. The lower your prices the lower the quality of customers. 

    - I only offer a managed hosting service for a premium price (1000% higher than the cheap hosts). Fewer clients but they are the best.

    - I offer a 50% discount to non-profits (and believe it or not they were the squeakiest wheels during this outage.) so you get my point.

3. Don't host email unless you want to hate life and lose money. Refer them to Google Hosted Apps or Office 365, and if you must resell email try GoDaddy Workspace.

   - I host email for only 3 clients, on MDD and charge for it separately. It is by far and away, what I get most support tickets on. My customers pay extra because they want to get me on the phone. None of them complained throughout this event even being down for 4 days.

   - I also charge separately (hourly) for DNS service and support, SSL service and support, as well as helping setup 3rd party email and supporting it ongoing.

4. In your pre-sales process make it clear that outages happen (this is technology) and you'll be there for them, that you have a plan and you'll always be honest and forthright. (if they don't do business with you because of this. GOOD)

5. If you build websites, build them locally and use a versioning system like Git (my favorite, especially with BitBucket), SVN, etc.. Need a LAMP stack look at Laragon (my favorite), Serverpress Desktop Server, or Local by Flywheel, there is also XAMPP and WAMPP

6. When you find a good hosting provider (like MDD) be loyal and stick with them.

   - I've been an administrator and made mistakes equal to what happened here, albeit on a smaller scale of customers but nonetheless, it sucks and I always had great friends, family and customers that stuck by me and encouraged me through the crap storm.

   - If you are hosting websites and put the onus on the security of your data on ANY third party provider you're a fool and need to get out.

 

Doing the above mitigated my stress to a minimum and my loss to zero through this. I understand that problems like these are relative, it hurts and it's never fun. However when I compare MDD to my prior experiences they have been stellar. Yes they had dramatic process and system failures but don't forget, their last line of defense worked. Not only that but they never give up trying new methods to speed up the process and I'm ultra impressed by the fact that things will wrap up soon.

 

MDD Hosting I'm not going anywhere.


  • 1

#369 Djqueball

Djqueball

    Newbie

  • Members
  • Pip
  • 5 posts

Posted 26 September 2018 - 10:31 AM

Can we get a status update on S4, please?

https://forums.mddho...09242018/page-5


  • 0

#370 smikwily

smikwily

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 26 September 2018 - 10:48 AM

Thanks, but that was posted 4 minutes after I asked for a status update. That post wasn't available when I posted.


  • 0

#371 digibread

digibread

    Newbie

  • Members
  • Pip
  • 9 posts

Posted 26 September 2018 - 11:00 AM

 

To the person who started a crowd funding initiative on behalf of MDDHosting, just let me express that MDDHosting is not the victim, all of the company's clients are.  All of the company's clients who trusted that MDDHosting would professionally manage their websites were injured in one way or the other.  If there is, indeed, a crowd funding initiative, then the recipients of any funds should be MDDHosting's customers, the people who suffered and were inconvenienced.  Here's why -- what happened was not an "act of God." It was not a server failure or a datacenter fire.  

 

What happened was human incompetence ... a inexcusable human mistake ... a self-inflicted error ... or, a "man made disaster" as a lawyer friend calls it.  It should not have happened.  And, liability (responsibility) is with MDDHosting. 

 

Perhaps even worse is the fact that MDDHosting’s backup system - which would have enabled expeditious recovery - had been deactivated …. turned OFF.  Another “man made disaster.”

 

You and I learned about this through at least two extraordinarily lengthy emails from Michael Denney, including details on the incorrect code entered that deleted everything.  As someone who respects Michael, I wish he had not spent time writing those emails and more time on finding a fix.  Additionally, those emails expose the company to untold legal exposure at this point.

 

As for our sites, we are discussing whether to throw in the towel.  Our primary site is BoomerCafe.com, a baby boomer story sharing initiative that was launched in 1999.  Nearly 20 years of equity has been wiped out in less than a week.  I hope Michael and MDDHosting will find a professional and responsible way to make us … ALL of us … whole with our respective damages due to this disaster that was clearly … CLEARLY … the result of human failure.

 

 

Sir, I started the fundraiser and I don't mean any disrespect to you, but you (and anyone else thinking like you) clearly did not read your end user agreement when you signed up. Your Data and Access to it, is Your Responsibility! That goes with any hosting provider out there. Your loss in equity is your fault and yours alone. There are so many ways (for site owners, hosting resellers, etc.) to mitigate what all of us experienced here that there is no excuse. The final responsibility will always and does now stop with you.

 

Human error will always play into anything we do as businesses and most companies could give a $h1t about their customers. MDD is not like that at all and Michael's transparency is refreshing and much appreciated. MDD is a small business with an exceptional owner with a young family and responsibilities to employees. They clearly have responsibilities to their customers and have performed to the best of their abilities given the situation. 

 

I had over 32 customers down for 4 days, including a National Food Chain and not one single loss or complaint like yours. You see, even if MDD was a terrible service provider and didn't have any backup I could have quickly setup new hosting and them all back online within a 24 and the most critical within a few. However, each one trusted me that MDD would come through and they did. 

 

You are correct in this: MDD is not a victim, but sir, neither are you! The fundraisers that piss me off are the ones for political douche bags. And they get untold amounts of support, Ridiculous!

 

Maybe we could all be a little more HUMAN and show some love, patience and understanding for one another. Geez, try and put yourself in their shoes.

 

Anyways, Thank you for giving me another opportunity to post a link to one worthy of starting.

https://www.gofundme...-in-crisis-help

 


  • 2

#372 AMC4x4

AMC4x4

    Newbie

  • Members
  • Pip
  • 24 posts

Posted 26 September 2018 - 11:07 AM

 

To the person who started a crowd funding initiative on behalf of MDDHosting, just let me express that MDDHosting is not the victim, all of the company's clients are.  All of the company's clients who trusted that MDDHosting would professionally manage their websites were injured in one way or the other.  If there is, indeed, a crowd funding initiative, then the recipients of any funds should be MDDHosting's customers, the people who suffered and were inconvenienced.  Here's why -- what happened was not an "act of God." It was not a server failure or a datacenter fire.  

 

What happened was human incompetence ... a inexcusable human mistake ... a self-inflicted error ... or, a "man made disaster" as a lawyer friend calls it.  It should not have happened.  And, liability (responsibility) is with MDDHosting. 

 

Perhaps even worse is the fact that MDDHosting’s backup system - which would have enabled expeditious recovery - had been deactivated …. turned OFF.  Another “man made disaster.”

 

You and I learned about this through at least two extraordinarily lengthy emails from Michael Denney, including details on the incorrect code entered that deleted everything.  As someone who respects Michael, I wish he had not spent time writing those emails and more time on finding a fix.  Additionally, those emails expose the company to untold legal exposure at this point.

 

As for our sites, we are discussing whether to throw in the towel.  Our primary site is BoomerCafe.com, a baby boomer story sharing initiative that was launched in 1999.  Nearly 20 years of equity has been wiped out in less than a week.  I hope Michael and MDDHosting will find a professional and responsible way to make us … ALL of us … whole with our respective damages due to this disaster that was clearly … CLEARLY … the result of human failure.

 

 

1) What happened was actually a series of events, not a single event. It might have seemed like a single event, but it wasn't *backups* that were offline, it was snapshots. The backup server was working, slow as it was, and thank god for that. Otherwise many of us would have been well and truly $crewed. 

2) I agree about the GoFundMe. While many of us are still fans of Michael and MDD, this has cost us as well as them, and while I'm keeping most of my sites with MDD, I'm not donating to the GoFundMe because we are all out something here. I spent money to move one of my sites elsewhere that I had an UpdraftPlus backup for. Regular backups and backup testing are a must for anyone who maintains content and deals with IT. I only had those for some of my sites. That's on me. If I had regular cPanel backups, I would have been back online within 24 hours at most. Again, shared responsibility. I believe the ToS say you are responsible for your own backups.

3) I believe the ToS covers Mike and MDD from "untold legal exposure" because at the end of the day, YOU are responsible for backing up your data.

4) Writing those letters did not take away from time spent on restoration. There was a well-documented bottleneck with the backup server that gave Mike and his crew time to answer tickets, keep us updated, etc. I'm sure if they could have sped that copying process along, they would have been happy to have spent time on restoration rather than taking the time to patiently respond to all our questions.

5) I see your site is up, with content as recent as the 18th. How do you figure "Nearly 20 years of equity has been wiped out in less than a week." Just curious


  • 2

#373 Dehenderson

Dehenderson

    Member

  • Members
  • PipPip
  • 26 posts
  • Gender:Male
  • Location:Arlington, VA

Posted 26 September 2018 - 11:35 AM

Just a note in response to "newbie" ... I, too, want to see MDDHosting grow and prosper.  I've used their services many years and referred many people.  But a couple of things:

 

1.  This is 2018, not the mid-2000s when those generalized and generic terms were drafted. 

 

2.  This series of devastating technical errors is more akin to what happened a decade ago. This is 2018.

 

3.  This is not so much about tech stuff, this is about company reputation ... how a person operates their business in 2018.  Perception about a company's reputation is paramount, and Michael's lengthy emails, now reposted on industry websites, won't help MDDHosting's reputation.  When I read them, I thought, "oh, no! Just say the service is down and it will be fixed."

 

4.  Just this week, MDDHosting has disappeared from some rating services as a "top" site host.

 

5.  Why not use your name?


  • 0

#374 AMC4x4

AMC4x4

    Newbie

  • Members
  • Pip
  • 24 posts

Posted 26 September 2018 - 11:45 AM

Just a note in response to "newbie" ... I, too, want to see MDDHosting grow and prosper.  I've used their services many years and referred many people.  But a couple of things:

 

1.  This is 2018, not the mid-2000s when those generalized and generic terms were drafted. 

 

2.  This series of devastating technical errors is more akin to what happened a decade ago. This is 2018.

 

3.  This is not so much about tech stuff, this is about company reputation ... how a person operates their business in 2018.  Perception about a company's reputation is paramount, and Michael's lengthy emails, now reposted on industry websites, won't help MDDHosting's reputation.  When I read them, I thought, "oh, no! Just say the service is down and it will be fixed."

 

4.  Just this week, MDDHosting has disappeared from some rating services as a "top" site host.

 

5.  Why not use your name?

 

Why so hostile? My name is Aaron Clow and I've been an MDD customer since maybe 2013 or so? I have two accounts with them - a corporate account for the company I work for and another account for my personal projects. If you care to look, many of us are "Newbie's" because MDD has had so few issues over the years that we haven't had to post here. "Newbie" is a classification, not my name. Also, when we sign up for a forum account, it doesn't put your name in as your username. You pick one. So I picked one. 

 

If you prefer a hosting company that hides their activities behind a simple "service is down," and that's all the information you require when something untoward happens, I'm sure there are plenty of cheap places you can go for that kind of service. That's not how I do business. I'm forthright with people I speak to, and the reason I'm still with MDD is precisely because they do business this way. Look at how many people are still here and who say they aren't leaving. Why do you think that is?

 

So what's your name? Do people call you "Dehenderson?"

 

Cheers 


  • 3

#375 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 26 September 2018 - 11:46 AM

@Michael Denney

 

Restoring from a snapshot taken 45 min ago, is not like a simple reboot.  You will loose all emails from the last 45 min.

I am talking about downtime incurred, I did say that some data could be lost from up to the last hour.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#376 djMot

djMot

    Newbie

  • Members
  • Pip
  • 11 posts
  • Gender:Male

Posted 26 September 2018 - 11:49 AM

Just a note in response to "newbie" ... 

 

His name isn't "newbie" just as yours isn't "member"


  • 0

#377 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 26 September 2018 - 11:50 AM

We just corrected a storage setting network wide and performance has improved quite a bit.  Our storage is fast enough we should not be relying on the Linux Kernel for I/O queuing and we were - so we turned that back off [as it should have been].


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#378 Bookproc

Bookproc

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 26 September 2018 - 12:11 PM

I do NOT want to see "Service is down and will be fixed." I DO want to see a company being transparent and providing information along the way, including ETAs, if and when they are feasible, and explanations of changes in ETAs. In short, MDD provided all the information I wanted, perhaps more than I needed--but more is always better than less for me--and much much more than I could have expected in the circumstances. This was all deeply reassuring.

 

I have been with hosting providers that did the "It's down, it will be fixed" route, and *never again* is all I can say, because the corollary, though not expressed in precisely these terms, is generally "Now shut up and leave us alone, that's all we're going to tell you, and you're just going to have to wait it out without having a clue what's going on or how long it will take. Sucks to be you." I'm stuck with that kind of service with the cable company because their only competitor is just as bad, if not worse. Fortunately, MDD doesn't take that route, and I would never stay with a hosting provider that did

 

Different strokes for different folks.


  • 5

#379 Djqueball

Djqueball

    Newbie

  • Members
  • Pip
  • 5 posts

Posted 26 September 2018 - 12:17 PM

I do NOT want to see "Service is down and will be fixed." I DO want to see a company being transparent and providing information along the way, including ETAs, if and when they are feasible, and explanations of changes in ETAs. In short, MDD provided all the information I wanted, perhaps more than I needed--but more is always better than less for me--and much much more than I could have expected in the circumstances. This was all deeply reassuring.

 

I have been with hosting providers that did the "It's down, it will be fixed" route, and *never again* is all I can say, because the corollary, though not expressed in precisely these terms, is generally "Now shut up and leave us alone, that's all we're going to tell you, and you're just going to have to wait it out without having a clue what's going on or how long it will take. Sucks to be you." I'm stuck with that kind of service with the cable company because their only competitor is just as bad, if not worse. Fortunately, MDD doesn't take that route, and I would never stay with a hosting provider that did

 

Different strokes for different folks.

 

 

i 100% agree with this....

MDD shows sincerity this is the business ill stay with 


  • 0

#380 digibread

digibread

    Newbie

  • Members
  • Pip
  • 9 posts

Posted 26 September 2018 - 12:24 PM

I do NOT want to see "Service is down and will be fixed." I DO want to see a company being transparent and providing information along the way, including ETAs, if and when they are feasible, and explanations of changes in ETAs. In short, MDD provided all the information I wanted, perhaps more than I needed--but more is always better than less for me--and much much more than I could have expected in the circumstances. This was all deeply reassuring.

 

I have been with hosting providers that did the "It's down, it will be fixed" route, and *never again* is all I can say, because the corollary, though not expressed in precisely these terms, is generally "Now shut up and leave us alone, that's all we're going to tell you, and you're just going to have to wait it out without having a clue what's going on or how long it will take. Sucks to be you." I'm stuck with that kind of service with the cable company because their only competitor is just as bad, if not worse. Fortunately, MDD doesn't take that route, and I would never stay with a hosting provider that did

 

Different strokes for different folks.

 

Or blow smoke up your ***!

Also, 100% agree. 


  • 0




3 user(s) are reading this topic

0 members, 3 guests, 0 anonymous users