Jump to content


Photo

Major Outage - 09/21/18+ - Client Discussion


  • Please log in to reply
419 replies to this topic

#341 digibread

digibread

    Newbie

  • Members
  • Pip
  • 9 posts

Posted 25 September 2018 - 04:33 PM

OK, dumb question time. I can get into my site through cPanel and FTP, and I can access them through browsers on other computers but not my primary. I've restarted the browser, cleared cache and cookies and still hit the error pages. What minor thing am I missing?

Hey Chris, if you're on windows you might try a "command prompt" and type in this without the quotes. "ipconfig /flushdns" a restart will likely do the same. I hope that helps.


  • 0

#342 Djqueball

Djqueball

    Newbie

  • Members
  • Pip
  • 5 posts

Posted 25 September 2018 - 04:41 PM

Hey Chris, if you're on windows you might try a "command prompt" and type in this without the quotes. "ipconfig /flushdns" a restart will likely do the same. I hope that helps.


if you behind a router rebooting the router sometimes resolves this.

another you can do is just change your ISP DNS in your local PC network settings to like google DNS or opendns
and see if you can resolve your site on your machine.

after 72 hours you can go back to your defaults


if you dont make any changes..only thing that is gonna work is
DNS propagation time takes 24 to 72 hours to resolve world wide when a change is made.

dont worry nothing wrong with your pc.
 


  • 1

#343 Shell

Shell

    Grand Poobahess

  • Members
  • Pip
  • 4 posts
  • Gender:Not Telling
  • Location:Metro Vancouver, BC Canada

Posted 25 September 2018 - 05:09 PM

thank you for the quick response.

 

I have to take the motorhome - currently living quarters - in for service tomorrow (2 month delay for appt) and was stressing a bit that I'd have to deal with MX entries for a couple of clients.  Ok, Freaking a bit to be honest

 

Shelley

 

 

MX Records are saved in the backups we have and should be restored as they were.  I actually went into our backup system and got the Outlook365 MX Records for one of our clients yesterday.


  • 0

#344 ADC

ADC

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 25 September 2018 - 05:22 PM

Is S3 fully restored? My site is up, but we use Shipstation to import our orders for shipping labels.  Shipstation says that the connection to my website's server is fine. However, I have 3 new orders that should have imported that haven't. Everything worked fine on Thursday.  I tried deleting and re-uploading and re-installing the plug-in that the site uses to communicate with Shipstation, but no avail. Shipstation says that orders should be importing from my site as they are showing no errors to the connection, but they are not.

Any ideas before I open a ticket?


  • 0

#345 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 25 September 2018 - 07:16 PM

Is S3 fully restored? My site is up, but we use Shipstation to import our orders for shipping labels.  Shipstation says that the connection to my website's server is fine. However, I have 3 new orders that should have imported that haven't. Everything worked fine on Thursday.  I tried deleting and re-uploading and re-installing the plug-in that the site uses to communicate with Shipstation, but no avail. Shipstation says that orders should be importing from my site as they are showing no errors to the connection, but they are not.

Any ideas before I open a ticket?

Not off hand - does it require any custom ports?


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#346 ithelpme2

ithelpme2

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 25 September 2018 - 07:31 PM

I'm on S2, using Joomla for all my sites.  S2 is completely restored.  All sites seems to be working.  I haven't checked all components in Joomla but Akeeba backup isn't working for any site.  It complains about missing angie.jpa file.  I also noticed that my disk usage on my account is 1.8 Gig rather than the 4 Gig before this outage.  How strange?  I've already open a ticket and Mike has restored my account but same symptoms.  No big deal, I was able to grab the angie.jpa file from my local xampp install and now Akeep Backup is working but it seems off that perhaps not all files are restored?  Perhaps some file extension types are not included as backup?  I don't know whether anyone else is having similar issues?


  • 0

#347 jsolo

jsolo

    Newbie

  • Members
  • Pip
  • 14 posts
  • Gender:Male

Posted 25 September 2018 - 07:32 PM

I've just lost almost 2 days worth of emails!!

 

My account on s3 was mostly recovered early monday morning to the extent that at least emails worked.  Recovery was done manually from a backup file with most recent data as of 9/21.

 

The ticket for that recovery was flagged as - underlined part added by MDD.

 

#657811 - Create new/empty account - Do not restore over this account.
#657811 - Create new/empty account - Do not restore over this account.
 
As of right now all emails since the restore and now (almost 2 days worth) are gone.  It appears my account was restored again as other uploaded files are gone too.
 
In fact I just received an email that recovery on s3 is complete.
 
I had my doubts about staying, even inquired about the cancellation process (ticket #811489 - still awaiting response).  But this really is making that decision easier.
 
@Mike, unless you're going to be offering some serious credits for the outage  I have a hard time justifying a reason to stay.

  • 0

#348 ninjaclan

ninjaclan

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 25 September 2018 - 07:48 PM

Hi Mike, ETA for S4 full restoration?

 

You wrote:

"Due to a unexpected reboot of the backup server we are going to have to swap s5 and s4 so we can not lose progress made towards s4's restoration"

 

What does it means?


  • 0

#349 Djqueball

Djqueball

    Newbie

  • Members
  • Pip
  • 5 posts

Posted 25 September 2018 - 07:51 PM

 

I've just lost almost 2 days worth of emails!!

 

My account on s3 was mostly recovered early monday morning to the extent that at least emails worked.  Recovery was done manually from a backup file with most recent data as of 9/21.

 

The ticket for that recovery was flagged as - underlined part added by MDD.

 

#657811 - Create new/empty account - Do not restore over this account.
#657811 - Create new/empty account - Do not restore over this account.
 
As of right now all emails since the restore and now (almost 2 days worth) are gone.  It appears my account was restored again as other uploaded files are gone too.
 
In fact I just received an email that recovery on s3 is complete.
 
I had my doubts about staying, even inquired about the cancellation process (ticket #811489 - still awaiting response).  But this really is making that decision easier.
 
@Mike, unless you're going to be offering some serious credits for the outage  I have a hard time justifying a reason to stay.

 


i feel your pain as a customer myself i feel the aggravation and the desire to split to another host
but yanno your gonna have some issues no matter what shared hosting provider you go to and the sad thing is

you never know if the next host is just a reseller host for someones main business which is even MORE of a headache
because you ll submit a ticket then they have to submit a ticket and and and....ya not worth it..

 

 

it all comes down to this..
it is our responsibility to keep our own backups of our precious files and keep our sites tip top shape

our HOST supplies a backup out of curiosity. only a Managed hosting provider is going to watch your site

and do backups and that is a heavy cost and not cheap per month either. (i rather own my own dedicated machine over that but even then its a work load)

you can set cron jobs to do hourly/daily/weekly/monthly backups..
you also want to keep your backups OFF your main host..

 

MDD is a small business they have grown alot since iv been with them..
in my honest opinion i rather stay then go. they watch out for you closer then some high end top tier host provider
that gives you some lame response with EVERYTHING you ask about..

now god forbid but if this happens again im sure alot of us is going to leave. that just means pure carelessness / laziness

 

i dont work for MDD im just a customer sharing my 2 cents.


  • 0

#350 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 25 September 2018 - 07:52 PM

 

I've just lost almost 2 days worth of emails!!

 

My account on s3 was mostly recovered early monday morning to the extent that at least emails worked.  Recovery was done manually from a backup file with most recent data as of 9/21.

 

The ticket for that recovery was flagged as - underlined part added by MDD.

 

#657811 - Create new/empty account - Do not restore over this account.
#657811 - Create new/empty account - Do not restore over this account.
 
As of right now all emails since the restore and now (almost 2 days worth) are gone.  It appears my account was restored again as other uploaded files are gone too.
 
In fact I just received an email that recovery on s3 is complete.
 
I had my doubts about staying, even inquired about the cancellation process (ticket #811489 - still awaiting response).  But this really is making that decision easier.
 
@Mike, unless you're going to be offering some serious credits for the outage  I have a hard time justifying a reason to stay.

 

 

As I told you in the ticket - initially you wanted it overwritten and then you asked for it not to be - so you ended up on both lists.  Admittedly we have to go out of our way to overwrite your account - so when we saw it on that list we did it.  Did we ****** up? Yes.  It's worth noting that we aren't restoring your accounts from our support ticket system - but from a restoration interface on the server itself that does not in any way reference the ticket system.  If we were to reference the ticket system for every restore - we'd have only a few hundred accounts online at this time instead of many thousands.

 

The automated system did not overwrite your account - we manually did it.  We checked the list of accounts to overwrite and then performed the overwrites.  We honestly didn't consider that an account could be on both lists.  It's an oversight - but one that won't affect anyone else because I checked and there are no other duplicates.  Now that we're aware of this potential issue it's something we can avoid.

 

Did we mess up? Absolutely - and I'm sorry for that.

 

2018-09-25_20-41-27.png

 

Hi Mike, ETA for S4 full restoration?

 

You wrote:

"Due to a unexpected reboot of the backup server we are going to have to swap s5 and s4 so we can not lose progress made towards s4's restoration"

 

What does it means?

It means that we were copying the file system image for the backup when the backup server crashed.  We can resume it - but we actually have to start the restore process on the other server to get the resume token.  In short - we moved on to backing up another server as not to lose the time we already sank into backing up S4.

We're looking to be ahead of the total ETA and are hoping to have everybody online by morning.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#351 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 25 September 2018 - 07:55 PM

now god forbid but if this happens again im sure alot of us is going to leave. that just means pure carelessness / laziness

 

  1. It is absolutely not going to happen again.
  2. Even if it did happen again - the changes we've made will allow us to recover the entire network in minutes - without actually having to do any 'restores' or IP changes or anything of the like.

This is due to just one change we've made - and there are dozens - some already implemented and some pending full restoration so we can focus on them.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#352 jsolo

jsolo

    Newbie

  • Members
  • Pip
  • 14 posts
  • Gender:Male

Posted 25 September 2018 - 08:05 PM

 

As I told you in the ticket - initially you wanted it overwritten and then you asked for it not to be - so you ended up on both lists.  Admittedly we have to go out of our way to overwrite your account - so when we saw it on that list we did it.  Did we ****** up? Yes.  It's worth noting that we aren't restoring your accounts from our support ticket system - but from a restoration interface on the server itself that does not in any way reference the ticket system.  If we were to reference the ticket system for every restore - we'd have only a few hundred accounts online at this time instead of many thousands.

 

Did we mess up? Absolutely - and I'm sorry for that.

 

2018-09-25_20-41-27.png

 

Chronologically, yes, initially I did want it restored until I discovered the backup file.  Once that backup file was restored I even confirmed in the ticket that the account won't be touched when the s3 restore happens.  No actual confirmation was given but I can only hope that tickets gets read thoroughly.

 

If you've got two lists, isn't it prudent to make sure someone isn't on both at the same time.

 

I'm stepping away from this... Frustrated enough as it is.


  • 0

#353 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 25 September 2018 - 08:14 PM

 

Chronologically, yes, initially I did want it restored until I discovered the backup file.  Once that backup file was restored I even confirmed in the ticket that the account won't be touched when the s3 restore happens.  No actual confirmation was given but I can only hope that tickets gets read thoroughly.

 

If you've got two lists, isn't it prudent to make sure someone isn't on both at the same time.

 

I'm stepping away from this... Frustrated enough as it is.

Yes, it's prudent to make sure nobody is on the list twice and that's the reason I checked for other duplicates after you brought this to our attention.  While I can't undo what happened in your case, I can learn from it and make sure it doesn't happen again.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#354 Stahlrosen

Stahlrosen

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 25 September 2018 - 08:43 PM

We are back up and running fine on s3. Thank you team MDD for all your hard work.


  • 0

#355 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 25 September 2018 - 09:20 PM

For those worried that something like this could happen again.

We have already enabled snapshots on our storage cluster. We're doing one snapshot every hour and keeping them for 10 hours.

So from a hypothetical standpoint - let's say that this did manage to happen again. We would simply mount a snapshot from before the incident - within the hour before - and boot everything back up.

Total downtime would be - ~5 minutes - for the whole network. Would there be any data loss? Possibly anything written in the preceding hour or less - but nothing compared to the losses of a multi-day outage.

It would look literally like we just shut everything down and booted it back up. No 'restorations', no lost emails, nothing. There's a great chance almost nobody would even notice.

This is something that our storage vendor, StorPool, set up for us immediately upon seeing what had happened. They actually apologized that it was not already set up and said that as a result of our disaster they are going to make sure that it is a default behavior that has to be actively disabled rather than the other way around.

Even with these snapshots and as powerful as they are - we are still going to overhaul our backup servers. We have identified the issues with the present setup that caused restorations to be so slow and already have fixes for those issues planned for once we are fully online and all of our clients are taken care of.

Snapshots are a very powerful tool against data loss and corruption. We actually used them a couple of times on our old storage platform, the Nimble CS500, to recover data on servers when clients made big mistakes themselves.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#356 bobptz

bobptz

    Newbie

  • Members
  • Pip
  • 9 posts

Posted 26 September 2018 - 02:17 AM

@Michael Denney

 

Restoring from a snapshot taken 45 min ago, is not like a simple reboot.  You will loose all emails from the last 45 min.


  • 0

#357 AMGill29

AMGill29

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 26 September 2018 - 02:39 AM

I was astonished to read the following statement on the locked outage forum today:  "If anybody has a backup of their own - we can use that to get you online immediately. "  I didn't know that. I wish it had been said in the MDDHosting emails about the outage. I meticulously keep a backup.  Well, I'm on S3 and should be restored by late tonight or very early a.m. tomorrow anyway, so I don't see the point of sending in my backup now. It would just add unnecessary work to an already overburdened staff.  And I still have confidence in MDDHosting. But talk about rubbing salt in a wound....

Same here.  I didn't read this till later and assumed it meant a Cpanel backup.  I have files and database and by then it was too late for me too.  I spent the whole weekend at my computer up late restoring as many sites as I could on another host to get my clients sites live.  This would have saved me so much hassle and a host of other DNS resolving issues if this was made clear to me. I could care less about the money I spent just about the stress and now some of my clients have lost trust in me.  I have bragged about MDD to everyone that signed up with me. I understand and feel bad about the domino of events that happened - mostly out of MDDs control but this one was a hard pill to swallow for me as well.


  • 0

#358 SarisIsop

SarisIsop

    Advancing Member

  • Members
  • PipPipPip
  • 155 posts
  • Gender:Not Telling

Posted 26 September 2018 - 03:51 AM

All up and running on S5

 

Have lost a few posts on our forum, but it's no big deal and we are just pleased to be back on-line.

 

I would like to say a big thank you to Michael and all the staff and helpers at MDD for getting us out of this. No it shouldn't have happened, but it did and you all rolled up your sleeves on got on with.

 

Michael should be commended for transparency and honesty to his clients, it takes guts to take it on the chin but he has proven that he will always put his customers first.

 

I also want to shout out to the person that made this happen. Don't dwell on it, it's the past and you can not change that. Dust yourself off, hold your head up high and carry on your journey into the future.

 

We are all to blame and I for one will be taking regular cPanel backups, had I have done that I would have been back on-line within a couple of hours instead of waiting till today. Lesson learnt.

 

I will not be leaving MDD. I'm only a small fish but they have always looked after me, sometimes far beyond what I pay them for. I also will not be requesting any refund/compensation as I have lost no money over this and I feel you will be losing enough.

 

I know all of you at MDD have still a lot of work ahead of you and yes you will undoubtedly be on the big side of losing, but this episode has made you stronger and I still have full faith in you. 

 

Sorry to those of you that have been posting your anger especially Avatar for telling me I will just have to "Be Annoyed" I do understand as I once ran my own business, but I just wanted the original thread to be updates and not questions or complaints. Anyway Michael did eventually split the thread and lock it, something I would recommend doing in future.

 

Good luck to everyone whichever way you have decided to go, and thank you MDD.


  • 4

#359 ninjaclan

ninjaclan

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 26 September 2018 - 07:25 AM

Any ETA for S4 full restoration?


  • 0

#360 rnoar

rnoar

    Newbie

  • Members
  • Pip
  • 1 posts

Posted 26 September 2018 - 07:54 AM

Please - ETA on S4 accounts fully restored.
  • 0




5 user(s) are reading this topic

0 members, 5 guests, 0 anonymous users