Jump to content


Photo

[Completed] Fresco Server - Restoration Completed - File System Check Competed


  • Please log in to reply
218 replies to this topic

#181 Software

Software

    Newbie

  • Members
  • Pip
  • 16 posts

Posted 21 June 2011 - 08:20 AM

Ok, so seriously... why am I STILL down?
  • 0

#182 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 21 June 2011 - 08:22 AM

Ok, so seriously... why am I STILL down?

Read this (it's from this thread):
http://forums.mddhos...ndpost__p__2720
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#183 benwiggy

benwiggy

    Newbie

  • Members
  • Pip
  • 5 posts

Posted 21 June 2011 - 08:42 AM

I would just like to say fantastic job Mike (and team) for all your hard work in restoring the server - it was a huge task and as always you didn't disappoint - everything is restored for us with 0 data loss and 0 email loss either (although emails were of course delayed).

Very well done.

When you've had some sleep I'd very much appreciate a complete summary of the causes for the downtime over the last day and the steps taken to resolve them so I can pass them onto my clients.

Best,
Ben
  • 0

#184 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 21 June 2011 - 08:44 AM

The file system issue looks to have stemmed from an unclean restart at some point - we've made some file system changes to help prevent this from happening such as reducing the commit time on the file system journal so that if something like this happens again it's MUCH less likely to cause an issue.

As far as the DDoS, there's no real way to prevent that unfortunately. All we can do, if it happens, is work hard to mitigate it.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#185 FloridaClay

FloridaClay

    Newbie

  • Members
  • Pip
  • 5 posts

Posted 21 June 2011 - 08:50 AM

Everyone who is going to perform a regular cpanel backup raise your hand (*hand raised*)
  • 0

#186 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 21 June 2011 - 08:51 AM

Everyone who is going to perform a regular cpanel backup raise your hand (*hand raised*)

Please wait until this evening - http://www.mddhosting.com/tos.php#7c

Doing so now will cause performance issues as we're doing a lot of work on the server right now to fix accounts that were affected by the DDoS.

Also for anybody who is following this thread and wants to open a ticket with "My site is not up yet!":
We're having to do some DNS fixes for accounts that were on the IP that came under DDoS attack due to having to restore back to before those changes were made and our DNS cluster being out of sync. It's a painfully manual process and is taking some time.

We're working on it as quickly as possible.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#187 008Rohit

008Rohit

    Member

  • Members
  • PipPip
  • 32 posts
  • Gender:Male
  • Location:Kolkata, India
  • Interests:Tech, SEO, Sports.

Posted 21 June 2011 - 08:52 AM

The server was sent 110,000 packets per second right? Is it the same as 110,000 bytes ping?
  • 0

TechTage - Hosted on S1

Genuine Web Hosting Reviews - Read my MDDHosting Review


#188 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 21 June 2011 - 08:53 AM

The server was sent 110,000 packets per second right? Is it the same as 110,000 bytes ping?

No, it's not the same thing.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#189 carbide

carbide

    Newbie

  • Members
  • Pip
  • 1 posts

Posted 21 June 2011 - 08:54 AM

Thanks Mike and team for bringing the server back online.

My site and more critically, my email, has been hit by this outage. As a tech guy I can appreciate the technical issue Mike and team has been updating. Its no easy job but you guys have pulled through.

Everyone can claim to provide affordable, value-for-money hosting backed with superior support but its times like this and most importantly your response that truly define a business.
  • 0

#190 Themeltingclock

Themeltingclock

    Newbie

  • Members
  • Pip
  • 13 posts

Posted 21 June 2011 - 09:09 AM

I think you guys have done the best you possibly could with what you were given. I must have been on the affected IP as I am still down.

I'm just hoping that my IP/DNS issues are fixed in the next thirty minutes before our business opens and employees arrive...if I have to explain even another second of downtime, I'm going to break :rage:

EDIT: As I was typing that...phone rings..."hey my email works but the site is still down.."
  • 0

#191 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 21 June 2011 - 09:11 AM

I think you guys have done the best you possibly could with what you were given. I must have been on the affected IP as I am still down.

I'm just hoping that my IP/DNS issues are fixed in the next thirty minutes before our business opens and employees arrive...if I have to explain even another second of downtime, I'm going to break :rage:

If you have a ticket in our system regarding this outage already, update it and I'll see what I can do to speed up the process but I can't make any promises. DNS does have propagation which there is no way to avoid.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#192 Themeltingclock

Themeltingclock

    Newbie

  • Members
  • Pip
  • 13 posts

Posted 21 June 2011 - 09:16 AM

If you have a ticket in our system regarding this outage already, update it and I'll see what I can do to speed up the process but I can't make any promises. DNS does have propagation which there is no way to avoid.

I don't and I won't. You guys do what you have to do. Our employees have emails at to occupy themselves with at this point.
  • 0

#193 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 21 June 2011 - 09:17 AM

I don't and I won't. You guys do what you have to do. Our employees have emails at to occupy themselves with at this point.

Yeah, the email should work - just the site itself may not load until the DNS updates. We're going alphabetically domain by domain that is affected.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#194 techshots

techshots

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 21 June 2011 - 09:44 AM

I'd like to see a real back up plan in place.

For example.

-Fresco goes down due to whatever.
-You have one machine on the ready to restore all accounts from backups
-Done!

I think the cost of one machine is pretty reasonable compared to having to hire an expert as you had to just do for this instance.

Congratulations for all the hard work you do is not enough for me, I think you do have a responsibility to put more effort into a back up plan to keep everyone's accounts up and running. You may say that yes u're not responsible for DDos attacks, but others I'm sure have lost clients due to something not their fault as well.


So we just have to endure this again and again?

I'm not blaming you, but more should be put into finding a solution for these situations.

If I were there, I surely would be heading up a project to do so. Nothing is impossible.
  • 0

#195 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 21 June 2011 - 09:50 AM

I'd like to see a real back up plan in place.

We have a "real" backup plan. I'm not sure what makes you think otherwise.

-Fresco goes down due to whatever.
-You have one machine on the ready to restore all accounts from backups
-Done!

This is exactly what we did.

Congratulations for all the hard work you do is not enough for me, I think you do have a responsibility to put more effort into a back up plan to keep everyone's accounts up and running. You may say that yes u're not responsible for DDos attacks, but others I'm sure have lost clients due to something not their fault as well.

Ok, well nobody is forcing you to stay - if you wish to leave, by all means do what you feel is best.

So we just have to endure this again and again?

DDoS attacks - there's nothing we can do to stop/prevent those all we can do is mitigate them when they happen. As far as file system issues - this is only the second time we've ever had to run file system checks in over three and a half years in business. If that is "again and again" then I'm sorry. Beyond that, when the file system is corrupted, we don't have a choice but to run a file system check.

I'm not blaming you, but more should be put into finding a solution for these situations.

DDoS - We mitigate it. File System Corruption - we repair the file system. This is exactly what we've done.

If I were there, I surely would be heading up a project to do so. Nothing is impossible.

Heading up a project to do what? If you know a way to prevent DDoS attacks then you could very easily make yourself a millionaire selling whatever product/solution you come up with to prevent such attacks. And beyond that - if you come up with a way to prevent file system corruption on a kernel panic or other unexpected issue where disk caches may not be fully flushed to the disks - you should patent it.

I understand you are upset and/or frustrated but to be honest you really don't seem to have a solid grasp as to what exactly happened and what we did to bring everything back online. By all means, if you feel the grass is greener on the other side of the fence - feel free to find a new provider. This could happen to any provider on any server at any time - what is important about it is how that provider handles the issue. If there is something more you feel we should have done, by all means, do let me know what you think we should have done.

Nobody is perfect and we don't claim to be - we just do our absolute best to provide the best service and to resolve issues as quickly and efficiently as possible. We're not above and beyond taking constructive criticism and advice but if you're going to give advice, make sure you know what you're talking about before you do so. So I'm all ears, let me know what you think should have been done differently.
  • 2
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#196 jack

jack

    Newbie

  • Members
  • Pip
  • 12 posts

Posted 21 June 2011 - 10:00 AM

None of the problems experienced were anyones fault and you did a fantastic job at solving them. However, I do agree with techshots that there should have been another option. Something such as having access to full cPanel backups of each account would have been perfect. At least that way we could have restored the "angry clients" on another server. This is something I imagine would have to be setup on a individual basis so I will taking you up on your offer and contacting you about this later today.

You mentioned earlier that a redundant solution would up the price, personally I'd be happy to pay that extra and I'm sure many others would as well.

As I'm sure you do also I feel like I've got 1 step forward slowly and 5 steps back in terms of relationship with clients.
  • 0

#197 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 21 June 2011 - 10:05 AM

None of the problems experienced were anyones fault and you did a fantastic job at solving them. However, I do agree with techshots that there should have been another option. Something such as having access to full cPanel backups of each account would have been perfect.

We keep our R1Soft backups as a courtesy for disaster recovery (which is exactly what this was). It is advised that you keep your own backups (See: http://www.mddhostin...upportscope.php). As I said a few times in this thread - if you have backups we're more than happy to restore them to an alternate server for you. If you don't have your own backups, you may want to re-think your data retention strategies. Obviously we're not going to do anything to intentionally lose data but if it were to happen, you should have your own copy of the accounts.

At least that way we could have restored the "angry clients" on another server. This is something I imagine would have to be setup on a individual basis so I will taking you up on your offer and contacting you about this later today.

I'll be happy to advise you on setting up cPanel backups for your accounts but as for where you store them and how you generate them, that's entirely up to you. There are services that will do it for you and then other ways to go about doing it using a cPanel backup cron per account (yes, it's a pain to set it up per account, I know) where you can have them FTPed to somewhere.

You mentioned earlier that a redundant solution would up the price, personally I'd be happy to pay that extra and I'm sure many others would as well.

Some would, most would not, if you really need a HA failover solution - they are out there and they are not cheap.

As I'm sure you do also I feel like I've got 1 step forward slowly and 5 steps back in terms of relationship with clients.

I'm sorry to hear that. A vast majority of our client base understands that these issues are inevitable and are the nature of being on a shared server with mechanical drives. Most are appreciative that we even have the backups we claim to take as many of our clients have been with other providers where the promised backups simply did not exist.

I do apologize for any trouble this may have caused you, but we've been caused just as much (if not more) trouble ourselves.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#198 HADNow

HADNow

    Newbie

  • Members
  • Pip
  • 8 posts

Posted 21 June 2011 - 10:07 AM

Congratulations for all the hard work you do is not enough for me, I think you do have a responsibility to put more effort into a back up plan to keep everyone's accounts up and running. You may say that yes u're not responsible for DDos attacks, but others I'm sure have lost clients due to something not their fault as well.

Ok, well nobody is forcing you to stay - if you wish to leave, by all means do what you feel is best.


Hi Mike, I believe thats not what techshots meant from what I understand. If you read carefully, its simply mentioned that some resellers may have lost clients due to the downtime. He did not mean to say he is leaving MDD.

We all undertand MDD has done an outstanding job over the past 24 hours and that really separates MDD from other web hosting companies. Keep up the good work :)
  • 0

#199 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 21 June 2011 - 10:10 AM

Hi Mike, I believe thats not what techshots meant from what I understand. If you read carefully, its simply mentioned that some resellers may have lost clients due to the downtime. He did not mean to say he is leaving MDD.

I worked on the server/issues personally for a solid 25 hours and had been awake for around 18 to 20 hours before the issue even happened so I may be slightly misinterpreting the post. If that is the case, I do apologize to techshots.

We all undertand MDD has done an outstanding job over the past 24 hours and that really separates MDD from other web hosting companies. Keep up the good work :)

Some surely won't see it that way and unfortunately there isn't anything we're going to be able to do about that.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#200 Themeltingclock

Themeltingclock

    Newbie

  • Members
  • Pip
  • 13 posts

Posted 21 June 2011 - 10:19 AM

I worked on the server/issues personally for a solid 25 hours and had been awake for around 18 to 20 hours before the issue even happened so I may be slightly misinterpreting the post. If that is the case, I do apologize to techshots.

Some surely won't see it that way and unfortunately there isn't anything we're going to be able to do about that.

Where do you stand, alphabetically, on the DNS restoration?
  • 0




1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users