Jump to content
MDDHosting Forums

[Completed] Fresco Server - Restoration Completed - File System Check Competed


Recommended Posts

So should the sites be back online? If so, I'm not seeing it...

If you were on the IP 173.248.191.65 then your site will be offline for at least a few more minutes while we re-sync the DNS and httpd.conf. We're working on this now (it's the last thing that needs done before I can get some sleep).
Link to comment
Share on other sites

Big thanks to Mike and the team for their hard work. Stuff happens, but it's how you respond to the crisis that shows the true colors of a company. MDD has been better than all the other hosting companies I've used over the years.

 

With that said, it's time to dig into that failover DNS post :-)

Link to comment
Share on other sites

I would just like to say fantastic job Mike (and team) for all your hard work in restoring the server - it was a huge task and as always you didn't disappoint - everything is restored for us with 0 data loss and 0 email loss either (although emails were of course delayed).

 

Very well done.

 

When you've had some sleep I'd very much appreciate a complete summary of the causes for the downtime over the last day and the steps taken to resolve them so I can pass them onto my clients.

 

Best,

Ben

Link to comment
Share on other sites

The file system issue looks to have stemmed from an unclean restart at some point - we've made some file system changes to help prevent this from happening such as reducing the commit time on the file system journal so that if something like this happens again it's MUCH less likely to cause an issue.

 

As far as the DDoS, there's no real way to prevent that unfortunately. All we can do, if it happens, is work hard to mitigate it.

Link to comment
Share on other sites

Everyone who is going to perform a regular cpanel backup raise your hand (*hand raised*)

Please wait until this evening - http://www.mddhosting.com/tos.php#7c

 

Doing so now will cause performance issues as we're doing a lot of work on the server right now to fix accounts that were affected by the DDoS.

 

Also for anybody who is following this thread and wants to open a ticket with "My site is not up yet!":

We're having to do some DNS fixes for accounts that were on the IP that came under DDoS attack due to having to restore back to before those changes were made and our DNS cluster being out of sync. It's a painfully manual process and is taking some time.

 

We're working on it as quickly as possible.

Link to comment
Share on other sites

Thanks Mike and team for bringing the server back online.

 

My site and more critically, my email, has been hit by this outage. As a tech guy I can appreciate the technical issue Mike and team has been updating. Its no easy job but you guys have pulled through.

 

Everyone can claim to provide affordable, value-for-money hosting backed with superior support but its times like this and most importantly your response that truly define a business.

Link to comment
Share on other sites

I think you guys have done the best you possibly could with what you were given. I must have been on the affected IP as I am still down.

 

I'm just hoping that my IP/DNS issues are fixed in the next thirty minutes before our business opens and employees arrive...if I have to explain even another second of downtime, I'm going to break :rage:

 

EDIT: As I was typing that...phone rings..."hey my email works but the site is still down.."

Link to comment
Share on other sites

I think you guys have done the best you possibly could with what you were given. I must have been on the affected IP as I am still down.

 

I'm just hoping that my IP/DNS issues are fixed in the next thirty minutes before our business opens and employees arrive...if I have to explain even another second of downtime, I'm going to break :rage:

If you have a ticket in our system regarding this outage already, update it and I'll see what I can do to speed up the process but I can't make any promises. DNS does have propagation which there is no way to avoid.
Link to comment
Share on other sites

If you have a ticket in our system regarding this outage already, update it and I'll see what I can do to speed up the process but I can't make any promises. DNS does have propagation which there is no way to avoid.

I don't and I won't. You guys do what you have to do. Our employees have emails at to occupy themselves with at this point.

Link to comment
Share on other sites

I don't and I won't. You guys do what you have to do. Our employees have emails at to occupy themselves with at this point.

Yeah, the email should work - just the site itself may not load until the DNS updates. We're going alphabetically domain by domain that is affected.
Link to comment
Share on other sites

I'd like to see a real back up plan in place.

 

For example.

 

-Fresco goes down due to whatever.

-You have one machine on the ready to restore all accounts from backups

-Done!

 

I think the cost of one machine is pretty reasonable compared to having to hire an expert as you had to just do for this instance.

 

Congratulations for all the hard work you do is not enough for me, I think you do have a responsibility to put more effort into a back up plan to keep everyone's accounts up and running. You may say that yes u're not responsible for DDos attacks, but others I'm sure have lost clients due to something not their fault as well.

 

 

So we just have to endure this again and again?

 

I'm not blaming you, but more should be put into finding a solution for these situations.

 

If I were there, I surely would be heading up a project to do so. Nothing is impossible.

Link to comment
Share on other sites

I'd like to see a real back up plan in place.
We have a "real" backup plan. I'm not sure what makes you think otherwise.

 

-Fresco goes down due to whatever.

-You have one machine on the ready to restore all accounts from backups

-Done!

This is exactly what we did.

 

Congratulations for all the hard work you do is not enough for me, I think you do have a responsibility to put more effort into a back up plan to keep everyone's accounts up and running. You may say that yes u're not responsible for DDos attacks, but others I'm sure have lost clients due to something not their fault as well.
Ok, well nobody is forcing you to stay - if you wish to leave, by all means do what you feel is best.

 

So we just have to endure this again and again?
DDoS attacks - there's nothing we can do to stop/prevent those all we can do is mitigate them when they happen. As far as file system issues - this is only the second time we've ever had to run file system checks in over three and a half years in business. If that is "again and again" then I'm sorry. Beyond that, when the file system is corrupted, we don't have a choice but to run a file system check.

 

I'm not blaming you, but more should be put into finding a solution for these situations.
DDoS - We mitigate it. File System Corruption - we repair the file system. This is exactly what we've done.

 

If I were there, I surely would be heading up a project to do so. Nothing is impossible.
Heading up a project to do what? If you know a way to prevent DDoS attacks then you could very easily make yourself a millionaire selling whatever product/solution you come up with to prevent such attacks. And beyond that - if you come up with a way to prevent file system corruption on a kernel panic or other unexpected issue where disk caches may not be fully flushed to the disks - you should patent it.

 

I understand you are upset and/or frustrated but to be honest you really don't seem to have a solid grasp as to what exactly happened and what we did to bring everything back online. By all means, if you feel the grass is greener on the other side of the fence - feel free to find a new provider. This could happen to any provider on any server at any time - what is important about it is how that provider handles the issue. If there is something more you feel we should have done, by all means, do let me know what you think we should have done.

 

Nobody is perfect and we don't claim to be - we just do our absolute best to provide the best service and to resolve issues as quickly and efficiently as possible. We're not above and beyond taking constructive criticism and advice but if you're going to give advice, make sure you know what you're talking about before you do so. So I'm all ears, let me know what you think should have been done differently.

  • Upvote 2
Link to comment
Share on other sites

None of the problems experienced were anyones fault and you did a fantastic job at solving them. However, I do agree with techshots that there should have been another option. Something such as having access to full cPanel backups of each account would have been perfect. At least that way we could have restored the "angry clients" on another server. This is something I imagine would have to be setup on a individual basis so I will taking you up on your offer and contacting you about this later today.

 

You mentioned earlier that a redundant solution would up the price, personally I'd be happy to pay that extra and I'm sure many others would as well.

 

As I'm sure you do also I feel like I've got 1 step forward slowly and 5 steps back in terms of relationship with clients.

Link to comment
Share on other sites

None of the problems experienced were anyones fault and you did a fantastic job at solving them. However, I do agree with techshots that there should have been another option. Something such as having access to full cPanel backups of each account would have been perfect.
We keep our R1Soft backups as a courtesy for disaster recovery (which is exactly what this was). It is advised that you keep your own backups (See: http://www.mddhosting.com/supportscope.php). As I said a few times in this thread - if you have backups we're more than happy to restore them to an alternate server for you. If you don't have your own backups, you may want to re-think your data retention strategies. Obviously we're not going to do anything to intentionally lose data but if it were to happen, you should have your own copy of the accounts.

 

At least that way we could have restored the "angry clients" on another server. This is something I imagine would have to be setup on a individual basis so I will taking you up on your offer and contacting you about this later today.
I'll be happy to advise you on setting up cPanel backups for your accounts but as for where you store them and how you generate them, that's entirely up to you. There are services that will do it for you and then other ways to go about doing it using a cPanel backup cron per account (yes, it's a pain to set it up per account, I know) where you can have them FTPed to somewhere.

 

You mentioned earlier that a redundant solution would up the price, personally I'd be happy to pay that extra and I'm sure many others would as well.
Some would, most would not, if you really need a HA failover solution - they are out there and they are not cheap.

 

As I'm sure you do also I feel like I've got 1 step forward slowly and 5 steps back in terms of relationship with clients.
I'm sorry to hear that. A vast majority of our client base understands that these issues are inevitable and are the nature of being on a shared server with mechanical drives. Most are appreciative that we even have the backups we claim to take as many of our clients have been with other providers where the promised backups simply did not exist.

 

I do apologize for any trouble this may have caused you, but we've been caused just as much (if not more) trouble ourselves.

Link to comment
Share on other sites

 

 

Congratulations for all the hard work you do is not enough for me, I think you do have a responsibility to put more effort into a back up plan to keep everyone's accounts up and running. You may say that yes u're not responsible for DDos attacks, but others I'm sure have lost clients due to something not their fault as well.

 

Ok, well nobody is forcing you to stay - if you wish to leave, by all means do what you feel is best.

 

Hi Mike, I believe thats not what techshots meant from what I understand. If you read carefully, its simply mentioned that some resellers may have lost clients due to the downtime. He did not mean to say he is leaving MDD.

 

We all undertand MDD has done an outstanding job over the past 24 hours and that really separates MDD from other web hosting companies. Keep up the good work :)

Link to comment
Share on other sites

Hi Mike, I believe thats not what techshots meant from what I understand. If you read carefully, its simply mentioned that some resellers may have lost clients due to the downtime. He did not mean to say he is leaving MDD.

I worked on the server/issues personally for a solid 25 hours and had been awake for around 18 to 20 hours before the issue even happened so I may be slightly misinterpreting the post. If that is the case, I do apologize to techshots.

 

We all undertand MDD has done an outstanding job over the past 24 hours and that really separates MDD from other web hosting companies. Keep up the good work :)
Some surely won't see it that way and unfortunately there isn't anything we're going to be able to do about that.
Link to comment
Share on other sites

I worked on the server/issues personally for a solid 25 hours and had been awake for around 18 to 20 hours before the issue even happened so I may be slightly misinterpreting the post. If that is the case, I do apologize to techshots.

 

Some surely won't see it that way and unfortunately there isn't anything we're going to be able to do about that.

Where do you stand, alphabetically, on the DNS restoration?

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...