[MikeDVB]

Original Message:
The fresco server is currently under a DDoS attack which is sending about 110,000 attack packets per second. We're working on mitigating this attack at this time.

Update at 06:35 AM GMT-5 on June 20th:
While shifting accounts off of the IP under attack the server crashed and upon reboot is requiring a file system check. We have not yet determined what caused the crash resulting in the file system check, however, there is no safe way for us to skip the file system check.

Once the check is completed we will reboot the server and then see where we stand, the check could take anywhere from 45 minutes to 6 hours (there is no ETA provided by the file system checker) and will vary greatly depending on how damaged (or not) the file system is.

We'll update the thread as we know more.

Update at 07:00 AM GMT-5 on June 20th:
The server crashed due to what we believe is hardware failure. This is coincidental and not due to the DDoS attacks. We are going to have to perform a file system check.

Update at 2:25 PM GMT-5 on June 20th:
The file system check completed around a half hour ago and it was determined that too much file system damage had been done. We have initiated a full restoration of the server, including all customer and account data, to our restoration point generated between 12:30 AM and 1:30 AM GMT-5 last night. The current ETA from our backup system is 5 hours and 30 minutes at the time of this update. Keep in mind that the ETA is exactly that, estimated, and it could finish faster or slower.

We will continue updating this thread as the restoration is performed.

Update at 9:22 PM GMT-5 on June 20th:
The restoration completed however the file system still had issues and as such we've initiated another restoration back one additional day. The ETA for this restoration is approximately 6 to 7 hours. We will keep this thread updated with periodic updates.

Update at 5:00 AM GMT-5 on June 21st:
The server restoration has approximately 1 hour remaining. Once it is completed there is a good chance we will need to conduct a file system check which may take 2 to 6 hours depending on the state of the file system after restoration. We will update this thread once the restoration is complete to let you know if a file system check was required or not.

Update at 5:38 AM GMT-5 on June 21st:
The server does require a file system check which can take anywhere from 30 minutes to 6 hours, we'll update this thread once the check is completed but there is no way to provide progress reports as it runs as there is no output but what is being repaired.

Update at 6:28 AM GMT-5 on June 21st:
The file system check completed and due to the repairs done we're running an additional file system check to ensure that once the server comes back online that it is stable and doesn't have to be taken back offline for another file system check later today or in the near future.

The first check took approximately 50 minutes and we estimate the second check will take approximately the same amount of time.

Update at 6:46 AM GMT-5 on June 21st:
The second pass of the file system check is 42% completed.

Update at 7:21 AM GMT-5 on June 21st:
The system is booting up, the restoration process is finished and the file system checks are done.

Update at 7:26 AM GMT-5 on June 21st:
The server booted up successfully and all services are restored. We do need to update the DNS records for a large number of accounts due to the half-completed DDoS mitigation when the server crashed. This process is expected to take between 2 and 12 hours as we have to manually process each and every DNS zone and make the necessary changes.

Update at 2:30 PM GMT-5 on June 21st:
We are approximately half way done fixing DNS zones, we estimate completion around 7 to 9 PM GMT-5.

Update at 9:38 PM GMT-5 on June 21st:
All DNS zones have been updated and all sites are 100% functional. If you've tried visiting your site and gotten a "cPanel Default" page then you will likely have to wait an additional 1 to 2 hours as your local computer and your ISP DNS caches expire. Anybody who hasn't visited the site recently (i.e. a new visitor to your site) will see your site instantly.

Update at 11:30 PM GMT-5 on June 21st:
We have checked all accounts whose DNS was updated and verified that they are all coming back correctly. If you are experiencing any issues at this point do please open a support ticket.

[MikeDVB]

Due to the size and nature of the attack we're forced to null-route the traffic to the affected IP. We will be shifting accounts off of this IP in small batches to bring sites back online. When or if the attack shifts with an account we'll then identify that site as the target of the attack and isolate it.

If you have any questions, do let us know. We'll update this thread when we have more information to share.

[ender]

which IP is the affected IP?

[MikeDVB]

ender, on 20 June 2011 - 04:15 AM, said:

which IP is the affected IP?

If your site is offline right now, then you're on the affected IP. If your site is not offline right now, then you are not on the affected IP.

Due to the null-route all accounts on the server *not* on the IP under attack are back online and fully operational. We're working now to shift accounts that are on the IP off of it so that they can come back online and so that we can identify the target of the attack, assuming the attackers are using DNS to look up the target IP.

[ender]

unfortunate, I am on the affected IP.
and how long you reckon that my websites will be online ?

[MikeDVB]

ender, on 20 June 2011 - 04:45 AM, said:

unfortunate, I am on the affected IP.
and how long you reckon that my websites will be online ?

They will be online somewhere between 5 minutes and potentially 2 to 3 hours from now. It really just depends on how quickly your DNS cache of your old IP address clears and whether you're using your ISP's DNS or Google Public DNS, etc... Generally the maximum is about 2 hours.

[ender]

should I need to do some thing? like change my domain's A record?
All my clients are use A record to poin to server

[MikeDVB]

ender, on 20 June 2011 - 05:05 AM, said:

should I need to do some thing? like change my domain's A record?
All my clients are use A record to poin to server

Ah, in that case you would need to update the A-records to point to the new IP which may vary as the accounts are being spread out across multiple IPs.

This, however, looks to be a bad night for Fresco as the server crashed and is forcing a file system check. This could easily take a few hours but is not something that we can safely skip. We'll update the original post in this thread shortly.

[MikeDVB]

Original post updated to be more detailed. The file system check is still in progress.

[ender]

MikeDVB, on 20 June 2011 - 05:32 AM, said:

Ah, in that case you would need to update the A-records to point to the new IP which may vary as the accounts are being spread out across multiple IPs.This, however, looks to be a bad night for Fresco as the server crashed and is forcing a file system check. This could easily take a few hours but is not something that we can safely skip. We'll update the original post in this thread shortly.

I have to update the A-records? because I am a reseller, it is hard to tell evry client change the A-records.

[MikeDVB]

ender, on 20 June 2011 - 06:11 AM, said:

I have to update the A-records? because I am a reseller, it is hard to tell evry client change the A-records.

This is why it's suggested to use a set of private nameservers (offered by us) or our white-label reseller nameservers (also offered by us). This isn't specific to this issue so if you have further questions about private name servers / a-records / etc, do please open or update a support ticket.

[MikeDVB]

We have no new information to provide at this time beyond that the filesystem check is still in progress.

[jack]

Your professionalism and updates are greatly appreciated during this time Mike.

Clients notified and will await further updates.

Jack

[shanz]

Any more updates as to when the sites will be up? It has been more than 6 hours

[milezone]

Hi Mike,

Any updates yet? I know our emails are down as well so this is our only communication we can have with you. Mondays are really busy for us as we have lots of weekend orders to ship. Hopefully we get back on soon. Thanks

Russ
Http://milezone.com

[Software]

This is really awful! My website is the only way we generate income. This is a whole day lost for me!

[Saulius]

As well as my page is not working all day and it seems that today we do not receive any earnings.

[Themeltingclock]

Saulius, on 20 June 2011 - 11:32 AM, said:

As well as my page is not working all day and it seems that today we do not receive any earnings.

I heard nothing but good reviews about MDDHosting before signing up - I'm not angry about this downtime because let's face it, stuff happens. However, we're going on 7 hours of downtime here, which is getting a bit frustrating to say the least. Like others, I've got no email and because our site hosts a web application, we can't sell any inventory.

[jian]

What is the progress of the fsck? Is it still ongoing or are there other problems as well?

[Frogsplash]

Yeah, MDD Hosting has been pretty rock solid for me so far, been with them about 6 months. They've been responsive, and even when hit with DDoS attacks, they have managed to be back online within 30 minutes in most cases.

This outage, however, has been really awful, relatively speaking. I've been fielding calls and emails all morning from grouchy clients that couldn't care less what a DDoS attack is – they just know their site is down, and I'd better fix it.

I'm not going to play pile on – not going to make a stressful night/day even worse – but if we could at least get an update, that would be appreciated.