Just wanted to say thanks again. ALL of my reseller accounts on R1 are back and working 100%, including a vBulletin forum and several wordpress sites. The restore was from last Tuesday afternoon, but that only matters for the forum. (and I auto back it up twice daily to my home server, so that's not really an issue)
NO problems here.
We had a similar problem back on 9/11/01, when our previous data center/isp was knocked out by the falling twin towers. They were down for more than a month.
Great work, team. After this, you may need to sleep for a week.
Having gone through 9-11 working in tech for a large financial firm, I disagree that this event is remotely comparable to what happened back then.
9-11 was an externally driven event; what happened here is self-inflicted.
Not counting on the critical human error (because we all make mistakes), the gap between the advertised backup / redundancy guarantees to the actual recovery effort is too far and too wide in my opinion.
I agree that MDD deserves some kudos for being transparent as it struggled to recover but I am not ready to write any congratulatory letters because I have two well paying customers who are angry and are ready fire me and I have to spend my time and money to fix this fiasco.
So the hard lesson for me is to NOT get lazy and rely on any given company, even those with positive reputations, because at the end of the day, I am responsible for my own business.
For those of you impacted, I implore you to:
1. Move any/all domains out to a separate registrar so that you can control your DNS
2. Regularly back up everything and store them in cloud and local storagte
3. Pay for an alternative hosting company as a stanby
4. Regularly restore backups with standby hosting and test!