Jump to content


Photo

[Completed] Gemini full restoration from backup system.


  • Please log in to reply
43 replies to this topic

#21 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,893 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 29 May 2011 - 03:56 PM

80 of 309 GBs restored.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#22 nbaintel

nbaintel

    Newbie

  • Members
  • Pip
  • 17 posts

Posted 29 May 2011 - 04:01 PM

Can you tell us what time the backups were made last night?
  • 0

#23 derekd

derekd

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 29 May 2011 - 04:06 PM

Well, this stinks...I was about to heap praise on this company for its reliability to some friends. Doh.

Any idea what caused a relatively recently deployed, low occupancy server to suddenly 'blow up'?
  • 0

#24 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,893 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 29 May 2011 - 04:08 PM

Can you tell us what time the backups were made last night?

Around 1:30 am gmt-5 (EST).

Well, this stinks...I was about to heap praise on this company for its reliability to some friends. Doh.

Any idea what caused a relatively recently deployed, low occupancy server to suddenly 'blow up'?

We believe it was a hardware issue with the raid controller and we swapped it out before beginning the restoration.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#25 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,893 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 29 May 2011 - 04:12 PM

135 of 309 GBs restored.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#26 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,893 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 29 May 2011 - 04:20 PM

160 of 309 GBs restored. ETA 58 minutes.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#27 nestito

nestito

    Newbie

  • Members
  • Pip
  • 3 posts

Posted 29 May 2011 - 04:27 PM

Well, this stinks...I was about to heap praise on this company for its reliability to some friends. Doh.

Any idea what caused a relatively recently deployed, low occupancy server to suddenly 'blow up'?


I noticed that I couldn't reach any of my websites tonight so I decided to first check Server Status and Twitter to see if there was something going on globally. Sure nuff, there was and MDD was handling it.

MDD is reliable. This sort of thing can happen to any host. What sets a good host apart from from other in a time like this is there customer service and the staff is doing a good job. They are doing good by keeping us updated on what's going on with Gemini. Let them do their thing and fix the issue :-)
  • 0

#28 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,893 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 29 May 2011 - 04:30 PM

Indeed, we're at 200 of 309 GB restored so it shouldn't be much longer. We don't anticipate any issues booting up the server once the restoration is completed but we're keeping extra staff on hand just in case.
  • 1
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#29 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,893 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 29 May 2011 - 04:39 PM

225 of 309 GB Restored. ETA 32 Minutes.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#30 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,893 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 29 May 2011 - 04:45 PM

245 of 309 GB restored.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#31 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,893 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 29 May 2011 - 04:54 PM

280 of 309 GBs restored, the system says around 11 Minute ETA.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#32 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,893 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 29 May 2011 - 05:03 PM

The server is booting up now.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#33 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,893 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 29 May 2011 - 05:09 PM

The server has fully booted up and all services are fully operational at this time. If you have any issues at all do please open a support ticket.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#34 rowanhippy

rowanhippy

    Newbie

  • Members
  • Pip
  • 4 posts
  • Gender:Male
  • Location:near the hills

Posted 29 May 2011 - 05:49 PM

Thanks very much Mike and the team for keeping us fully updated with what was going on and rectifying the problem. Its very much appreciated.
  • 0
rowanhippy

"To touch the earth is to have harmony with nature." Oglala Sioux

#35 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,893 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 29 May 2011 - 05:52 PM

Thanks very much Mike and the team for keeping us fully updated with what was going on and rectifying the problem. Its very much appreciated.

Absolutely - we identified the cause of the failure as a raid controller and we swapped the controller out prior to restoring the data. We're going to have LSI (the manufacturer) analyze the card to see if they can identify the cause as all the rest of the hardware checked out OK.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#36 nbaintel

nbaintel

    Newbie

  • Members
  • Pip
  • 17 posts

Posted 29 May 2011 - 05:54 PM

While it's good that you were able to recover files from last night.

Many customers, including myself, lost several hours of work which is quite depressing.

The up time for May is now down to 99.45% which is below your guaranteed 99.9%.

I believe you should compensate those who are effected.
  • 0

#37 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,893 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 29 May 2011 - 05:57 PM

Losing hours of work is better than losing all work (i.e. all data). I'm not saying that things worked out how we wanted them to or that we're happy about it - but the current resolution is better than many providers would have been able to offer.

I do apologize for the trouble, however, we have no control over hardware failure unfortunately. It will inevitably happen no matter what provider you are with or what kind of systems they are running. We're operating an expensive backup infrastructure just for this specific type of instance and it's paid off in this situation.

If you have any specific questions, do please open a ticket.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#38 Al Stewart

Al Stewart

    Newbie

  • Members
  • Pip
  • 1 posts
  • Gender:Male
  • Location:Oregon

Posted 29 May 2011 - 06:35 PM

I just signed up with MDDhosting and this thread has dropped my jaw! I left enotch.com because they have been down 3 days and not one word was sent explaining the outage or the ETA of the fix. Even their own web sites went away! Wow! Talk about keeping customers informed!
  • 0

#39 Charles

Charles

    Newbie

  • Members
  • Pip
  • 7 posts

Posted 29 May 2011 - 08:09 PM

Mike has went above and beyond to keep everyone informed of what was going on along with the status of the restore. Not many hosts would provide updates as often as Mike has.

There are even some hosts that claim to have backups but when it comes right down to it, they have backups that are over 2 weeks old or none at all!!!

Mike thanks for keeping everyone updated and for having backups like you do.
  • 0

#40 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,893 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 30 May 2011 - 01:09 AM

We are very sorry for anybody who has lost any data - we wish we could bring it back and we did our absolute best to keep it before going to the last resort which is a full server restoration.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/




1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users