Michael D. Posted May 29, 2011 Report Share Posted May 29, 2011 The file system on the Gemini server has gone Read-Only and we're rebooting the server and investigating the cause at the moment. Update: The file system was irreparably damaged and we're restoring the server from our backup system. Update 2 (6:09 PM EST (GMT-5)): The server has been fully restored - the restoration process took approximately 1 hour and 41 minutes. The server is back online and will be sluggish for 5 to 10 minutes as it catches up to missed requests and email. If you have any issues do please open a support ticket. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted May 29, 2011 Author Report Share Posted May 29, 2011 The file system on the server appears to be severely damaged. We do have backups from last night however restoring from those backups is going to be a last resort. We will update this thread as we know more. Quote Link to comment Share on other sites More sharing options...
nbaintel Posted May 29, 2011 Report Share Posted May 29, 2011 Does that mean that we might lose our data from last night till now? Quote Link to comment Share on other sites More sharing options...
Michael D. Posted May 29, 2011 Author Report Share Posted May 29, 2011 Does that mean that we might lose our data from last night till now?That is a possibility, yes. We'll need to see if we can repair the file system or if we need to go right into a restoration. Our team is working on this now. Quote Link to comment Share on other sites More sharing options...
nbaintel Posted May 29, 2011 Report Share Posted May 29, 2011 That is a possibility, yes. We'll need to see if we can repair the file system or if we need to go right into a restoration. Our team is working on this now.Well, I hope you guys are able to save all the data. Thanks. Quote Link to comment Share on other sites More sharing options...
SFDigital Posted May 29, 2011 Report Share Posted May 29, 2011 Does the possibility exist that we will lose ALL data, ie, the entire site? Or, worse case scenario, just the data from last night? Quote Link to comment Share on other sites More sharing options...
Michael D. Posted May 29, 2011 Author Report Share Posted May 29, 2011 Worst case should be restoring to the backup from last night. We take backups every night of every server. Quote Link to comment Share on other sites More sharing options...
SFDigital Posted May 29, 2011 Report Share Posted May 29, 2011 Thanks, Mike. Of course, this is the week all the PR for our site begins. Argh. ETA when you have it would be much appreciated. Glad you guys are working on it! Quote Link to comment Share on other sites More sharing options...
Patrick.B Posted May 29, 2011 Report Share Posted May 29, 2011 So... I just purchased my plan today, and was working on setting up my forum when this happened. Will I loose everything I had done up to this point? What about my account, and domain name registrations? Quote Link to comment Share on other sites More sharing options...
Michael D. Posted May 29, 2011 Author Report Share Posted May 29, 2011 So... I just purchased my plan today, and was working on setting up my forum when this happened.Will I loose everything I had done up to this point?If we're unable to recover the data directly from the server, then yes, anything created today will be gone. What about my account, and domain name registrations?Your domain registration will be fine - if you wish - I can go ahead and create a new account for you on another server since you've just registered so you can get back to work. Just open a support ticket. An ETA will be posted here as soon as we have one. Quote Link to comment Share on other sites More sharing options...
Patrick.B Posted May 29, 2011 Report Share Posted May 29, 2011 Is it 100% positive you will have to use your backup or not? I would not be able to work on it for another couple hours due to other stuff I need to get finished. Quote Link to comment Share on other sites More sharing options...
ReneB Posted May 29, 2011 Report Share Posted May 29, 2011 I really really hope you get the filesystem fixed. i have done so much work the last 24 hours :/ Quote Link to comment Share on other sites More sharing options...
Michael D. Posted May 29, 2011 Author Report Share Posted May 29, 2011 Is it 100% positive you will have to use your backup or not? I would not be able to work on it for another couple hours due to other stuff I need to get finished.If your account was created today, then it wouldn't matter because the backup was created before your account was created. We're restoring the server now as the file system repairs left the file system in a state where there was simply too much data loss. It will need to run for a short bit before we will have an ETA. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted May 29, 2011 Author Report Share Posted May 29, 2011 5.1 GB of 309 GB have been restored so far. The system ETA says 2.5 hours however this can fluctuate. We'll update this thread with any updates. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted May 29, 2011 Author Report Share Posted May 29, 2011 11 GB Restored. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted May 29, 2011 Author Report Share Posted May 29, 2011 All accounts created on the server today by direct customers have been recreated on another server and account details dispatched. We are restoring the server to our backup from last night which has an ETA of approximately 2.5 hours according to the system. 25.1 GB of 309 GB have been restored so far. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted May 29, 2011 Author Report Share Posted May 29, 2011 35 of 309 GB Restored. Quote Link to comment Share on other sites More sharing options...
nbaintel Posted May 29, 2011 Report Share Posted May 29, 2011 Mike, are you restoring the files from last night? Quote Link to comment Share on other sites More sharing options...
XTremo Posted May 29, 2011 Report Share Posted May 29, 2011 Appreciate the updates and quick response Mike. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted May 29, 2011 Author Report Share Posted May 29, 2011 Mike, are you restoring the files from last night?Yes, we are as the file system was beyond repair. Appreciate the updates and quick response Mike.Absolutely. 58.5 of 309 GBs restored. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted May 29, 2011 Author Report Share Posted May 29, 2011 80 of 309 GBs restored. Quote Link to comment Share on other sites More sharing options...
nbaintel Posted May 29, 2011 Report Share Posted May 29, 2011 Can you tell us what time the backups were made last night? Quote Link to comment Share on other sites More sharing options...
derekd Posted May 29, 2011 Report Share Posted May 29, 2011 Well, this stinks...I was about to heap praise on this company for its reliability to some friends. Doh. Any idea what caused a relatively recently deployed, low occupancy server to suddenly 'blow up'? Quote Link to comment Share on other sites More sharing options...
Michael D. Posted May 29, 2011 Author Report Share Posted May 29, 2011 Can you tell us what time the backups were made last night?Around 1:30 am gmt-5 (EST). Well, this stinks...I was about to heap praise on this company for its reliability to some friends. Doh. Any idea what caused a relatively recently deployed, low occupancy server to suddenly 'blow up'?We believe it was a hardware issue with the raid controller and we swapped it out before beginning the restoration. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted May 29, 2011 Author Report Share Posted May 29, 2011 135 of 309 GBs restored. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.