43 replies to this topic

[MikeDVB]

The file system on the Gemini server has gone Read-Only and we're rebooting the server and investigating the cause at the moment.

Update:

The file system was irreparably damaged and we're restoring the server from our backup system.

Update 2 (6:09 PM EST (GMT-5)):

The server has been fully restored - the restoration process took approximately 1 hour and 41 minutes. The server is back online and will be sluggish for 5 to 10 minutes as it catches up to missed requests and email. If you have any issues do please open a support ticket.

[MikeDVB]

The file system on the server appears to be severely damaged. We do have backups from last night however restoring from those backups is going to be a last resort. We will update this thread as we know more.

[nbaintel]

Does that mean that we might lose our data from last night till now?

[MikeDVB]

Does that mean that we might lose our data from last night till now?

That is a possibility, yes. We'll need to see if we can repair the file system or if we need to go right into a restoration. Our team is working on this now.

[nbaintel]

That is a possibility, yes. We'll need to see if we can repair the file system or if we need to go right into a restoration. Our team is working on this now.

Well, I hope you guys are able to save all the data.

Thanks.

[SFDigital]

Does the possibility exist that we will lose ALL data, ie, the entire site? Or, worse case scenario, just the data from last night?

[MikeDVB]

Worst case should be restoring to the backup from last night. We take backups every night of every server.

[SFDigital]

Thanks, Mike. Of course, this is the week all the PR for our site begins. Argh. ETA when you have it would be much appreciated. Glad you guys are working on it!

[Patrick.B]

So... I just purchased my plan today, and was working on setting up my forum when this happened.

Will I loose everything I had done up to this point?

What about my account, and domain name registrations?

[MikeDVB]

So... I just purchased my plan today, and was working on setting up my forum when this happened.
Will I loose everything I had done up to this point?

If we're unable to recover the data directly from the server, then yes, anything created today will be gone.

What about my account, and domain name registrations?

Your domain registration will be fine - if you wish - I can go ahead and create a new account for you on another server since you've just registered so you can get back to work. Just open a support ticket.

An ETA will be posted here as soon as we have one.

[Patrick.B]

Is it 100% positive you will have to use your backup or not?

I would not be able to work on it for another couple hours due to other stuff I need to get finished.

[ReneB]

I really really hope you get the filesystem fixed. i have done so much work the last 24 hours :/

[MikeDVB]

Is it 100% positive you will have to use your backup or not?

I would not be able to work on it for another couple hours due to other stuff I need to get finished.

If your account was created today, then it wouldn't matter because the backup was created before your account was created.

We're restoring the server now as the file system repairs left the file system in a state where there was simply too much data loss. It will need to run for a short bit before we will have an ETA.

[MikeDVB]

5.1 GB of 309 GB have been restored so far. The system ETA says 2.5 hours however this can fluctuate. We'll update this thread with any updates.

[MikeDVB]

11 GB Restored.

[MikeDVB]

All accounts created on the server today by direct customers have been recreated on another server and account details dispatched. We are restoring the server to our backup from last night which has an ETA of approximately 2.5 hours according to the system. 25.1 GB of 309 GB have been restored so far.

[MikeDVB]

35 of 309 GB Restored.

[nbaintel]

Mike, are you restoring the files from last night?

[XTremo]

Appreciate the updates and quick response Mike.

[MikeDVB]

Mike, are you restoring the files from last night?

Yes, we are as the file system was beyond repair.

Appreciate the updates and quick response Mike.

Absolutely.

58.5 of 309 GBs restored.