Michael D. Posted February 26, 2017 Report Share Posted February 26, 2017 Hello, We were alerted this morning to file system issues with the R3 server. Upon investigation we've identified file system errors and are working to recover from the errors and to restore service as soon as possible. I do not presently have an ETA but I will provide updates as I can. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted February 26, 2017 Author Report Share Posted February 26, 2017 I am actively working to repair the damage and have also engaged a couple of our Disaster Recovery procedures in the event that I'm not able to repair the damage. This may end up being more of an extended outage but we are doing our best to restore service as quickly as possible with as little data loss as possible. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted February 26, 2017 Author Report Share Posted February 26, 2017 The upside is that we have 3 different kinds of backups of this server - I'm working on getting a snapshot backup of the machine online which should offer near-instant recovery. Depending on exactly when the snapshot was taken there may be some data loss between the snapshot and right now. We can use one of our other backup systems to replace missing data once the outage is rectified. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted February 26, 2017 Author Report Share Posted February 26, 2017 The backup snapshot from last night is online - we're working on booting the server from that snapshot now. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted February 26, 2017 Author Report Share Posted February 26, 2017 A snapshot of the server is booting up - it should be online in a few minutes. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted February 26, 2017 Author Report Share Posted February 26, 2017 The server is online but has no networking - we're working to restore connectivity. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted February 26, 2017 Author Report Share Posted February 26, 2017 Networking has been restored - the machine is coming back up now. As it's running from a snapshot it may be slower than usual but once it's online we will migrate it back to our normal storage. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted February 26, 2017 Author Report Share Posted February 26, 2017 The server is fully online - we are in the process of migrating it from our snapshot to our normal storage. This is going to cause slowness while it happens. There is about ~1.3 TB of data to copy. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted February 26, 2017 Author Report Share Posted February 26, 2017 Migration is 34% Completed. Quote Link to comment Share on other sites More sharing options...
forumite Posted February 26, 2017 Report Share Posted February 26, 2017 Michael, appreciate you jumping on this one, and also appreciate the updates. Tom Quote Link to comment Share on other sites More sharing options...
Michael D. Posted February 26, 2017 Author Report Share Posted February 26, 2017 Migration is 36% Completed. There is a fair bit of data to copy. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted February 26, 2017 Author Report Share Posted February 26, 2017 38% Quote Link to comment Share on other sites More sharing options...
Michael D. Posted February 26, 2017 Author Report Share Posted February 26, 2017 42% Quote Link to comment Share on other sites More sharing options...
Michael D. Posted February 26, 2017 Author Report Share Posted February 26, 2017 The server is for the most part stabilized. There are still things to be done on our end - but I likely won't be updating this thread as often with the migration process as it appears, at this time, to have a minimal impact on performance. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted February 26, 2017 Author Report Share Posted February 26, 2017 44% - I am still actively monitoring the migration as well as the server itself. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted February 26, 2017 Author Report Share Posted February 26, 2017 46% Quote Link to comment Share on other sites More sharing options...
Michael D. Posted February 26, 2017 Author Report Share Posted February 26, 2017 50% - The server is still stable and I do not anticipate any further issues from this server today. Quote Link to comment Share on other sites More sharing options...
Dehenderson Posted February 26, 2017 Report Share Posted February 26, 2017 Your candor and transparency are always respected and appreciated. It's the level of professionalism that makes MDDHosting truly exceptional. Thanks. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted February 26, 2017 Author Report Share Posted February 26, 2017 The migration process has completed by 72%. The only upside I can think of as a result of this - is that we were able to experience our DR Processes in a real world scenario and have identified several changes that we will be making to improve the process and speed things up should we ever have to do this again. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted February 27, 2017 Author Report Share Posted February 27, 2017 The migration completed at 5:08 PM ET [approximately 34 minutes after my last update before this one]. Sorry again for the trouble. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.