Jump to content


Photo

Major Outage - 09/21/18 - 09/24/2018


  • This topic is locked This topic is locked
97 replies to this topic

#41 ericr

ericr

    Staff

  • Staff Administrator
  • PipPipPip
  • 219 posts
  • Gender:Male

Posted 24 September 2018 - 09:19 AM

Current ETA based on the progress up till now(Eastern Time):
s1    Completed

p1    Completed
r1    Monday, September 24, 2018 at 11:00:00 PM
p2    Monday, September 24, 2018 at 3:00:00 PM
s2    Tuesday, September 25, 2018 at 1:00:00 AM
r2    Tuesday, September 25, 2018 at 9:00:00 AM
s3    Tuesday, September 25, 2018 at 4:00:00 PM
r3    Wednesday, September 26, 2018 at 12:00:00 AM
s4    Wednesday, September 26, 2018 at 4:00:00 AM
r4    Wednesday, September 26, 2018 at 6:00:00 AM
s5    Wednesday, September 26, 2018 at 12:00:00 PM
s0    Wednesday, September 26, 2018 at 12:00:00 PM

  • 0

#42 richardmichael

richardmichael

    Newbie

  • Members
  • Pip
  • 3 posts

Posted 24 September 2018 - 12:17 PM

I have some general questions that anyone may answer, as I do not want to take up any of MDD's time.

I am not well versed in any of this hosting stuff.  I am a mechanical engineer whose company website and e-mail server are hosted by MDD.  I am one of the two employees, total, and will be the one to handle any of the work for this.

 

1.  How do I find out what server I am on?

2.  Is it safe to assume that being a newer client (January 2018), my server will be one of the last restored?

3.  Will I have to do anything once things are back up and running?

 

Thanks in advance.

1. You should be able to find it on the MDDHosting website, under your services.   Go here https://www.mddhosti...action=services, and for each service listed, click on it, then scroll down a bit and you will see something similar: https://www.dropbox....ervers.PNG?dl=0

You can see that my service is hosted on R4.

 

2. Probably likely

 

3. If you are using 3rd party DNS, you have a chance of needing to update your IP address configuration.  If you don't use 3rd party DNS, you will want to still monitor anyway and possibly open a ticket after restoration if you get an error about the IP address having changed.
 

Edit: After re-reading I see you may not be familiar about your DNS situation.  You can visit https://intodns.com and plug in your domain name(s) (one at a time, though), and run the report.  the very first section of the resulting report will tell you the Nameservers being used.  If you see supportedns like in my screenshot, you are using MDDHosting's DNS and are less likely to have an IP Address issues after the restoration reaches your server.


  • 2

#43 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 24 September 2018 - 12:50 PM

Here are revised ETA's:

 

 

s1    Completed
p1    Completed
r1    Tuesday, September 25, 2018 at 3:00:00 AM EDT
p2    Monday, September 24, 2018 at 5:00:00 PM EDT
s2    Tuesday, September 25, 2018 at 5:00:00 AM EDT
r2    Tuesday, September 25, 2018 at 1:00:00 PM EDT
s3    Tuesday, September 25, 2018 at 8:00:00 PM EDT
r3    Wednesday, September 26, 2018 at 4:00:00 AM EDT
s4    Wednesday, September 26, 2018 at 8:00:00 AM EDT
r4    Wednesday, September 26, 2018 at 9:00:00 AM EDT
s5    Wednesday, September 26, 2018 at 4:00:00 PM EDT
s0    Wednesday, September 26, 2018 at 4:00:00 PM EDT

 

While I do apologize they keep changing - they are estimates based off of our best working information at the time.  This is one of the reasons I'm not a huge fan of ETA's - they create expectations that maybe aren't kept up with.  This is one of the reasons we keep updating them as we can to remain as accurate as possible.

 

We are presently copying the backups for R1 and P2 from the intermediary SSDs in FastBackup to the restoration array and the S2 backup from SlowBackup to a set of intermediary SSDs.

 

We will begin restoring R1 and P2 as soon as possible and it looks like P2 will begin in about an hour.  R1 looks to be taking longer than expected with an estimated start time of about 3 hours from now.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#44 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 24 September 2018 - 12:53 PM

I am working quickly to separate out client discussions in this thread into the new thread for this purpose -> https://forums.mddho...ent-discussion/


  • 1
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#45 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 24 September 2018 - 01:19 PM

I had an email sent to all account holders with an active account on S1 and P2 that outlines DNS / IP Changes / Signing into cPanel and finding the new IP.

 

We're doing our best to keep up with support tickets, however, we are swamped and an hour or so behind on responses right now.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#46 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 24 September 2018 - 01:20 PM

We will be starting the restorations of the P2 server in approximately 50 minutes and the R1 server in approximately 3 hours and 20 minutes.  These are simply the ETA's provided by the data copy processes inside of FastBackup to move the data to the array for restoration.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#47 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 24 September 2018 - 02:14 PM

The restoration of accounts on the P2 server has started.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#48 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 24 September 2018 - 02:15 PM

Current ETA to restoration on R1 is 2 hours 45 minutes.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#49 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 24 September 2018 - 02:36 PM

The P2 server is almost completely restored.  Not many accounts left to go.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#50 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 24 September 2018 - 02:39 PM

If you are not online yet - make sure to try accessing your cPanel from our client area.  If you can log into your cPanel your account is already restored and your domain is most likely not pointing to the right IP.

 

If your account isn't restored yet and you have backups of your own you want us to restore, please open a support ticket.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#51 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 24 September 2018 - 02:42 PM

Some accounts on S1 were restored without MySQL databases - I don't know why this is as of yet as every backup includes database backups - but if you find your databases aren't listed you can restore them via cPanel -> JetBackup -> Database Backups.

 

After doing so you will also need to go to cPanel -> MySQL Databases and re-create your database user and apply it to your databases.

 

We're happy to help you with it if you need it in a ticket and we're doing our best to keep up.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#52 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 24 September 2018 - 03:22 PM

Time to R1 Restoration is approximately 1 hour 40 minutes.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#53 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 24 September 2018 - 03:22 PM

The last 8 accounts on P2 are restoring now.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#54 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 24 September 2018 - 03:30 PM

The copy of data for the S2 server from Slow Backup to Intermediate Transfer DIsks is about 75% completed.  Once it's 100% we'll move the intermediate disks over to FastBackup and get it ready for restoration.

 

1 hour 30 minutes left preparing R1 for restoration from Fast Backup.


  • 2
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#55 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 24 September 2018 - 04:10 PM

P2 is pretty much done - the only accounts still restoring are absolutely massive in size.  Only two accounts still restoring.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#56 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 24 September 2018 - 04:59 PM

P2 server is done.  ETA to starting R1 Restorations is ~20 minutes.


  • 2
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#57 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 24 September 2018 - 05:51 PM

We are starting the restoration of the R1 server now.


  • 3
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#58 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 24 September 2018 - 06:14 PM

We upgraded a storage component on the new Fast Backup server and we are seeing a huge increase in performance over what we were already seeing.  This should cut down the total restoration time - we will have updated ETAs once we have conducted some more transfers.


  • 5
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#59 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 24 September 2018 - 06:28 PM

We performed some changes on the storage in both backup servers.  The fast backup server is now doing 4,770 GB/hour data rates while it was doing 500gb/hour.  The old backup server has been doing 318 gb/hour.  We're watching to see how much of an improvement the old backup server gets as a result of these changes too.

 

The bottle neck is getting the data from the old backup server to the fast one - but this process is substantially faster than restoring directly from the old backup server which was only doing about 50gb/hour.


  • 5
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#60 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 24 September 2018 - 07:06 PM

R1 is restoring now, should be done soon.

 

R4 got finished faster than expected and we should be restoring it soon.

 

S2 is also close to being ready to get restored.


  • 4
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users