Jump to content


Photo

Kobold Server - Outage

Resolved

  • Please log in to reply
10 replies to this topic

#1 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,902 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 17 February 2016 - 10:04 AM

The Kobold server has gone unresponsive. We believe this to be a kernel [operating system] panic.

Normally we remote into the server's management platform and reset the server if this happens, however, the management platform on this server is not functioning normally so we've had to engage the facility support to have them reset the server.

This is the sort of issue that should not happen on our new platform and, even if it did, the server would reset on it's own. We will be migrating customers on the Kobold and Jasmine servers to the new infrastructure this next week and I am going to be sending out an email about that shortly.

I apologize for any trouble this may cause you and we will have it back online as soon as possible.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#2 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,902 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 17 February 2016 - 10:18 AM

The server is back online at this time - it will take 15 to 20 minutes for it to stabilize [an issue we don't have on our new infrastructure].

I will be investigating the cause but I do still believe it to be R1Soft Related.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#3 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,902 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 17 February 2016 - 10:26 AM

MySQL needs repaired - any static sites or sites with good caching will function properly.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#4 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,902 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 17 February 2016 - 10:33 AM

MySQL Crash Recovery is still in progress. Unfortunately it doesn't go alphabetically so I have no way of providing an ETA/Estimation.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#5 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,902 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 17 February 2016 - 10:41 AM

MySQL is online and operational.

We're still going to perform a server-wide database check/optimization/repair. It goes alphabetically.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#6 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,902 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 17 February 2016 - 10:45 AM

And it's gone back offline - I do not know why but I'm working on it/investigating.

It's too bad we didn't get this server migrated sooner [plans were to migrate it almost a month ago].
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#7 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,902 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 17 February 2016 - 10:51 AM

The second outage was due to the server being absolutely slammed with requests due to the first outage and exhausting RAM. It has 64 GB of ram which is a lot [new hardware has 256G] but 1,200+ PHP processes were fired off at around the same time causing a memory shortage.

I cleared the issue and it should be good to go - but I will keep an eye on it.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#8 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,902 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 17 February 2016 - 11:43 AM

The server was up/down for a short while but I was watching it and resolving the issues as they arose.

It should be stable at this point.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#9 Shelley

Shelley

    Newbie

  • Clients
  • Pip
  • 4 posts
  • Gender:Female

Posted 02 March 2016 - 03:44 PM

Afternoon - I've noticed some issues and wanted to quickly address here.  I have the email sent re the upgrade that will start tonite.  8:30pm eastern time is 5:30pm Pacific Time but hopefully most of my clients will have shut down by that point.

 

I've noticed multiple copies of emails coming through which I've tentatively attributed to whatever issues you are dealing with on the Kobold server.  Happens frequently and often enough that I am doing mass deletes.  I'll wait until after the upgrade to see if this still happens and if it does, start a support ticket.  It's not a problem, just an annoyance :-)

 

I've also had a couple clients complain of email issues which again we hope won't be an issue after tonite - support tickets will be started if so.

 

Finally, I hope you can advise server name asap - I would like to send out an email to my clients to advise them of the change with respect to downloading their emails which currently is kobold.supportedns.com.  It would be better if I could do this prior to the change :-)

 

Hopefully I've understood the basis of the email notification and appreciate MDD's efforts to resolve server issues and migrate to a new server.

 

thanks

Shelley


  • 0

#10 Scott

Scott

    MDDHosting Staff

  • Staff Administrator
  • PipPipPipPip
  • 421 posts
  • Gender:Male

Posted 02 March 2016 - 08:49 PM

Afternoon - I've noticed some issues and wanted to quickly address here.  I have the email sent re the upgrade that will start tonite.  8:30pm eastern time is 5:30pm Pacific Time but hopefully most of my clients will have shut down by that point.

 

I've noticed multiple copies of emails coming through which I've tentatively attributed to whatever issues you are dealing with on the Kobold server.  Happens frequently and often enough that I am doing mass deletes.  I'll wait until after the upgrade to see if this still happens and if it does, start a support ticket.  It's not a problem, just an annoyance :-)

 

I've also had a couple clients complain of email issues which again we hope won't be an issue after tonite - support tickets will be started if so.

 

Finally, I hope you can advise server name asap - I would like to send out an email to my clients to advise them of the change with respect to downloading their emails which currently is kobold.supportedns.com.  It would be better if I could do this prior to the change :-)

 

Hopefully I've understood the basis of the email notification and appreciate MDD's efforts to resolve server issues and migrate to a new server.

 

thanks

Shelley

 

Hi Shelly,

 

If you notice any issues with email, please do open a ticket so we can investigate. Everything should be going smoothly.

 

As for the server name, a mass email was just sent out with details for each account. Check your inbox. If you have any questions regarding tonights transfer, we have a forum post dedicated to that at http://forums.mddhos...pm-et-03022016/


  • 0
Scott S - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#11 Shelley

Shelley

    Newbie

  • Clients
  • Pip
  • 4 posts
  • Gender:Female

Posted 02 March 2016 - 10:14 PM

Yes, thank you - have received the email.

 

I won't open a ticket until I see how everything goes.  If still dealing with my or client issues, I will start one.  Until then, it can wait.

 

thanks

Shelley


  • 1





1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users