[SarisIsop]

Hello my site and cPanel, is down! your website was down too.

Back up now but slow, just thought I would let you know.

[fshagan]

I suspect Mike has been in "panic mode" the last hour or so. My monitors indicated somewhere around a 13 minute outage (that could be as little as 5 minutes based on the monitoring frequency). Or it could have been between my ISP and the data center (but then you noted the same issue).

It takes a while for the servers to come back up to full speed after an outage.

[SarisIsop]

Thanks for that, I'm still new to this.

I will put an announcement on my forum but they have all logged off now.

[MikeDVB]

After getting with the networking team, it seems that a distribution switch in our cabinet decided to go kaput. Thankfully we have a secondary switch that picks up and takes over in this event, but the change over can take 2 to 4 minutes and then there is the possibility of intermittence for 5 to 15 minutes afterwards. The total downtime registered by our monitoring is right at about 2 minutes but it's possible individual routes to individual IPs took longer to update.

The switches are identical, so we will be swapping the failed switch with a replacement and then setting it up as the backup (so that there is no downtime switching back to a new switch).

[SarisIsop]

Thanks Mike glad you know what your saying but I don't have a clue what your talking about.

It's working now but things still seem a little slower than usual, It's time for bed here so I will see how it is tomorrow.

Cheers.

[MikeDVB]

Here is a layman's example:
1. We have a neighborhood (our network of servers).
2. This neighborhood has two major roads that allow outsiders to get into the neighborhood, and people in the neighborhood to get out. (Traffic)
3. Only one road is used at a time, the other road is just there in case there is an issue with the primary road.

The primary road that runs to the neighborhood became inaccessible, and it took time for the traffic on that road to re-route to the other major road into the neighborhood. (around 2 minutes). Even after this change some cars (web requests) were still trying the old main road, before they realized it wasn't working and had to witch to the backup road.

The technical view:
Our network has dual redundant uplinks and dual redundant network distribution switches. The reason for having two of everything (redundancy) is so that if one component fails we aren't offline until somebody manually steps in and switches a piece of hardware, or worse that we are down until we get replacement hardware. One of the distribution switches (the primary) went offline and it took a couple of minutes for traffic to re-route over the backup distribution switch.

One thing to keep in mind is that both switches are identical, the uplinks are identical, etc... There is no reason for one switch or path to be faster than another. If things seem slower to you, it's likely just your perception since you're paying attention. A good layman's example of this is that if your car makes a certain noise that you don't notice - you may suddenly notice this noise after you do something to the car (like replacing a part) because you're paying specific attention to the sound and listening for differences where you were not previously.

[SarisIsop]

I can understand that thank you.

[MikeDVB]

No problem .

[SarisIsop]

Update: just wanted to say thank you all is working fine now.