MikeDVB

On April 11th our monitoring alerted us to an outage on the network.

This outage affected the entire facility in which we have our servers and was unfortunately outside of our realm of control.

We received the first alert at 04/11/2019 12:47:56 PM and verified all services back online at 04/11/2019 01:11:49 PM, after 24m of downtime. Times are eastern.

The RFO (reason for outage) can be seen at our upstream provider here: https://handynetwork...RFO 4.11.19.pdf. Ive also attached it to this post.

Hello!

 

Unfortunately a very high packets-per-second Distributed Denial of Service attack hit an IP on the S3 server tonight.  This attack wasn't large in the sense that it overwhelmed our network capacity but was large in the sense that it was a high enough number of packets that it was exhausting the web server's sockets and queues rendering sites on the IP offline.

 

We did identify the target of the attack and have moved them off to their own IP address - should the attack recur or adapt and we have to take action it should only affect the target site and not others on the server.

 

This attack was a new variant we haven't seen prior to tonight so we're using our packet captures to investigate how we could handle such an attack better and more efficiently should anything like it recur in the future.

 

If you have any questions about the attack do please open a support ticket.  Do feel free to reference this thread.

Hello!
 
We're seeing a couple of very large attacks that are targeting a couple of our servers - S0 and S1.  While all client sites are online and operational the IPs used for cPanel, Webmail, and most email access are currently un-routed.  Due to a misconfiguration in our Anti-DDoS protection that we're working to fix we're not presently able to route those IPs through our Anti-DDoS services.  We expect this to be corrected within a couple of hours.
 
In the meantime you can make the following changes to access cPanel and Webmail.
 
To access cPanel you would want to access the "cpanel" subdomain on your primary domain. So if, for example, your cPanel's primary domain is "test.com" you would go to "cpanel.test.com" in your browser. You may get an SSL warning but you can safely accept it/pass it.

 

To access webmail would be similar to cPanel in that you would connect to the "webmail" subdomain of your primary domain. For example if your cPanel's primary domain is "test.com" you would go to "webmail.test.com" in your browser. You may get an SSL warning but you can safely accept it/pass it.

Email Clients [Mac Mail, Outlook, Thunderbird, etc] - if you have them configured to connect to "s0.supportedns.com" or "s1.supportedns.com" you can change this to point to the mail subdomain of your cPanel's primary domain. If, for example, your primary cPanel domain is "test.com" you would connect your mail client to "mail.test.com". You may get an SSL warning from your mail client which you can permanently accept.

 

FTP - in most cases you can simply connect to your domain name.  There are some situations where this wouldn't work such as if you're using CloudFlare or Sucuri CloudProxy in which case you can connect directly to your account IP address.  You can find the IP address in your cPanel under 'Server Information' or at CloudFlare or Sucuri.

 

We do expect this to be resolved within an hour or two so if you just want to wait it out you can.  If you do make any changes to your mail or FTP clients - you do not have to revert them.

Drupal announced an update to Drupal core today to address two critical vulnerabilities. Drupal recommends users update their core:

If you are using Drupal 8.6.x, upgrade to Drupal 8.6.6.
If you are using Drupal 8.5.x or earlier, upgrade to Drupal 8.5.9.
If you are using Drupal 7.x, upgrade to Drupal 7.62.
 
Note: Versions of Drupal 8 prior to 8.5.x are end-of-life and do not receive security coverage. Sites on 8.5.x will receive security coverage until May 2019.
 
The vulnerabilities are announced as
Drupal Core - Third-party libraries - SA-CORE-2019-001

Drupal Core - Remote code execution - SA-CORE-2019-002

We're going to be rebooting all servers to apply a security patch.  The reboot will take 2 minutes or less per server.