Michael D. Posted January 6, 2023 Report Share Posted January 6, 2023 Many providers would be dishonest about this sort of situation, although that's not how we operate. We are always as open and honest as possible, and I will outline the details of this incident. If you have any questions at all, please do not hesitate to reply to this thread. At 1:18 AM ET we received several notices of service disruptions and immediately brought all hands on deck. We had four hyper-converged storage and compute nodes go offline all at the same time. We store data in triplicate, although losing four nodes at once is enough to disrupt services noticeably. Some client servers did go fully offline, while others experienced degraded performance. This disruption affected all Turbo Shared, Turbo Reseller, and VPS services. Most client servers were back online within 5 minutes, while some were offline for approximately 20 to 25 minutes. We had to perform extra checks before bringing some back online to avoid potential corruption. We had performed operating system [kernel] updates on all of our underlying compute nodes yesterday, January 5th, 2023, and initially believed that the outage was related to this. Ultimately it was discovered that there had been unauthorized access to the server management hardware [DRAC]. During the unauthorized access, the machines were rebooted to reset the system password. Once the reboot was completed, a script was executed to install a crypto miner [MoneroOcean]. We traced the actions taken with 100% certainty, and no client data was accessed in any way. Due to how we have configured our services, it's not a technical possibility to directly access client data from DRAC or even by obtaining access to the host node. While it's not possible to access client data from the host nodes, no attempts were made to do so. We monitor access to our DRAC systems, and we are sure that there was no prior unauthorized access before this incident. How did this happen? We have relied on our upstream provider to limit access to our DRAC system management platforms for many years. We have always known that DRAC is a target and, as such, never intended to have them exposed to the public. This imitation was removed at some point, exposing our DRAC systems to the public internet. How was the issue resolved and what steps are being taken moving forward? We immediately disabled all DRAC systems by moving them to our private management network. We also immediately disabled all network access to all host nodes - even those not affected - to be sure all unauthorized access was terminated. Our fast action on this allowed us to trace what actions were taken during the unauthorized access completely. We cut off the unauthorized access before they could hide their tracks and clean up after themselves. Our quick actions allowed us to see every command and script run and this has allowed us to reverse all actions taken completely. Due to the nature of our services, we can perform significant maintenance without disruption. While we have completely reversed all unauthorized actions taken and are confident there were no backdoors, we will still perform zero-trust updates of the four hardware nodes with unauthorized access. The zero-trust updates of the four nodes mean migrating client servers off, completely reinstalling the operating systems and virtualization from scratch, and then migrating client servers back. You will not see any disruption from these activities. In the process, we will upgrade from CentOS7 to AlmaLinux8 on our hardware nodes, which is a plus as we need to perform this network-wide maintenance soon. What is being done to prevent this from happening again? We will no longer rely on our upstream to protect our DRAC systems. We have moved all DRAC systems to our private management network. They are only accessible moving forward by authenticated staff members on our private management network. Accessing the private management network requires authentication and physical access to our office. ---- I fully expect that you will have questions. Please do not hesitate to reply to this thread with your questions. You are welcome to open a support ticket if your question(s) is/are specific to your account. However, I ask you to put more general questions here as others may have the same questions. We apologize for the disruption this caused and we appreciate your understanding. Thank you. 2 Quote Link to comment Share on other sites More sharing options...
SarisIsop Posted January 6, 2023 Report Share Posted January 6, 2023 Thank you Michael 1 Quote Link to comment Share on other sites More sharing options...
BikeGremlin Posted January 6, 2023 Report Share Posted January 6, 2023 Clearly explained and, just as importantly, caught in time and prevented from recurring. That's how it should be, and that's why I think MDDHosting is one of the best and most trustworthy hosting providers. Stay cool. Quote Link to comment Share on other sites More sharing options...
cziv Posted January 6, 2023 Report Share Posted January 6, 2023 Ips of the hackers? Country? Quote Link to comment Share on other sites More sharing options...
Michael D. Posted January 6, 2023 Author Report Share Posted January 6, 2023 10 minutes ago, cziv said: Ips of the hackers? Country? We have those details but I don't have them handy at the moment. We've already reported this the the appropriate law enforcement agencies. As there were no damages I don't know that they'll care - but it would be remiss of us not to report it. 1 Quote Link to comment Share on other sites More sharing options...
ivlad Posted January 7, 2023 Report Share Posted January 7, 2023 This has has proved to me that I made the correct choice when moving some of my client to MDD. I love the honestly. Thanks. 1 Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.