Jump to content


MikeDVB

Member Since 27 Sep 2008
Offline Last Active Oct 23 2020 02:00 PM

#3940 Security Advisory: Use SSL/TLS to connect to IMAP, POP3, and SMTP

Posted by MikeDVB on 23 October 2012 - 09:33 AM

Hello,

The big issue used to be FTP passwords that were sniffed due to non-encrypted FTP connections, but these days the biggest issue is email. When you configure your mail client (Outlook, MacMail, Thunderbord, Eudora, etc) you need to ensure that you are connecting via SSL or TLS. When you do not connect using SSL or TLS, you send your email address and password across the internet in plain-text for anybody malicious to see.

There are compromised systems out there on the internet that do nothing but sit and watch for email address and password combinations and the result is that the email address is used to send a ton of spam, the hosting account gets disabled/suspended, and the IP ends up blacklisted affecting everybody else on the server. In an event such as this, we very well could charge $75/hour for our time spent cleaning up the black-list and resolving to tickets opened by other customers as a result from such an issue. We really do not want to have to bill anybody for this.

Preventing the issue is so simple, there really is no justification for not updating your mail client to use a secure authentication method. We are looking into the possibility of forcing only secure connection methods, however, we are unsure at this time if the mail software will support such a limitation. In the event that this requirement is put into place, we will email our entire client base a few times over the course of a few weeks advising of the change.

If you have any questions at all, please do let us know. You are welcome to respond to this thread if you have a general question, or to open a support ticket if you have an account-specific question.
  • 1


#3874 WordPress - WP Super Cache - and LiteSpeed

Posted by MikeDVB on 01 September 2012 - 03:14 AM

Thanks for the great tips, Mike! I have it enabled on one of my sites now to test performance, and it is noticbly faster.

Indeed, serving a static file is a LOT faster than proessing PHP+MySQL queries and then building the page from that every load (when nothing has changed).
  • 1


#3665 Demeter Server Reboot - Friday, May 11th, 2012

Posted by MikeDVB on 10 May 2012 - 04:11 PM

Usually do - if you look through the older posts in the forum they all say EST :). Not saying you should have known that, just saying "Oops" :). Edited the original post.
  • 1


#3500 Gemini RAM Upgrade

Posted by MikeDVB on 02 February 2012 - 10:13 PM

We ordered new ram for the Echo and Gemini servers to upgrade them to 24 GB from their current 12 GB. This will allow much more cache for MySQL and disk I/O resulting in an overall improvement in performance across the board for all customers on these servers. These two servers are our last that are on 12 GB, and were upgrading them as a part of our continual process to improve our services. This upgrade will be performed tomorrow (February 3rd) between 9 PM and 12 AM EST and the estimated downtime is approximately 15 minutes for each server.

If you have any questions at all about this process, by all means, do let us know.
  • 1


#3496 How to track down bot ip's

Posted by MikeDVB on 29 January 2012 - 01:56 PM

It's possible to block a whole range by using the first three octets such as this:
255.255.255.

But listing more than 50-100 individual IPs makes the file very long and the server has to process it on each reload (which adds overhead).

I'm sure you can probably find somebody who will help you block it, probably for far less than $1000.
  • 1


#3486 How to track down bot ip's

Posted by MikeDVB on 27 January 2012 - 07:10 AM

To ban a bot by the name, you'd need to manually process the headers sent to your script by the bot and then selectively allow or deny them. It's not an easy task and not something you'd likely be able to sort on your own. It's well outside of our support scope so you would need to acquire a developer to handle that for you.

As far as blocking them via IP, you could do it via your .htaccess but I wouldn't suggest blocking more than 50~100 IPs using that method.

Realistically, welcome to the wild wild internet. It's a hostile place - you're going to find all kinds of things from bots scanning for exploitable software in your account to bots that click on your ads and de-value your site. Unfortunately there is little to nothing that can be done about it.
  • 1


#3418 Domain Transfers - A few new user questions?

Posted by MikeDVB on 05 January 2012 - 03:50 PM

It's industry wide - when you transfer it does include a one year renewal by default and it extends off of the current expiration date (not based off of when you transfer it).
  • 1


#3373 So... I'm going to be a father!

Posted by MikeDVB on 17 December 2011 - 02:56 AM

The wife had her first pregnancy confirmed by blood test and ultrasound! Estimated due date is mid-August. ;-)
  • 1


#3342 Should I worry about Disk /dev/sda5 (/)

Posted by MikeDVB on 03 December 2011 - 11:53 PM

These are the server statistics and not specific to your account.

Abstract it a bit ... 81% of what? 100gb would leave 19gb free but what it it's 1,000gb? That leaves 190gb free.

Basing alerts on raw percentages isn't great but there isn't really a better way to do it that is generic enough for all cPanel servers large and small.

Our internal alerts go off at 90% but at this time there is no way to customize the alert percentage inside of cPanel.

You can ignore the message, as it's not specific to you.
  • 1


#3263 All Down!

Posted by MikeDVB on 20 November 2011 - 06:32 PM

Here is a layman's example:
1. We have a neighborhood (our network of servers).
2. This neighborhood has two major roads that allow outsiders to get into the neighborhood, and people in the neighborhood to get out. (Traffic)
3. Only one road is used at a time, the other road is just there in case there is an issue with the primary road.

The primary road that runs to the neighborhood became inaccessible, and it took time for the traffic on that road to re-route to the other major road into the neighborhood. (around 2 minutes). Even after this change some cars (web requests) were still trying the old main road, before they realized it wasn't working and had to witch to the backup road.

The technical view:
Our network has dual redundant uplinks and dual redundant network distribution switches. The reason for having two of everything (redundancy) is so that if one component fails we aren't offline until somebody manually steps in and switches a piece of hardware, or worse that we are down until we get replacement hardware. One of the distribution switches (the primary) went offline and it took a couple of minutes for traffic to re-route over the backup distribution switch.

One thing to keep in mind is that both switches are identical, the uplinks are identical, etc... There is no reason for one switch or path to be faster than another. If things seem slower to you, it's likely just your perception since you're paying attention. A good layman's example of this is that if your car makes a certain noise that you don't notice - you may suddenly notice this noise after you do something to the car (like replacing a part) because you're paying specific attention to the sound and listening for differences where you were not previously.
  • 1


#2751 [Completed] Fresco Server - Restoration Completed - File System Check Competed

Posted by MikeDVB on 21 June 2011 - 09:50 AM

I'd like to see a real back up plan in place.

We have a "real" backup plan. I'm not sure what makes you think otherwise.

-Fresco goes down due to whatever.
-You have one machine on the ready to restore all accounts from backups
-Done!

This is exactly what we did.

Congratulations for all the hard work you do is not enough for me, I think you do have a responsibility to put more effort into a back up plan to keep everyone's accounts up and running. You may say that yes u're not responsible for DDos attacks, but others I'm sure have lost clients due to something not their fault as well.

Ok, well nobody is forcing you to stay - if you wish to leave, by all means do what you feel is best.

So we just have to endure this again and again?

DDoS attacks - there's nothing we can do to stop/prevent those all we can do is mitigate them when they happen. As far as file system issues - this is only the second time we've ever had to run file system checks in over three and a half years in business. If that is "again and again" then I'm sorry. Beyond that, when the file system is corrupted, we don't have a choice but to run a file system check.

I'm not blaming you, but more should be put into finding a solution for these situations.

DDoS - We mitigate it. File System Corruption - we repair the file system. This is exactly what we've done.

If I were there, I surely would be heading up a project to do so. Nothing is impossible.

Heading up a project to do what? If you know a way to prevent DDoS attacks then you could very easily make yourself a millionaire selling whatever product/solution you come up with to prevent such attacks. And beyond that - if you come up with a way to prevent file system corruption on a kernel panic or other unexpected issue where disk caches may not be fully flushed to the disks - you should patent it.

I understand you are upset and/or frustrated but to be honest you really don't seem to have a solid grasp as to what exactly happened and what we did to bring everything back online. By all means, if you feel the grass is greener on the other side of the fence - feel free to find a new provider. This could happen to any provider on any server at any time - what is important about it is how that provider handles the issue. If there is something more you feel we should have done, by all means, do let me know what you think we should have done.

Nobody is perfect and we don't claim to be - we just do our absolute best to provide the best service and to resolve issues as quickly and efficiently as possible. We're not above and beyond taking constructive criticism and advice but if you're going to give advice, make sure you know what you're talking about before you do so. So I'm all ears, let me know what you think should have been done differently.
  • 2


#2538 [Completed] Fresco Server - Restoration Completed - File System Check Competed

Posted by MikeDVB on 20 June 2011 - 03:53 AM

Original Message:
The fresco server is currently under a DDoS attack which is sending about 110,000 attack packets per second. We're working on mitigating this attack at this time.

Update at 06:35 AM GMT-5 on June 20th:
While shifting accounts off of the IP under attack the server crashed and upon reboot is requiring a file system check. We have not yet determined what caused the crash resulting in the file system check, however, there is no safe way for us to skip the file system check.

Once the check is completed we will reboot the server and then see where we stand, the check could take anywhere from 45 minutes to 6 hours (there is no ETA provided by the file system checker) and will vary greatly depending on how damaged (or not) the file system is.

We'll update the thread as we know more.

Update at 07:00 AM GMT-5 on June 20th:
The server crashed due to what we believe is hardware failure. This is coincidental and not due to the DDoS attacks. We are going to have to perform a file system check.

Update at 2:25 PM GMT-5 on June 20th:
The file system check completed around a half hour ago and it was determined that too much file system damage had been done. We have initiated a full restoration of the server, including all customer and account data, to our restoration point generated between 12:30 AM and 1:30 AM GMT-5 last night. The current ETA from our backup system is 5 hours and 30 minutes at the time of this update. Keep in mind that the ETA is exactly that, estimated, and it could finish faster or slower.

We will continue updating this thread as the restoration is performed.

Update at 9:22 PM GMT-5 on June 20th:
The restoration completed however the file system still had issues and as such we've initiated another restoration back one additional day. The ETA for this restoration is approximately 6 to 7 hours. We will keep this thread updated with periodic updates.

Update at 5:00 AM GMT-5 on June 21st:
The server restoration has approximately 1 hour remaining. Once it is completed there is a good chance we will need to conduct a file system check which may take 2 to 6 hours depending on the state of the file system after restoration. We will update this thread once the restoration is complete to let you know if a file system check was required or not.

Update at 5:38 AM GMT-5 on June 21st:
The server does require a file system check which can take anywhere from 30 minutes to 6 hours, we'll update this thread once the check is completed but there is no way to provide progress reports as it runs as there is no output but what is being repaired.

Update at 6:28 AM GMT-5 on June 21st:
The file system check completed and due to the repairs done we're running an additional file system check to ensure that once the server comes back online that it is stable and doesn't have to be taken back offline for another file system check later today or in the near future.

The first check took approximately 50 minutes and we estimate the second check will take approximately the same amount of time.

Update at 6:46 AM GMT-5 on June 21st:
The second pass of the file system check is 42% completed.

Update at 7:21 AM GMT-5 on June 21st:
The system is booting up, the restoration process is finished and the file system checks are done.

Update at 7:26 AM GMT-5 on June 21st:
The server booted up successfully and all services are restored. We do need to update the DNS records for a large number of accounts due to the half-completed DDoS mitigation when the server crashed. This process is expected to take between 2 and 12 hours as we have to manually process each and every DNS zone and make the necessary changes.

Update at 2:30 PM GMT-5 on June 21st:
We are approximately half way done fixing DNS zones, we estimate completion around 7 to 9 PM GMT-5.

Update at 9:38 PM GMT-5 on June 21st:
All DNS zones have been updated and all sites are 100% functional. If you've tried visiting your site and gotten a "cPanel Default" page then you will likely have to wait an additional 1 to 2 hours as your local computer and your ISP DNS caches expire. Anybody who hasn't visited the site recently (i.e. a new visitor to your site) will see your site instantly.

Update at 11:30 PM GMT-5 on June 21st:
We have checked all accounts whose DNS was updated and verified that they are all coming back correctly. If you are experiencing any issues at this point do please open a support ticket.
  • 1


#2449 [Completed] Gemini full restoration from backup system.

Posted by MikeDVB on 29 May 2011 - 04:30 PM

Indeed, we're at 200 of 309 GB restored so it shouldn't be much longer. We don't anticipate any issues booting up the server once the restoration is completed but we're keeping extra staff on hand just in case.
  • 1


#2399 [Resolved] Fresco Outage - DDoS

Posted by MikeDVB on 18 May 2011 - 12:55 PM

Thank you for your hard work, Mike :)

Absolutely, myself and our networking team :) I wish the internet wasn't such a hostile place.

Not that any of you have compromised computers (surely we all have secured systems!) but if you're not actively doing this, make sure you do it:
  • Make sure you have quality up-to-date virus scanning running.
  • Make sure that you do periodic scans for malware such as with malwarebytes.
  • Make sure that you keep your operating system up to date.
  • If you're not running a server or something that requires your computer to be on, do turn it off when you're not using it.

DDoS attacks come from compromised systems that are left online and do not have the necessary security patches, virus scanning, etc... If everybody followed these simple guidelines a vast majority of DDoS attacks wouldn't be possible as the computers that were a part of the botnets would be secured.
  • 1


#2397 [Resolved] Fresco Outage - DDoS

Posted by MikeDVB on 18 May 2011 - 01:29 AM

All accounts that were on the affected IP have been shifted off of the IP so service should be fully restored to everybody. It is likely that the attack will shift to the new IP of whichever account was originally targeted at which point we will further isolate the target to fully mitigate the attack.
  • 1