Jump to content

Highest Reputation Content

#4992 Facility-Wide Disruption of Connectivity

Posted by MikeDVB on 28 February 2014 - 11:34 AM



We were able to identify the issue down to the Hurricane Electric and Telia transit providers.  We turned off those providers and all traffic is now routing over alternate unaffected routes - i.e. everybody should be back online at this point.


Understand due to the nature of this issue it is possible there will be further downtime/connectivity issues as there is no way for me to predict what changes have to be made to fully resolve the issue.  Turning off these transit providers is a band-aid over the symptoms and not a solution to the problem.  The goal was to bring everybody back online as quickly as possible.


We couldn't have isolated this issue without the traceroutes - thank you.


That said - if you are still experiencing connectivity issues do please email maintenance@ with a traceroute.

  • 3

#5546 SR1, SD1, VPS1 Outage

Posted by MikeDVB on 22 April 2015 - 05:50 PM

Yup - servers coming online.


It looks like a critical piece of software wasn't properly licensed.  I do believe I can place the fault for this matter squarely on my shoulders as it looks like I didn't install the licenses properly.  I do personally apologize for that.  Thankfully the process is well documented now and will not recur.


Due to the nature of the issue the servers were shut down gracefully so no data loss or damage is expected.  It would be as though it were simply a long reboot.

  • 2

#4412 Jasmine Emergency Kernel Update Status

Posted by MikeDVB on 14 May 2013 - 08:20 PM

Jasmine is online, it's playing catch-up to the monumental pile of requests it's facing due to being offline. I estimate 5~10 minutes before things normalize.
  • 2

#2751 [Completed] Fresco Server - Restoration Completed - File System Check Competed

Posted by MikeDVB on 21 June 2011 - 09:50 AM

I'd like to see a real back up plan in place.

We have a "real" backup plan. I'm not sure what makes you think otherwise.

-Fresco goes down due to whatever.
-You have one machine on the ready to restore all accounts from backups

This is exactly what we did.

Congratulations for all the hard work you do is not enough for me, I think you do have a responsibility to put more effort into a back up plan to keep everyone's accounts up and running. You may say that yes u're not responsible for DDos attacks, but others I'm sure have lost clients due to something not their fault as well.

Ok, well nobody is forcing you to stay - if you wish to leave, by all means do what you feel is best.

So we just have to endure this again and again?

DDoS attacks - there's nothing we can do to stop/prevent those all we can do is mitigate them when they happen. As far as file system issues - this is only the second time we've ever had to run file system checks in over three and a half years in business. If that is "again and again" then I'm sorry. Beyond that, when the file system is corrupted, we don't have a choice but to run a file system check.

I'm not blaming you, but more should be put into finding a solution for these situations.

DDoS - We mitigate it. File System Corruption - we repair the file system. This is exactly what we've done.

If I were there, I surely would be heading up a project to do so. Nothing is impossible.

Heading up a project to do what? If you know a way to prevent DDoS attacks then you could very easily make yourself a millionaire selling whatever product/solution you come up with to prevent such attacks. And beyond that - if you come up with a way to prevent file system corruption on a kernel panic or other unexpected issue where disk caches may not be fully flushed to the disks - you should patent it.

I understand you are upset and/or frustrated but to be honest you really don't seem to have a solid grasp as to what exactly happened and what we did to bring everything back online. By all means, if you feel the grass is greener on the other side of the fence - feel free to find a new provider. This could happen to any provider on any server at any time - what is important about it is how that provider handles the issue. If there is something more you feel we should have done, by all means, do let me know what you think we should have done.

Nobody is perfect and we don't claim to be - we just do our absolute best to provide the best service and to resolve issues as quickly and efficiently as possible. We're not above and beyond taking constructive criticism and advice but if you're going to give advice, make sure you know what you're talking about before you do so. So I'm all ears, let me know what you think should have been done differently.
  • 2

#2601 [Completed] Fresco Server - Restoration Completed - File System Check Competed

Posted by mk88 on 20 June 2011 - 05:29 PM

for all you guys that say you can't afford to lose one day of business etc, you need to use dns failover.

We started using dnsmadeeasy. Just use their DNS service and for each site that you need online at all times, you setup failover service. So it pings your site every 5 minutes. If it goes down, it will automatically change the DNS to point to another server that you have the site mirrored on.

For example, i use two reseller accounts. This one and another with just copies of our important sites. When everything is down like now, my DNS points to the IP that is online, and everything is good. Using this setup i have had virtually no downtime for a long time.
  • 2

#2337 [Resolved] Fresco Outage - DDoS Attack Mitigated

Posted by SnakEyez on 05 May 2011 - 06:48 PM

The total downtime from this incident was 12 minutes from start to finish.

I have to admit these updates are nice. But let's be serious, I own a Playstation 3, I laugh at 12 minutes of downtime.
  • 2

#1673 New Backup Node Online! More Restoration Points

Posted by TotalZen on 07 December 2010 - 09:18 PM

Here's a php script that will perform automated backups of your database(s) and email them to you.

What I did was set up a separate Gmail account for it, and then filter all the emails/attachments to the Trash folder so the account doesn't max out. The trash can automatically deletes emails after 20 days, so the whole thing is on auto-pilot with zero maintenance.

Simply run the script nightly as a cron job...

/usr/bin/php / /home/...(your path goes here).../cron_backup.php


$datestamp = date("Y-m-d");      // Current date to append to filename of backup file in format of YYYY-MM-DD


$dbuser = "database_user";            // Database username
$dbpwd = "database_pass";            // Database password
$dbname = "database_name";            // Database name. Use --all-databases if you have more than one
$filename= "dump_filename-$datestamp.gz";   // The name (and optionally path) of the dump file
$to = "email_address";      // Email address to send dump file to
$from = "mysql_backup@domain.com";      // Email address message will show as coming from.
$subject = "domain.com MySql Backup For: $datestamp";      // Subject of email


$command = "mysqldump --opt --quick --skip-extended-insert -u $dbuser --password=$dbpwd $dbname | gzip > $filename";
$result = passthru($command);

$attachmentname = array_pop(explode("/", $filename));   // If a path was included, strip it out for the attachment name

$message = "Compressed database backup file $attachmentname attached.";
$mime_boundary = "<<<:" . md5(time());
$data = chunk_split(base64_encode(implode("", file($filename))));

$headers = "From: $from\r\n";
$headers .= "MIME-Version: 1.0\r\n";
$headers .= "Content-type: multipart/mixed;\r\n";
$headers .= " boundary=\"".$mime_boundary."\"\r\n";

$content = "This is a multi-part message in MIME format.\r\n\r\n";
$content.= "--".$mime_boundary."\r\n";
$content.= "Content-Type: text/plain; charset=\"iso-8859-1\"\r\n";
$content.= "Content-Transfer-Encoding: 7bit\r\n\r\n";
$content.= $message."\r\n";
$content.= "--".$mime_boundary."\r\n";
$content.= "Content-Disposition: attachment;\r\n";
$content.= "Content-Type: application/x-gzip; name=\"$attachmentname\"\r\n";
$content.= "Content-Transfer-Encoding: base64\r\n\r\n";
$content.= $data."\r\n";
$content.= "--" . $mime_boundary . "\r\n";

mail($to, $subject, $content, $headers,"-f $from");

unlink($filename);   // Delete the backup file from the server (optional)


  • 2

#6274 S1 Email Issues.

Posted by Tim on 15 November 2016 - 11:36 AM

The cPanel update is now running.

  • 1

#6142 Customer service

Posted by jackydelarue on 03 June 2016 - 06:24 PM

Excellent customer service; could not be better.

Had answer and resolution to my problem in a few minutes.

Congratulations; good work.

  • 1

#5916 Jasmine, Kobold, SR2, SR3 Reseller Account Transfers - Beginning 9 PM ET 03/0...

Posted by MikeDVB on 03 March 2016 - 06:32 PM

All four servers: SR2, SR3, Jasmine, and Kobold have been migrated to their respective target servers.


As far as those that had MX entries disappear - the cause in every case so far is that the domain was set to 'LOCAL' or 'AUTO (Local)' via cPanel -> MX Entry.


In short people are editing their DNS zone and updating MX records or doing it via the MX Tool and not setting it to 'remote' when using remote mail.


We've written a script to check for this and correct it but unfortunately we can't go back into the past and fix it.  Moving forward it shouldn't happen again.


This has affected a small portion of our customers - but I am sorry for those that it has affected.


Open a ticket if you're having issues and we'll address your issues as quickly as possible.

  • 1

#5375 SD1, SR1, VPS1 Outage on 10/29/2014 @ 2:04 AM

Posted by MikeDVB on 29 October 2014 - 12:37 PM



First and foremost I want to apologize for this outage.  We were alerted at 2:04 AM to this issue by our internal monitoring and have been working on restoring since that time.  All hands were brought on deck and we were able to solve an obscure and undocumented issue restoring all services successfully after approximately 7 hours.  I wanted to go over the issue in a bit more detail and you will find that below.

We use Logical Volume Management to provide distinct storage to each piece of hardware.  The volumes are then thinly provisioned so that they can share the same overall pool of storage without using more space than they actually need allowing maximum efficiency of the storage.

Today at 2:04 AM ET the storage system for the SD1, SR1, and VPS1 servers went read-only with file system errors.  All server administrators were alerted and brought on duty to investigate and resolve the issue.  Upon initial investigation we determined that due to a single configuration error the servers were not giving back free space to the pool resulting in them growing and never shrinking.  We do monitor the storage but we were not correctly watching this metric.

All initial research on this specific issue indicated that the data was irreparably destroyed and we determined we needed to begin restoring our backups from 10:30 PM [3.5 hours before the issue] to hardware.  We brought up extra hardware and began restoring backups immediately.  While restorations were in progress we continued to work at recovering the data on the original storage.  It took us about 7 hours to successfully repair the data at which time we checked the restoration and it showed at least 3 more hours remaining.  

We then brought up the servers with the repaired storage and SD1 and SR1 came online immediately.  VPS1 needed a file system check but due to the storage being solid state these were completed within minutes and VPS1 was brought back online.

We have identified several changes that will prevent this from happening again.
* We will be properly monitoring for storage pool free space to catch this issue before it becomes a problem.
* We will be converting the storage to a state more easily managed
* We are looking at high availability storage that will prevent this issue.


You may find it unusual that we were not keeping the forums up to date as this is something we do when there is an outage.  I normally handle the forum updates personally but due to the critical nature of this outage I was focused on resolving the issue.  In the future should an issue arise where I am focused on restoring the service I will bring an additional staff member in for the sole purpose of keeping customers updated on changes as they happen if I'm not able to do it myself.


Outages such as these are extremely rare for us and we do apologize again for any trouble it may have caused you.

  • 1

#4931 DDoS Attack on Echo /

Posted by MikeDVB on 20 February 2014 - 11:32 PM

The filtration is almost in place, ETA ~10 minutes.


We're changing some of our routing as well to route our traffic exclusively through the 10GBPS links so that, with luck, it won't overwhelm any links and we can keep everybody online through the attack.

  • 1

#4060 Hi! Some questions!

Posted by MikeDVB on 22 December 2012 - 01:08 PM

Is the WHMCS free only for the first 3 years? It says Triennially.

It's free so long as your account is active, it's configured triennially so that you don't get a $0 invoice every month.

Can I create cPanel accounts that are "Unlimited" so I don't have to fiddle with resources constantly in case one site gets a traffic boost? It's gotten quite pesky to do, sadly.

Unlimited? No. Extremely high limits like 100000000 GB, yes. There isn't really any distinction between the two to be honest.

Are there coupons for returning customers for their loyalty if we have been with you guys for a while?

If you've been a customer in the past and wish to come back, you would just want to get with sales and see what we can do for you. There are a lot of factors that would affect what kind of discount if any we were able to offer for a returning customer, far too many to really lay out here.

Are we limited in domains or subdomains?

You set the limits yourself, but ultimately there is a finite limit (i.e. you can't host 1,000,000,000 domains on a reseller account).

Can we pipe email? Or control our MX entries (I've had a host long ago that disabled cpanel mx entries editing/mail all together).

Yes, Yes.

Can we use .htaccess files?


Can we use per-directory php.ini files?

No, php_value and php_flag within .htaccess - there is an entry in our knowledebase search it for "PHP Environment"

Will the sites listed above in the cPanel listings be fine on MDD Hosting reseller?

So long as you've reviewed our Terms of Service and you don't feel you'll be in violation I imagine you're fine - if you have specific questions about any of them, it's best to open a sales ticket.

Over a year ago we had one WordPress blog that had a faulty plugin. Our host disabled that site and told us the file using resources. I understand the disabling of the site. Will you guys do the same/disable and tell us the file/command that's causing the high usage so it's easier to track down and turn off if this happens again? (It was the old Flickr plugin, it had a loop)

We do what we can to work with you, but may not be able to tell you which specific file - something like the P3 Performance plugin for WordPress can, however, tell you which plugins are using the most resources, etc...

Can I mass email hosted users through WHM?

I believe so, but it's not something I recommend. I would suggest doing so via either WHMCS or something like phpList.

Can we disable cPanel accounts from using servermail at all so we can require them to use offsite like Google Apps instead?

You can disable their ability to create email accounts via packages, but cannot disable their ability to use php's mail() so yes and no.

Can I redesign my own cPanel?

If you have the skills to do so, yes :).

For the transfer. Can we only transfer a few sites over to see how they hold before transferring the rest with MDD's assistance?

If you're coming from cPanel you would just upload or otherwise make available a full cPanel backup and we can restore it, it doesn't take long. If you're coming from something other than cPanel you'll need to discuss the specifics with the sales department.

Can we use mod_rewrite?


Is it fine if we keep our domains off site/not controlled by MDD (We use name.com).

I don't see why not, so long as you have the ability to point to our nameservers. You can even use external DNS, but it's not recommended.

Some of the RPGs do have some adult content (it's very rare though). Is that acceptable? I see no clause in the TOS about this. The content is legal.

So long as it's not "XXX RPG 2.0" you should be fine. We only disallow sites whose sole purpose is distributing pornography, and not sites that may have occasional adult content.

Do you offer installation services for the SSL or are we on our own?

WHM->Install SSL Certificate - we can do it for you, or you can do it, but it's the exact same set of steps and takes only a moment either way.

Can we disable auto rebilling?

If you pay with PayPal and don't create a subscription, sure, you can pay manually each and every time. Even if you choose "PayPal" you can still pay with a card on the invoice itself. This will prevent the system from billing automatically but still allow you to use a card for convenience if you wanted.

When having a our previous reseller site transferred do you guys fix the DNS zones ips/information?

We don't do anything to the zones beyond what cPanel does itself to point the domain to the correct IP. I.e. if you have modified any of the default records such as the "www" cname or the a-record for the domain itself they may get reset - this is something you would need to check.

Does MDDHosting have any affiliation or link or connection with MDHosting? (The one D one, not two).

No. They were a terrible provider and went out of business quite a while ago.
  • 1

#3373 So... I'm going to be a father!

Posted by MikeDVB on 17 December 2011 - 02:56 AM

The wife had her first pregnancy confirmed by blood test and ultrasound! Estimated due date is mid-August. ;-)
  • 1

#2769 [Completed] Fresco Server - Restoration Completed - File System Check Competed

Posted by Saulius on 21 June 2011 - 02:23 PM

Hmm.. Wonder how long it takes for the DNS to update...

use http://host-tracker.com/
  • 1

#2463 [Completed] Gemini full restoration from backup system.

Posted by XTremo on 30 May 2011 - 04:28 AM

The only way you ever find out how good a provider is, is when something goes wrong.

I've seen numerous companies over the years who've built up a good reputation, they've been "flavour of the month" with a substantial following of enthusiastic clients.

But when their back has been up against the wall they've fallen apart....simply because they didn't have the systems in place to deal with it. Essentially there was no substance behind the style.

In this particular case, Mike and his team came through with KH levels of precision. Mike knows what I mean by that and it's no small compliment.
  • 1

#2081 Terms of Service Updated

Posted by MikeDVB on 24 March 2011 - 01:00 PM

It generally applies to the cPanel backup system (i.e. "full backup" or "home directory backup") for larger accounts. If your account is only 50~500 MB you could use these just about any time of the day without issues because the process completes so quick it's not an issue for performance on the server in most cases.

Dumping a database or copying files around while working on your site is just fine :)

Ultimately the provision hasn't been added so that we can start looking for this and take action, it's been added so that when and if it does cause issues we have the appropriate provision in the agreement to take action upon. It was inspired by the fact that we had a reseller with around 25 fairly large accounts start backing all of them up simultaneously mid-day resulting in extremely slow performance and numerous complaints.

We obviously got onto killing those processes off quickly to restore full service, notified the reseller that they needed to perform the backups during off-peak and not simultaneously, and updated the Terms of Service to address the issue.
  • 1

#1863 PHP 5.3?

Posted by kuemerle5 on 31 January 2011 - 06:12 PM

Was just wondering if/when MDDHosting will be upgrading to PHP 5.3. I'm asking because a bug tracker I would like to use requires at least 5.3 (The Bug Genie). Thanks in advance for any responses!
  • 1

#1347 [Completed] Cypress & Demeter Server Reboots

Posted by MikeDVB on 08 September 2010 - 03:16 PM

The reboots were not performed on the 5th and were postponed until the 12th (one week later). I neglected to update this thread, sorry about that.
  • 1

#1024 Video Streaming Options at MDD

Posted by MikeDVB on 18 March 2010 - 01:51 AM

The size of the binary has absolutely nothing to do with how CPU intensive it is. I could write a binary that is 100kb and could crash a server in a heart beat or a 120mb binary that uses next to no cpu or ram to run.

Where the resource usage comes in is the intense calculations in the transcoding the audio and video streams from one format to another. I personally run a Quad Core Q8400 at home and whenever I am transcoding a video from one format to another - as long as that process takes - I am using 100% CPU on all 4 cores and unless I greatly reduce the priority of that process the entire system will become very sluggish. While this is fine on my home PC when I am transcoding it is not ok in a shared environment.

You may ask why don't we simply reduce the priority of transcoding processes and the answer is that there is no easy way to pick and choose which processes are going to get lower or higher priorities on an automated basis. Your script could launch the transcoding process at a reduced priority but most scripts aren't configured to be able to do this - especially not drop-in plugins for scripts such as Joomla.

If you really need FFMPEG for your gallery script, Joomla, etc... open a ticket with the sales department about upgrading to a VPS and we will see what we can do to get you onto a VPS that you can afford and will allow you and your clients to do what it is that you are trying to do.

Our Virtual Private Servers capable of FFMPEG transcoding start at $40.50/month fully managed with cPanel+WHM and have no hard set limits on the number of accounts that you can host.

While it may seem as though we are limiting you on the standard shared/reseller plans - in truth what we are doing is working to guarantee the performance and reliability of said services. If everybody in our shared and reseller environments was able to launch FFMPEG conversion processes the server could potentially bog down very quickly under the extreme load those processes cause not only resulting in slow and sub-standard service but potential downtime.
  • 1