Anybody who experienced some sluggishness just a short bit ago, it was due to a rather large DDoS attack hitting our network.

Here is a network graph of the event:
<< Removed image from quote to shorten post and remove redundancy >>

As you can see in the sharp drop in traffic (the green spike) we mitigated the attack rather quickly and returned speed and service to normal.

Wow! That's huge. Do you have a 1,000 MB/s pipe to the internet?

We've audited our MySQL configurations and optimized them better for the specific needs of each server and the customers on that server. You should see increased speed across the board for any MySQL powered sites (including but not limited to WordPress, vBuletin, IPB, etc...).

If you have any questions at all, don't hesitate to ask!

Earlier this month, I moved off of cypress (the original semi-dedicated server) to a VPS. Given the timing of my move, I wasn't able to experience this MySQL optimization first-hand. I would love to hear from current shared and semi-dedicated customers to see if they "feel" a noticeable difference.

Peace,
Brian

Indeed the server is having some issues right now - it looks like R1Soft decided to verify it's full backup from yesterday which I've never seen it do before. R1Soft has been less than helpful but ultimately we have three choices:
1. Let it finish and have accurate verified backups of the server and your data.
2. Cancel it now, after it's been running so long, have it run tomorrow and cause issues again tomorrow.
3. Cancel the backup and then there is no backup should there be hardware failure or another unexpected form of data loss.

We worked to prioritize the web server and MySQL and de-prioritize the backup system but it seems that there is still an underlying disk performance issue at work that is causing problems. Try as hard as we can to get things to stabilize some sites are operating normally while others are simply not operating well or at all.

We do estimate once the backup verification finishes that things should speed up and get back to normal however if not we're going to have to continue investigating to see if there is another underlying issue causing performance degradation. The backup verification itself should have finished in less than 10 hours however it has been running for 15 hours and reports 17 minutes remaining.

We're actually going to be making big changes to the semi-dedicated platform as we're not happy with the current server's ability to keep up with intensive disk situations such as long-term backups.

I'm also working on bringing online a VPS temporarily where I can stage any accounts that are under 10 GB of disk usage that wish to be moved off of Cypress until the changes to the semi-dedicated offerings are completed.

I'm going to be posting a very detailed message as to why Cypress has been facing issues and what we plan on doing to fix the issue permanently. All customers on Cypress will be getting an email with the information as well as a link to the thread I'm going to post once it's ready.

I would love a temporary VPS until we can find a permanent fix -- assuming it's fully managed. My Linux skills are C+.

I'm surprised you're able to access your site. For me, my site doesn't even load.

The current server load doesn't appear bad. Must be an I/O issue?:
Server Load 7.13 (16 cpus)
Memory Used 45.9 %
Swap Used 0.61 %

Just a note, because it might be related seeing as this is the most recent action taken on the server I'm on. We've been extraordinarily slow for about an hour now. Cypress server. 8+ seconds to load a page on my sites.

P.s.
Ticket sent.

I can confirm that my site on cypress is experiencing the same issue. CloudFlare cache is kicking in b/c my site is timing out.

May be a belated reply, but I use CloudFlare as well on http://tangobohemia.com
(Wordpress, W3 Total Cache)

Started with CF about 6 months ago, liked the performance, except these days - but I think it is because of the hosting upgrades/downgrades/reboots?

When we were dealing with the I/O issues, I could definitely feel the sluggishness even through CloudFlare. The reason is that CF's Content Delivery Network doesn't deliver cached HTML when your site is operational. CloudFlare's nginx web server does an immediate pass-thru of MDDHosting's HTML response to your browser.

If MDDH is getting pegged with a DDOS, or is experiencing a technical issue that causes a performance hit, you'll feel it. CloudFlare actually thought my site was offline at the peak of the issues were were having in March, so the "Always Online" feature occasionally kicked in and a cached version of my site was delivered instead.

A good way to get a sense of the bottleneck is to visit the CloudFlare-optimized version of your site, then visit your site directly (e.g. direct.yourdomain.com or whatever you setup in CF's DNS). Third party tools like YSlow can help pinpoint the problem. If static files like javascript, css, and image files are taking forever to load on the Cloudflare-Optimized version, it's a cloudflare issue. Check out their system status page for more info: http://www.cloudflar...tem-status.html

Peace,
Brian

It's a MacBook Pro (mid 2010 edition), AirPort Express (Dual Band, Gigabit), and then the other system I'd be connecting to is Windows7. I can run many of the Linux utilities on the Mac and I can toss a different disk (just for convenience) into the Windows 7 machine and boot it up into Linux or I could use a LiveCD

The pv tool is pretty slick! You'll likely need to yum/apt-get it for linux or use homebrew to install it for mac.
http://www.ivarch.co...ograms/pv.shtml

The dd command might need to be tweaked with different bs (block size) and count depending on how much junk you need to send across the wire.

Whatever option you choose, I would be interested in the results.

Peace,
Brian

I'm seeing a lot of tools but most of them are fairly complicated. Essentially all I want to do is blast X MBPS traffic to IP Y over TCP or UDP and then measure how much traffic actually makes it through versus what is sent

Assuming you access to *nix-based boxes, this solution should work for you:
http://serverfault.c...two-points/5119

1. Listen and pipe to /dev/null on one machine

nc –l –p 7000 | /dev/null

2. Connect and pipe 100MiB of random data on the other

dd if=/dev/urandom bs=1M count=100 | nc 192.168.1.120 7000 –q 10

2 (alt). For realtime stats use pipeviewer

dd if=/dev/urandom bs=1M count=100 | pv | nc 192.168.1.120 7000 -q 10

I have been trying to server all my static content from a cookie-less subdomain because it is supposed to improve site performance. I am having trouble getting my subdomain to not leave cookies. I am running a wordpress site on that domain and I put this into the wp-config.php:

define('COOKIE_DOMAIN', 'lrastart.org');

My site is in my signature.

Give this a try:

define('COOKIE_DOMAIN', '.lrastart.org');

Congrats on over 10,000 domains hosted! You guys should be proud of yourselves for coming this far.

You beat me to the congratulations! Well done Michael!

Thanks for that. I use WordPress with W3 Total Cache plugin and Cloudflare:

http://lrastart.org/

And I do like the speed

Excellent response time on page loads! W3 Total Cache is doing a much better job than Boost.

We just got with the network operations team at our facility and they're definitely seeing an ingress 500 MBPS+ hitting the server. We're putting a null-route in place for the target at which point the server will come back online.

The IP that is going to be null-routed does have several customers on it and we will be migrating them individually to new IPs so that we can identify the target of the attack more specifically and then that one particular account will be null routed until the attack subsides.

If you have any questions we do ask that you try and ask them here unless they are account specific so that we can provide centralized disbursement of information.

Is the new ip address temporary or permanent? I'll need to update my nameserver as I have my DNS hosted at cloudflare.

Peace,
Brian

http://www.screen-shot.net/2011-03-12_1725.png

It seems that today's issue and yesterday's issue are similar and likely related. It's a large influx of traffic into the server that causes the connection tracking in the server to fail resulting in the server becoming unresponsive. We're still working on identifying the source/destination and investigating ways to prevent this from happening again.

The server is online however it does take 10 to 15 minutes for it to "catch up" after being rebooted.

Thanks for the update.

It's not entirely accurate at all times and is only there just for those who are curious and shouldn't be used to diagnose issues Just see this graph of our network for example over the same period of time:
<< Edited out image, scroll up to see it in the quoted post >>
We're working on making these nicer more-accurate graphs available on the public side of things, but I can't promise when or if that will happen.

As far as the crash - the server went from having around 8 GB of RAM free (which is a lot, more than a lot of providers have total in their servers) to 0 and the server started killing processes to get some free ram back. This issue happend so quickly that the logging stopped before anything useful could be written to the disks to diagnose this and we were forced to perform a reboot.

We're setting up some additional internal monitoring on a very fast interval (something like 5 seconds) for the next 48 hours so if it happens again we will have some useful information to diagnose what happened.

At this point the server is back online and we're going to mark this as closed. If you have any questions at all, do feel free to ask them.

Hmm.. maybe cacti.mddhosting.com and cypress.mddhosting.com are the same box? If logging stopped, that would explain why the the graph shows some dead air during the outage.

Thanks for digging into this one Michael.

The server was rebooted within seconds of the alerts from our internal monitoring and took about 8 minutes to get back up to speed and we're still investigating the cause for the outage and will update you.

The network graph is particularly interesting:
http://www.mddhostin...erverstatus.php

I have noticed a steady decline on my video site for the last month or so. It appears that Google are now using a new crawler Algorithm. Today I noticed that my site isn't even being crawled by Google anymore - at least today - and my traffic has died a death. No traffic no dosh from my adverts - end of site! Pretty sure my site has been banned entirely! Posted a question about it on Google's forum and waiting for the answer. Don't think it will be good news though as the same site has been banned from Google adsense for ages.

What is your URL?

Has anyone used CloudFlare with MDDHosting's shared hosting? If you have, can you provide me with a URL so I can test the speed?

I am on the Semi-Dedicated plan, using the Drupal 6.x content management system, enabled the "Boost" module for Drupal (creates an auto-expiring static version of the HTML to minimize PHP/SQL lag time), and using CloudFlare.

Let me know if you like the speed!
http://www.brianstevenson.com/