Jump to content


Mike W's Content

There have been 8 items by Mike W (Search limited from 03-December 19)


By content type

See this member's

Sort by                Order  

#3498 How to track down bot ip's

Posted by Mike W on 31 January 2012 - 10:08 PM in Web Hosting

You can ban bots that don't adhere to the robots.txt file by having a "secret" folder and file that is disallowed in robots.txt and not visible to normal web surfers (no links to it, etc.) Typically people put a no-index, no-follow header in it so it doesn't show up in search engines.

You then scan your logs for visits to that page, and ban the IPs that find it. Some programs have been written that provide the list of IPs ... see http://www.kloth.net...net/bottrap.php and http://danielwebb.us...tware/bot-trap/

You could hire someone to implement that kind of thing for you.


Thanks for the tip. Much appreciated. I'm waiting for a reply from a few people right now. Wish me luck. B)



#3488 How to track down bot ip's

Posted by Mike W on 27 January 2012 - 11:12 PM in Web Hosting

Hello Mike, thanks for the response. Let me first say that I'm glad I switched to MDD hosting from iPage. My pages now load a lot faster.

Why would you not recommend blocking more than 50 - 100 ip's? I've read that when people ban bots, they sometimes have to ban a whole range of ip's.

Banning this bot would justify spending good money at odesk or something. It literally cut my Adsense earnings in half. And I'm willing to spend some money to get it to disappear for a period of time.

I'm willing to spend a $1,000 or more to get rid of this bot. Do you think this would be enough $$$ to hire an American from odesk to ban the bot by name? I have 4 websites.

Wow, what a nightmare...But I try to take it in stride :)



#3485 How to track down bot ip's

Posted by Mike W on 26 January 2012 - 11:31 PM in Web Hosting

Can I ban a bot by it's name, or just ip?

It looks like in the example I mentioned earlier, the person banned the bot by it's name.



#3484 How to track down bot ip's

Posted by Mike W on 26 January 2012 - 10:39 PM in Web Hosting

Don't forget Amazon has a huge cloud hosting service so it's not actually Amazon generating that clicks it's something someone is running on one of their servers.

--
If I were you I wouldn't worry much about it. Adsense will automatically filter out 'bad' clicks so long as they aren't coming from you IP address Google isn't likely to penalize you.

Block the IP range via htaccess (or cPanel's deny access feature) to prevent that bot from sucking down your bandwidth.


It's not just about not getting banned, my $1 to $4 clicks have vanished since this started. I've been smartpriced and it's affecting my whole account.

How can I ban these bots that show up in Awestats? Here's an example of one: Unknown robot (identified by 'spider') Hits 641+58

I don't know how to find the ip range for this bot. Can I ban it by what it identifies itself as ('spider')? And what does the "+" mean under 'hits'? Either way, it appears as if it has visited my site more than 600 times this month. The bot that is causing the invalid clicks is surely visiting my sites quite a bit. Maybe that would be a good place for me to start.



#3481 How to track down bot ip's

Posted by Mike W on 25 January 2012 - 09:21 PM in Web Hosting

It seems I'm not the only one having this problem. In this thread, an Amazon bot was clicking ads. One guy used the .htaccess file to redirect the bot back to Amazon. B)

In this thread, a bot was clicking flash ads only. Here's another one involving the Amazon bot. But I haven't seen any Amazon bots.

Hmmm...just had a thought. @fshagan I mentioned in the OP that I banned a Sitelock bot, but I just banned the ip I found for it. You said these bots used rotating ips. So it could be coming back.

Sitelock is an upsell that my old host (iPage) uses. Apparently, it's still crawling around my sites looking for 'vulnerabilities.' Why does it not appear in Awestats under spiders, but it shows up in my raw log files? It clearly said 'Sitelock Bot'

I've read you can ban a bot by it's name or where it originates from. Can I do this with info from my raw access logs?

I know I'm kinds clueless and am asking a lot of vague questions. I will study up on this immediately. But wow! Where to start...

What would you do first if you were me?



#3479 How to track down bot ip's

Posted by Mike W on 24 January 2012 - 09:24 PM in Web Hosting

@Big Dan and fshagan

Yes, I did report it to Adsense twice through their invalid click activity link. They never responded. I posted the same info as above.

I asked for the ip of the bot that was clicking the ad.

Adsense is discounting the clicks on that ad type. Clicks on that ad type range between 5 cents and 0. My average CPC is around 50 cents.

But this seems to be hurting my CPC as a whole. My average CPC used to be over 80 cents.

And as I said above, I switched all my ad units over to "text only," but for some reason, the "animated image" ad still appears. For about every 1000 impressions, the offending ad unit will appear about 5 times. And it gets clicked a lot. How could this be? I don't know much about bots, but I don't understand how this could be possible. If a bot was constantly visiting my site, wouldn't my page impression stats be super inflated?

Here's an idea of some stats from individual ad units.

Text only ad CTR-about 2% CPC- about 50 cents
Rich media ad CTR-about 2% CPC- almost $2
Image ad CTR-under 2% CPC- over $1
Flash ad CTR-about 2% CPC- over $1
Animated Image ad CTR-about 15% CPC-5 cents The stats for this ad unit used to be about the same as the flash ad unit, I've been using Adsense for almost a year.

Weird, maybe Adsense is giving me bogus stats.

I'll try to contact them again.

Thanks for the responses.



#3476 How to track down bot ip's

Posted by Mike W on 24 January 2012 - 02:56 PM in Web Hosting

I think the stats programs list bots that "self identify", so they won't list ones that don't identify themselves.

You can limit bots using a robots.txt file in your site root, but ill behaved bots will simply ignore robots.txt. One way to catch them is to create a "bad bots trap" ... there are several different variations out there, but most create a folder or link specifically denied to all bots in robots.txt and then track and see which bots ignore your directives.

The problem with banning IP addresses of bots is that they are often shared IPs, and you are also banning any legitimate users with that IP address. You can get lists of bad bots by Googling "bad bots list", but they list from 120 to 4,500 IP addresses.


Thanks for the response fshagan, hopefully I can work this out somehow. It's wreaking havoc on my Adsense earnings.

I wonder why the bot only clicks "animated image" ads. And why it keeps coming back. Even though I've switched to text only ads, 7 or so "animated image" ads still are displayed everyday. Apparently, this bot still finds them and clicks them.

This makes it appear as if it's crawling my sites constantly. Doesn't it have something else to do? :)



#3473 How to track down bot ip's

Posted by Mike W on 23 January 2012 - 07:14 PM in Web Hosting

Hello everybody. I'm having a bit of a problem here. I'm new to cpanel and am probably missing something, so please bear with me.

About a week ago, I noticed a huge increase in the CTR for my Adsense ads. After looking into it, I found out that only one type of add was getting increased clicks. It was the "animated image" ad type. It went from a CTR of under 5% to 35%+. The CTR on my other ad types are all normal. Since I changed the ads back to "text" only, everything has gone back to normal (though I now seem to be smart priced) <_<

I'm almost sure this is some misguided bot. If it were malicious, they would be clicking more than just "animated image" ads (which are by far in the minority)

I've been monitoring with Statcounter, but it doesn't appear to even show all bot activity. In Awestats, it shows a list of 10 or so bots. I want to ban all the ones that aren't Google, Yahoo etc...But I don't see the ip for these bots listed anywhere. I looked through some of the other options in cpanel and didn't see anything useful except for the raw access logs. But it takes a huge amount of time to go through these, and I'm not even sure what I'm looking for.

Also, does Awestats show all bot activity. I located a Sitelock bot in my raw access logs and banned it. But this bot was not listed in Awestats.

Sorry this is so long, I'm cpanel noob. I tried getting in contact with Adsense through their invalid click activity link. No luck so far.