Jump to content


Photo

How to track down bot ip's


  • Please log in to reply
15 replies to this topic

#1 Mike W

Mike W

    Newbie

  • Members
  • Pip
  • 8 posts

Posted 23 January 2012 - 07:14 PM

Hello everybody. I'm having a bit of a problem here. I'm new to cpanel and am probably missing something, so please bear with me.

About a week ago, I noticed a huge increase in the CTR for my Adsense ads. After looking into it, I found out that only one type of add was getting increased clicks. It was the "animated image" ad type. It went from a CTR of under 5% to 35%+. The CTR on my other ad types are all normal. Since I changed the ads back to "text" only, everything has gone back to normal (though I now seem to be smart priced) <_<

I'm almost sure this is some misguided bot. If it were malicious, they would be clicking more than just "animated image" ads (which are by far in the minority)

I've been monitoring with Statcounter, but it doesn't appear to even show all bot activity. In Awestats, it shows a list of 10 or so bots. I want to ban all the ones that aren't Google, Yahoo etc...But I don't see the ip for these bots listed anywhere. I looked through some of the other options in cpanel and didn't see anything useful except for the raw access logs. But it takes a huge amount of time to go through these, and I'm not even sure what I'm looking for.

Also, does Awestats show all bot activity. I located a Sitelock bot in my raw access logs and banned it. But this bot was not listed in Awestats.

Sorry this is so long, I'm cpanel noob. I tried getting in contact with Adsense through their invalid click activity link. No luck so far.
  • 0

#2 fshagan

fshagan

    Member

  • Members
  • PipPip
  • 145 posts

Posted 24 January 2012 - 09:47 AM

I think the stats programs list bots that "self identify", so they won't list ones that don't identify themselves.

You can limit bots using a robots.txt file in your site root, but ill behaved bots will simply ignore robots.txt. One way to catch them is to create a "bad bots trap" ... there are several different variations out there, but most create a folder or link specifically denied to all bots in robots.txt and then track and see which bots ignore your directives.

The problem with banning IP addresses of bots is that they are often shared IPs, and you are also banning any legitimate users with that IP address. You can get lists of bad bots by Googling "bad bots list", but they list from 120 to 4,500 IP addresses.
  • 0

#3 Mike W

Mike W

    Newbie

  • Members
  • Pip
  • 8 posts

Posted 24 January 2012 - 02:56 PM

I think the stats programs list bots that "self identify", so they won't list ones that don't identify themselves.

You can limit bots using a robots.txt file in your site root, but ill behaved bots will simply ignore robots.txt. One way to catch them is to create a "bad bots trap" ... there are several different variations out there, but most create a folder or link specifically denied to all bots in robots.txt and then track and see which bots ignore your directives.

The problem with banning IP addresses of bots is that they are often shared IPs, and you are also banning any legitimate users with that IP address. You can get lists of bad bots by Googling "bad bots list", but they list from 120 to 4,500 IP addresses.


Thanks for the response fshagan, hopefully I can work this out somehow. It's wreaking havoc on my Adsense earnings.

I wonder why the bot only clicks "animated image" ads. And why it keeps coming back. Even though I've switched to text only ads, 7 or so "animated image" ads still are displayed everyday. Apparently, this bot still finds them and clicks them.

This makes it appear as if it's crawling my sites constantly. Doesn't it have something else to do? :)
  • 0

#4 fshagan

fshagan

    Member

  • Members
  • PipPip
  • 145 posts

Posted 24 January 2012 - 08:18 PM

Big Dan brings up an interesting point ... I wonder if you reported it to Adsense and let them know you aren't generating the fake clicks, if they might have a way to identify the culprit for you.

You might also check out Webmaster Tools from Google and Google Analytics; they might be able to help identify the bot as well.
  • 0

#5 Mike W

Mike W

    Newbie

  • Members
  • Pip
  • 8 posts

Posted 24 January 2012 - 09:24 PM

@Big Dan and fshagan

Yes, I did report it to Adsense twice through their invalid click activity link. They never responded. I posted the same info as above.

I asked for the ip of the bot that was clicking the ad.

Adsense is discounting the clicks on that ad type. Clicks on that ad type range between 5 cents and 0. My average CPC is around 50 cents.

But this seems to be hurting my CPC as a whole. My average CPC used to be over 80 cents.

And as I said above, I switched all my ad units over to "text only," but for some reason, the "animated image" ad still appears. For about every 1000 impressions, the offending ad unit will appear about 5 times. And it gets clicked a lot. How could this be? I don't know much about bots, but I don't understand how this could be possible. If a bot was constantly visiting my site, wouldn't my page impression stats be super inflated?

Here's an idea of some stats from individual ad units.

Text only ad CTR-about 2% CPC- about 50 cents
Rich media ad CTR-about 2% CPC- almost $2
Image ad CTR-under 2% CPC- over $1
Flash ad CTR-about 2% CPC- over $1
Animated Image ad CTR-about 15% CPC-5 cents The stats for this ad unit used to be about the same as the flash ad unit, I've been using Adsense for almost a year.

Weird, maybe Adsense is giving me bogus stats.

I'll try to contact them again.

Thanks for the responses.
  • 0

#6 Mike W

Mike W

    Newbie

  • Members
  • Pip
  • 8 posts

Posted 25 January 2012 - 09:21 PM

It seems I'm not the only one having this problem. In this thread, an Amazon bot was clicking ads. One guy used the .htaccess file to redirect the bot back to Amazon. B)

In this thread, a bot was clicking flash ads only. Here's another one involving the Amazon bot. But I haven't seen any Amazon bots.

Hmmm...just had a thought. @fshagan I mentioned in the OP that I banned a Sitelock bot, but I just banned the ip I found for it. You said these bots used rotating ips. So it could be coming back.

Sitelock is an upsell that my old host (iPage) uses. Apparently, it's still crawling around my sites looking for 'vulnerabilities.' Why does it not appear in Awestats under spiders, but it shows up in my raw log files? It clearly said 'Sitelock Bot'

I've read you can ban a bot by it's name or where it originates from. Can I do this with info from my raw access logs?

I know I'm kinds clueless and am asking a lot of vague questions. I will study up on this immediately. But wow! Where to start...

What would you do first if you were me?
  • 0

#7 Mike W

Mike W

    Newbie

  • Members
  • Pip
  • 8 posts

Posted 26 January 2012 - 10:39 PM

Don't forget Amazon has a huge cloud hosting service so it's not actually Amazon generating that clicks it's something someone is running on one of their servers.

--
If I were you I wouldn't worry much about it. Adsense will automatically filter out 'bad' clicks so long as they aren't coming from you IP address Google isn't likely to penalize you.

Block the IP range via htaccess (or cPanel's deny access feature) to prevent that bot from sucking down your bandwidth.


It's not just about not getting banned, my $1 to $4 clicks have vanished since this started. I've been smartpriced and it's affecting my whole account.

How can I ban these bots that show up in Awestats? Here's an example of one: Unknown robot (identified by 'spider') Hits 641+58

I don't know how to find the ip range for this bot. Can I ban it by what it identifies itself as ('spider')? And what does the "+" mean under 'hits'? Either way, it appears as if it has visited my site more than 600 times this month. The bot that is causing the invalid clicks is surely visiting my sites quite a bit. Maybe that would be a good place for me to start.
  • 0

#8 Mike W

Mike W

    Newbie

  • Members
  • Pip
  • 8 posts

Posted 26 January 2012 - 11:31 PM

Can I ban a bot by it's name, or just ip?

It looks like in the example I mentioned earlier, the person banned the bot by it's name.
  • 0

#9 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 27 January 2012 - 07:10 AM

To ban a bot by the name, you'd need to manually process the headers sent to your script by the bot and then selectively allow or deny them. It's not an easy task and not something you'd likely be able to sort on your own. It's well outside of our support scope so you would need to acquire a developer to handle that for you.

As far as blocking them via IP, you could do it via your .htaccess but I wouldn't suggest blocking more than 50~100 IPs using that method.

Realistically, welcome to the wild wild internet. It's a hostile place - you're going to find all kinds of things from bots scanning for exploitable software in your account to bots that click on your ads and de-value your site. Unfortunately there is little to nothing that can be done about it.
  • 1
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#10 Mike W

Mike W

    Newbie

  • Members
  • Pip
  • 8 posts

Posted 27 January 2012 - 11:12 PM

Hello Mike, thanks for the response. Let me first say that I'm glad I switched to MDD hosting from iPage. My pages now load a lot faster.

Why would you not recommend blocking more than 50 - 100 ip's? I've read that when people ban bots, they sometimes have to ban a whole range of ip's.

Banning this bot would justify spending good money at odesk or something. It literally cut my Adsense earnings in half. And I'm willing to spend some money to get it to disappear for a period of time.

I'm willing to spend a $1,000 or more to get rid of this bot. Do you think this would be enough $$$ to hire an American from odesk to ban the bot by name? I have 4 websites.

Wow, what a nightmare...But I try to take it in stride :)
  • 0

#11 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 29 January 2012 - 01:56 PM

It's possible to block a whole range by using the first three octets such as this:
255.255.255.

But listing more than 50-100 individual IPs makes the file very long and the server has to process it on each reload (which adds overhead).

I'm sure you can probably find somebody who will help you block it, probably for far less than $1000.
  • 1
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#12 fshagan

fshagan

    Member

  • Members
  • PipPip
  • 145 posts

Posted 30 January 2012 - 09:19 AM

Banning this bot would justify spending good money at odesk or something. It literally cut my Adsense earnings in half. And I'm willing to spend some money to get it to disappear for a period of time.


You can ban bots that don't adhere to the robots.txt file by having a "secret" folder and file that is disallowed in robots.txt and not visible to normal web surfers (no links to it, etc.) Typically people put a no-index, no-follow header in it so it doesn't show up in search engines.

You then scan your logs for visits to that page, and ban the IPs that find it. Some programs have been written that provide the list of IPs ... see http://www.kloth.net...net/bottrap.php and http://danielwebb.us...tware/bot-trap/

You could hire someone to implement that kind of thing for you.
  • 0

#13 Mike W

Mike W

    Newbie

  • Members
  • Pip
  • 8 posts

Posted 31 January 2012 - 10:08 PM

You can ban bots that don't adhere to the robots.txt file by having a "secret" folder and file that is disallowed in robots.txt and not visible to normal web surfers (no links to it, etc.) Typically people put a no-index, no-follow header in it so it doesn't show up in search engines.

You then scan your logs for visits to that page, and ban the IPs that find it. Some programs have been written that provide the list of IPs ... see http://www.kloth.net...net/bottrap.php and http://danielwebb.us...tware/bot-trap/

You could hire someone to implement that kind of thing for you.


Thanks for the tip. Much appreciated. I'm waiting for a reply from a few people right now. Wish me luck. B)
  • 0

#14 fshagan

fshagan

    Member

  • Members
  • PipPip
  • 145 posts

Posted 01 February 2012 - 09:19 AM

Thanks for the tip. Much appreciated. I'm waiting for a reply from a few people right now. Wish me luck. B)


Absolutely ... let us know about your solution when you get there!
  • 0

#15 traditionalpainter

traditionalpainter

    Newbie

  • Members
  • Pip
  • 5 posts

Posted 03 November 2012 - 07:20 PM

Thanks for the tip. Much appreciated. I'm waiting for a reply from a few people right now. Wish me luck. Posted Image


How did you get on with the bots issue. I noticed an unnamed bot in Awstats used over 2GB bandwidth, would like to nail them
  • 0

#16 anmol77

anmol77

    Newbie

  • Members
  • Pip
  • 1 posts

Posted 20 April 2016 - 09:38 PM

I don't know how to find the ip range for this bot. Can I ban it by what it identifies itself as ('spider')??????


  • 0




1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users