Unknown robot eating up bandiwdth

gangsta1

Regular Member

Oct 12, 2009

418

160

• Oct 27, 2012

• #1

In my AW stats I have almost 1tb being eaten up by:

Unknown robot (identified by empty user agent string)

How can I block this?

 

kvmcable

Supreme Member

Dec 28, 2010

1,354

2,823

• Oct 27, 2012

• #2

Block the IP or IP range if the bot isn't identifiable or ignore robots file

 

gangsta1

Regular Member

Oct 12, 2009

418

160

• Oct 27, 2012
• Thread Starter Thread Starter

• #3

The bot is unknown therefore I cannot find the IP. I cannot ignore bots as there are valid bots crawling the site. Is there a way to fin the IP of that "unknown" bot?

 

alternatesword

Elite Member

Aug 25, 2012

4,926

1,085

• Oct 27, 2012

• #4

PM me the screenshot. will try.

 

gangsta1

Regular Member

Oct 12, 2009

418

160

• Oct 27, 2012
• Thread Starter Thread Starter

• #5

alternatesword said:

PM me the screenshot. will try.

Click to expand...

[TABLE="class: aws_data, width: 100%"]
[TR]
[TD="class: aws"]Unknown robot (identified by empty user agent string)[/TD]
[TD]73,913+96[/TD]
[TD]73.38 GB[/TD]
[/TR]
[/TABLE]

 

T

TheViceroy

BANNED

Jul 3, 2012

201

39

• Oct 27, 2012

• #6

On one of my proxy sites I have this issue, and I've activated cloudflare to stop it. So far, it's been surprisingly effective. One day I had 20,000 bogus robot requests and almost all of those have been eliminated.

 

gangsta1

Regular Member

Oct 12, 2009

418

160

• Oct 27, 2012
• Thread Starter Thread Starter

• #7

TheViceroy said:

On one of my proxy sites I have this issue, and I've activated cloudflare to stop it. So far, it's been surprisingly effective. One day I had 20,000 bogus robot requests and almost all of those have been eliminated.

Click to expand...

What cloudflare package do you use? Did it stop all the bogus bots?

 

T

TheViceroy

BANNED

Jul 3, 2012

201

39

• Oct 27, 2012

• #8

I believe I'm just using the free package. It didn't cost me anything.

 

gangsta1

Regular Member

Oct 12, 2009

418

160

• Oct 27, 2012
• Thread Starter Thread Starter

• #9

Cloudflare requires DNS change I am not sure I want to do that.
Was thinking it would be a htaccess or robots.txt edit...

Anyone?!

 

C

cgimaster

Power Member

Jun 30, 2012

569

330

• Oct 27, 2012

• #10

Use htaccess and make a rule to block browsers with empty useragent string


Something like this "untested may need some tweeks"

Code:

RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^ScumBot [OR]
RewriteCond %{HTTP_USER_AGENT} ^AnotherIntruder [OR]
RewriteCond %{HTTP_USER_AGENT} ^\s+$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^$
RewriteRule ^.* - [F]

The F flag stands for forbidden so it will redirect those 4 different user agents to a forbidden page.

3rd useragent is for useragents that are only spaces without any identification and last one is for empty useragents, the 2 above are for custom bots if you want to block any.

Also will recommend you to check your apache access log and grab some of the ips being used by the spammer and from what network they are coming from, depending if they are exclusive dedicated ip, you can block that datacenter range to prevent this for being further scaled.

 

Last edited: Oct 27, 2012

Scripteen

Elite Member

Sep 19, 2009

1,818

1,930

• Oct 27, 2012

• #11

Awstats has a section where it tells you the top ips that consumed bandwidth. Do whois on the first 3 and see which one you want to block.

You can also do htaccess blocking or php blocking. Or better redirect them somewhere they don't expect.

 

gangsta1

Regular Member

Oct 12, 2009

418

160

• Oct 28, 2012
• Thread Starter Thread Starter

• #12

Scripteen said:

Awstats has a section where it tells you the top ips that consumed bandwidth. Do whois on the first 3 and see which one you want to block.

You can also do htaccess blocking or php blocking. Or better redirect them somewhere they don't expect.

Click to expand...

The unknown robot does not show up an ip, but the top ip's are comcast and other big cable companies?!