Get Robots.txt - STOP BAD BOTS FROM CRAWLING YOUR WEB PAGES

TheAdvertisingMan

Power Member

Nov 22, 2012

657

630

• Aug 14, 2013

• #1

Hi, just found this. It can be useful so I'm sharing.

Upload this (or add it to) robots.txt to your root folder, it will stop bad bots from crawling your web pages.

Cheers

Download http://www.sendspace.com/file/9gi8wo

P.S. It's a .txt file so I think virus scan is not necesarry

 

M

moshapollo

Newbie

Jul 18, 2013

4

0

• Aug 14, 2013

• #2

thanks! robots.txt is an important file for google. i use it for every blog i have

 

neutralhatter

Regular Member

Jun 23, 2010

437

334

• Aug 14, 2013

• #3

Dude, robots.txt won't stop spiders or bots unless they respect it.

 

Vic Sage

Elite Member

Sep 5, 2010

1,757

2,198

• Aug 14, 2013

• #4

Better do it via .htaccess.

 

seeplusplus

Power Member

Aug 18, 2008

525

179

• Aug 14, 2013

• #5

neutralhatter said:

Dude, robots.txt won't stop spiders or bots unless they respect it.

Click to expand...

Yep, they don't even look to see if you have one, or maybe some do, to see where you DON'T want them to go, such as an admin directory or something.

 

innozemec

Elite Member

Aug 19, 2011

5,404

1,815

• Aug 14, 2013

• #6

Nobody cares about robots.txt, even Google..

 

gullsinn

Elite Member

Dec 24, 2009

2,556

2,311

• Aug 14, 2013

• #7

innozemec said:

Nobody cares about robots.txt, even Google..

Click to expand...

No mate, that's not true!

 

innozemec

Elite Member

Aug 19, 2011

5,404

1,815

• Aug 14, 2013

• #8

gullsinn said:

No mate, that's not true!

Click to expand...

yep, mate it is true

robots.txt tells G not to index the pages, but doesn't stop him from crawling them and OP is sharing the script as a "stop crawling" measure..

 

natorob

Junior Member

Jul 7, 2011

189

63

• Aug 14, 2013

• #9

I saw a WP plugin that actually does this better than a robots.txt file. Haven't bought it yet, but it's on my list. I believe the name of the plugin is SpyderSpanker (or similar). This allows you to block the BS crawlers like Yandex etc from crawling your site... May be the solution you are looking for. But cannot vouch for it since I haven't bought/used it yet....

 

TZ2011

Senior Member

Jun 26, 2011

1,095

1,131

• Aug 14, 2013

• #10

you don't need to pay for such a basic stuff. just doing a google search for .htaccess protection, blocking bots and even hack protection (5G firewall list) will do better job than any plugin.

 

TheAdvertisingMan

Power Member

Nov 22, 2012

657

630

• Aug 14, 2013
• Thread Starter Thread Starter

• #11

Of course it can't stop all bots. But it will stop all the bots you can find in this txt file. I guess it can help a bit at least Never mind

 

WizGizmo

Mod-On-Hiatus

Mod-On-Hiatus

Mar 28, 2008

4,476

57,707

• Aug 14, 2013

• #12

Since this is not an Internet Marketing product,
I have moved this thread to the Web Design section.

"Wiz"

 

ShabbySquire

Power Member

Jr. VIP

Nov 30, 2011

720

174

• Aug 14, 2013

• #13

I use password protect directories in cpanel. Unless SE crawlers have the capacity to brute force a password, then nothing is shown.

 

sm754

Registered Member

Mar 21, 2012

93

38

• Aug 15, 2013

• #14

If you want to be a real hindrance to bots, there's software out there that will simulate a tree of pages with hundreds of fake e-mail addresses; hitting one of those things can really mess up a spammer database.