What should my .htaccess file look like?

M

MySeoEmail

Regular Member

Mar 15, 2015

235

24

• Apr 2, 2016

• #1

If i wanted to block crawlers on my PBNs what should my .htaccess file look like? I see so many threads saying diff things just wondering what works for you guys and what it should look like when I edit my .htaccess file?

 

kbklash

Supreme Member

Jun 2, 2012

1,202

466

• Apr 2, 2016

• #2

RewriteEngine On


RewriteCond %{HTTP_USER_AGENT} (googlebot|bingbot|Baiduspider) [NC]
RewriteRule .* - [R=403,L]

This is what most of us do.You might need to add more crawlers ,if you want.

 

Furious Man

Elite Member

Jr. VIP

Aug 4, 2015

4,568

679

• Apr 2, 2016

• #3

check out this

http://www.inmotionhosting.com/supp...top-search-engines-from-crawling-your-website

 

S

Sristy

Compromised Account

Aug 17, 2010

3,302

1,020

• Apr 2, 2016

• #4

This is a good thread to follow - http://www.blackhatworld.com/blackhat-seo/black-hat-seo/752005-list-approx-200-bots-block-pbn.html

 

N

noobfriendly

Newbie

Feb 26, 2016

28

1

• Apr 2, 2016

• #5

kbklash said:

RewriteEngine On RewriteCond %{HTTP_USER_AGENT} (googlebot|bingbot|Baiduspider) [NC] RewriteRule .* - [R=403,L] This is what most of us do.You might need to add more crawlers ,if you want.

Click to expand...

Isn't the purpose google to index your pbn network? If you block 'googlebot' how google will index it? Or i am missing something.

 

512 Designs

Junior Member

Jun 10, 2015

134

9

• Apr 2, 2016

• #6

its right i think RewriteCond %{HTTP_USER_AGENT} (googlebot|bingbot|Baiduspider) [NC] RewriteRule .* - [R=403,L]

 

B

Banned member 185991

Guest

• Apr 2, 2016

• #7

Mostly white, with some black characters on it.

Depends what program you open it in.

 

agent-x

Regular Member

Apr 4, 2012

337

139

• Apr 2, 2016

• #8

512 Designs said:

its right i think RewriteCond %{HTTP_USER_AGENT} (googlebot|bingbot|Baiduspider) [NC] RewriteRule .* - [R=403,L]

Click to expand...

Try that and let us know how it works out for you!