Google ignores Robots.txt

mindmaster

Elite Member

Sep 16, 2010

5,671

3,112

• Jun 27, 2014

• #1

I just saw 2 domains that uses robots.txt (disallow: / ) are reindexed by google.


A week ago google had deindexed this 2 domains after I implemented robots.txt. Today they are indexed again.


Is there any solution to keep the domains not indexed by google?

 

Scripteen

Elite Member

Sep 19, 2009

1,818

1,930

• Jun 27, 2014

• #2

Block google bot through htaccess

Code:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (googlebot|bingbot|Baiduspider) [NC]
RewriteRule .* - [R=403,L]

Sources

http://stackoverflow.com/questions/...spiders-for-a-special-directory-with-htaccess

https://www.google.com.eg/search?nu...s+block+googlebot&oq=htaccess+block+googlebot

 

Tanckom

Power Member

May 4, 2014

592

184

• Jun 27, 2014

• #3

Wordpress? disable robots.txt with seo plugins

 

thesam

Power Member

Aug 13, 2013

548

94

• Jun 27, 2014

• #4

Maybe domain is getting some strong backlinks, in that case google index domain in its search results and show description as its blocked by robots.txt

You can also send removal request via webmaster tools

 

Tealover

Junior Member

Aug 5, 2013

182

65

• Jun 27, 2014

• #5

You may also try to use robots meta tags.
Something like <meta name="robots" content="noindex" /> or <meta name="googlebot" content="noindex"> in template.

 

N

netgeek

Registered Member

Jan 5, 2011

53

16

• Jun 27, 2014

• #6

Strange and i wonder why it is still indexing, how ever if you are using wordpress ,then you can go to dashboard-settings-reading and here you have check where it says " Discourage search engines from indexing this site " and save changes .

Hope this helps.

 

R

roodie

Newbie

Jun 8, 2012

3

1

• Jun 27, 2014

• #7

You can use <meta name="robots" content="noindex,nofollow">. This is the best to de-index site from Google indexeing.

Robots.txt only prevent the bots from crawling the webpages, url can be indexed in Google's database.