Will this robots file block all search engines?
- Thread starter Holla1
- Start date
Holla1
Regular Member
- Aug 26, 2014
- 230
- 4
a***ole
_________________________________________________________________________________________
Tobbe co
Junior Member
- Sep 29, 2014
- 171
- 139
Ehm, okey.. Sorry for answering your question ("Will this robots file block all search engines?")..
You always this nice?
The "User-agent: *" means this section applies to all robots (wildcard). The "Disallow: /" tells the robot that it should not visit any pages on the site.
If you just want to block certain folder or categories you could use "Disallow: /category/", that will tell the spiders to not visit your urls looking like domain.com/category/xyz. While a single slash refers to the root of your domain.
Last edited:
snarky
Junior Member
- Nov 21, 2009
- 116
- 73
Since following robots.txt directives is voluntary and not all robots will obey, use an .htaccess file as well;
insert the robots you wish to bock.
Now... can I be an a-hole as well?
Code:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (googlebot|bingbot) [NC]
RewriteRule .* - [R=403,L]insert the robots you wish to bock.
Now... can I be an a-hole as well?
tony_d
Elite Member
- Jun 22, 2013
- 2,673
- 3,294
You certainly have a unique style...
When someone answers my question, I typically thank them.