Will this robots file block all search engines?

[Holla1]

User-agent: *
Disallow: /


This is what it says on google.

 

[Tobbe co]

Yes .

 

[BabyAngel]

only crawlers that obey robots.txt

 

[Holla1]

Yes .

a***ole

_________________________________________________________________________________________

 

[Tobbe co]

a***ole

Ehm, okey.. Sorry for answering your question ("Will this robots file block all search engines?")..

You always this nice?

The "User-agent: *" means this section applies to all robots (wildcard). The "Disallow: /" tells the robot that it should not visit any pages on the site.
If you just want to block certain folder or categories you could use "Disallow: /category/", that will tell the spiders to not visit your urls looking like domain.com/category/xyz. While a single slash refers to the root of your domain.

 

[rob1977]

That escalated quick lol

+1 for what tobbe says

 

[snarky]

Since following robots.txt directives is voluntary and not all robots will obey, use an .htaccess file as well;

RewriteEngine On

RewriteCond %{HTTP_USER_AGENT} (googlebot|bingbot) [NC]
RewriteRule .* - [R=403,L]

insert the robots you wish to bock.

Now... can I be an a-hole as well?

 

[navinb]

use both robot.txt and .htaccess. should block 99%

 

[tony_d]

a***ole

You certainly have a unique style...

When someone answers my question, I typically thank them.