Hiding automation plugins from Google?

Is it still necessary to hide WP Robot, etc. from Google using KB Robots.txt.
I suppose there are other robots that do the same thing but, that's the one I know about.

Do we still need to disallow access to our plugin folder or is that no longer necessary?

If it is necessary what lines do I need to put in to hide it?

Thanks for your consideration.

Disallowing the plugin directory in robots.txt wouldn't help you against Google.
If they have algorithms to dicover automation content it will certainly ignore your robots.txt.

Use .htaccess instead and return your not found page when something requests the automation plugins.

lancis said:

Disallowing the plugin directory in robots.txt wouldn't help you against Google.
If they have algorithms to dicover automation content it will certainly ignore your robots.txt.

Use .htaccess instead and return your not found page when something requests the automation plugins.

Click to expand...

How do you do that?

Do not use robots.txt for that... By saying disallow wp-robot for example, you are saying to G, hey man, I have wp-robot here, please don't index it. So that's not good idea.
I didn't tested it personally, but .htaccess may be good idea to do.

Use something like that:


It basically tells Apache to return 404 when anything identified as Googlebot tries to access /wp-plugins directory.

I will suggest to test it first with your user agent, to make sure that the rule is correct, as I wrote it out of my head without any testing.

Thanks. I think it's useful.

lancis

Disallowing the plugin directory in robots.txt wouldn't help you against Google.
If they have algorithms to dicover automation content it will certainly ignore your robots.txt.

Use .htaccess instead and return your not found page when something requests the automation plugins.

Click to expand...

lancis is right, this is one and only sure approach

lancis said:

Use something like that:

Code:

RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} .*Googlebot.*$ [NC]
RewriteRule ^/wp-plugins(.*)$ [L,R=404]

It basically tells Apache to return 404 when anything identified as Googlebot tries to access /wp-plugins directory.

I will suggest to test it first with your user agent, to make sure that the rule is correct, as I wrote it out of my head without any testing.

Click to expand...

Nice clean code. Thank you, I'll give it a try.

lucabrasi said:

Is it still necessary to hide WP Robot, etc. from Google using KB Robots.txt.
I suppose there are other robots that do the same thing but, that's the one I know about.

Click to expand...

It is extremely important to hide the autoblogging plugins from Google because otherwise your site will be penalized. On the other hand, you can't hide the plugin from search engines using robots.txt. Furthermore, putting anything into robots.txt is a sure way to expose it to anyone, including search engines.

Every autoblogging plugin must be able to hide itself. I would not recommend to use the plugin which is unable to hide itself from anyone.

lancis said:

Use something like that:

Code:

RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} .*Googlebot.*$ [NC]
RewriteRule ^/wp-plugins(.*)$ [L,R=404]

It basically tells Apache to return 404 when anything identified as Googlebot tries to access /wp-plugins directory.

I will suggest to test it first with your user agent, to make sure that the rule is correct, as I wrote it out of my head without any testing.

Click to expand...

Unfortunately Google bots got smarter and they not always reveal themselves as "Googlebot" agents. So this won't work. I tell more. It will be VERY VERY bad if Google will caught your site using cloaking methods...

If you want to protect the autoblogging path via .htaccess to must redirect it to 404 for ALL, bot just for "Googlebot" agent.

Be careful before using any script or code posted on here. I used lancis script on one of my autoblogs and the site has since been de-indexed (PR was 2, now 0 & cannot get any results in Google search when I type in mydomain.com...

I guess Google is now targeting this kind of cheats. I did learn it the hard way...

Create to Robotics.
But which language is used.
I think that you are used to AI.
as a result you get a better result.