1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Hiding automation plugins from Google?

Discussion in 'Blogging' started by lucabrasi, Feb 9, 2011.

  1. lucabrasi

    lucabrasi Regular Member

    Joined:
    Mar 22, 2010
    Messages:
    337
    Likes Received:
    17
    Occupation:
    Layed off from real job but working at a survival
    Location:
    Pacific Northwest, Oregon Washington border
    Home Page:
    Is it still necessary to hide WP Robot, etc. from Google using KB Robots.txt.
    I suppose there are other robots that do the same thing but, that's the one I know about.

    Do we still need to disallow access to our plugin folder or is that no longer necessary?

    If it is necessary what lines do I need to put in to hide it?

    Thanks for your consideration.
     
  2. lancis

    lancis Elite Member

    Joined:
    Jul 31, 2010
    Messages:
    1,632
    Likes Received:
    2,384
    Occupation:
    Entrepreneur
    Location:
    Milky Way
    Home Page:
    Disallowing the plugin directory in robots.txt wouldn't help you against Google.
    If they have algorithms to dicover automation content it will certainly ignore your robots.txt.

    Use .htaccess instead and return your not found page when something requests the automation plugins.
     
  3. lucabrasi

    lucabrasi Regular Member

    Joined:
    Mar 22, 2010
    Messages:
    337
    Likes Received:
    17
    Occupation:
    Layed off from real job but working at a survival
    Location:
    Pacific Northwest, Oregon Washington border
    Home Page:
    How do you do that?
     
  4. littlewebdragon

    littlewebdragon Jr. VIP Jr. VIP Premium Member

    Joined:
    Dec 30, 2007
    Messages:
    869
    Likes Received:
    228
    Occupation:
    Occupation
    Location:
    Location
    Do not use robots.txt for that... By saying disallow wp-robot for example, you are saying to G, hey man, I have wp-robot here, please don't index it. :) So that's not good idea.
    I didn't tested it personally, but .htaccess may be good idea to do.
     
  5. lancis

    lancis Elite Member

    Joined:
    Jul 31, 2010
    Messages:
    1,632
    Likes Received:
    2,384
    Occupation:
    Entrepreneur
    Location:
    Milky Way
    Home Page:
    Use something like that:

    Code:
    RewriteEngine on
    RewriteCond %{HTTP_USER_AGENT} .*Googlebot.*$ [NC]
    RewriteRule ^/wp-plugins(.*)$ [L,R=404]
    
    It basically tells Apache to return 404 when anything identified as Googlebot tries to access /wp-plugins directory.

    I will suggest to test it first with your user agent, to make sure that the rule is correct, as I wrote it out of my head without any testing.
     
    • Thanks Thanks x 3
  6. zzds_tony

    zzds_tony Newbie

    Joined:
    Jun 27, 2012
    Messages:
    17
    Likes Received:
    2
    Thanks. I think it's useful.
     
  7. .::BlackHat::.

    .::BlackHat::. Registered Member

    Joined:
    Dec 8, 2007
    Messages:
    56
    Likes Received:
    5

    lancis is right, this is one and only sure approach
     
  8. noober

    noober Junior Member

    Joined:
    Jul 18, 2008
    Messages:
    131
    Likes Received:
    35
    Nice clean code. Thank you, I'll give it a try.
     
  9. CyberSEO

    CyberSEO Senior Member

    Joined:
    Jul 14, 2011
    Messages:
    939
    Likes Received:
    256
    Occupation:
    programmer
    Home Page:
    It is extremely important to hide the autoblogging plugins from Google because otherwise your site will be penalized. On the other hand, you can't hide the plugin from search engines using robots.txt. Furthermore, putting anything into robots.txt is a sure way to expose it to anyone, including search engines.

    Every autoblogging plugin must be able to hide itself. I would not recommend to use the plugin which is unable to hide itself from anyone.

    Unfortunately Google bots got smarter and they not always reveal themselves as "Googlebot" agents. So this won't work. I tell more. It will be VERY VERY bad if Google will caught your site using cloaking methods...

    If you want to protect the autoblogging path via .htaccess to must redirect it to 404 for ALL, bot just for "Googlebot" agent.
     
    • Thanks Thanks x 1
    Last edited: Sep 18, 2012
  10. mtlcanuck

    mtlcanuck Newbie

    Joined:
    Jul 6, 2012
    Messages:
    24
    Likes Received:
    1
    Be careful before using any script or code posted on here. I used lancis script on one of my autoblogs and the site has since been de-indexed (PR was 2, now 0 & cannot get any results in Google search when I type in mydomain.com...

    I guess Google is now targeting this kind of cheats. I did learn it the hard way...
     
  11. SnowWar

    SnowWar Power Member

    Joined:
    Mar 3, 2012
    Messages:
    595
    Likes Received:
    48
    Occupation:
    Pure student :p
    Create to Robotics.
    But which language is used.
    I think that you are used to AI.
    as a result you get a better result.