1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

What does your robot.txt file look like?

Discussion in 'Black Hat SEO' started by asiansamurai, Sep 18, 2014.

  1. asiansamurai

    asiansamurai Junior Member

    Joined:
    Feb 23, 2014
    Messages:
    185
    Likes Received:
    35
    Hi,

    I'm editing my robot.txt file for my PBN's and wanted to ask if you could share how yours look like.

    Also how do I block search engines from seeing my plugins?

    Do I use this:

    Disallow: /plugins/

    or do I use this:

    Disallow: /wp-content/plugins/
     
  2. RedLable

    RedLable Regular Member

    Joined:
    Feb 16, 2011
    Messages:
    244
    Likes Received:
    32
    robot-txt.png
    ..........
     
    • Thanks Thanks x 1
  3. SEO Power

    SEO Power Elite Member

    Joined:
    Jul 14, 2014
    Messages:
    2,637
    Likes Received:
    680
    Occupation:
    Self employed
    Location:
    Houston, TX
    Install Spyder Spanker (it's a plugin) on all your PBN sites if you want to block all link crawlers. There'll be no need to edit your robots.txt file after you do that.
     
  4. asiansamurai

    asiansamurai Junior Member

    Joined:
    Feb 23, 2014
    Messages:
    185
    Likes Received:
    35
    That's exactly why I want to edit my robot.txt file as using this plugin across all sites leaves a footprint.

    Better to be on the safe side after I've spent over $1000 on my PBN.