1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Need help on robots.txt

Discussion in 'White Hat SEO' started by dwpg002, Sep 2, 2010.

  1. dwpg002

    dwpg002 Senior Member

    Joined:
    Dec 29, 2008
    Messages:
    913
    Likes Received:
    47
    Hi

    I want to add this feature in robots.txt. Are where i need to entry


    <meta name="googlebot" content="nofollow">
    <meta name="robots" content="noarchive">
     
  2. dwpg002

    dwpg002 Senior Member

    Joined:
    Dec 29, 2008
    Messages:
    913
    Likes Received:
    47
    I added this but dont know where to add noarchive

    # robots.txt generated at http://mysite.com
    User-agent: twiceler
    Disallow: /
    User-agent: *
    Disallow:
    Disallow: /cgi-bin/
    Disallow: /wp-admin/
     
  3. black code

    black code Junior Member

    Joined:
    May 19, 2009
    Messages:
    184
    Likes Received:
    101
    if you are using WP, just install a plugin called "PC robot.txt" and its will automatically do the job for you and blocks all the smap spiders and others...
     
  4. dwpg002

    dwpg002 Senior Member

    Joined:
    Dec 29, 2008
    Messages:
    913
    Likes Received:
    47
    thanks. I added it in header.php