Need help on robots.txt

Discussion in 'White Hat SEO' started by dwpg002, Sep 2, 2010.

  1. dwpg002

    dwpg002 Senior Member

    Joined:
    Dec 29, 2008
    Messages:
    1,041
    Likes Received:
    55
    Hi

    I want to add this feature in robots.txt. Are where i need to entry


    <meta name="googlebot" content="nofollow">
    <meta name="robots" content="noarchive">
     
  2. dwpg002

    dwpg002 Senior Member

    Joined:
    Dec 29, 2008
    Messages:
    1,041
    Likes Received:
    55
    I added this but dont know where to add noarchive

    # robots.txt generated at http://mysite.com
    User-agent: twiceler
    Disallow: /
    User-agent: *
    Disallow:
    Disallow: /cgi-bin/
    Disallow: /wp-admin/
     
  3. black code

    black code Jr. VIP Jr. VIP

    Joined:
    May 19, 2009
    Messages:
    199
    Likes Received:
    120
    if you are using WP, just install a plugin called "PC robot.txt" and its will automatically do the job for you and blocks all the smap spiders and others...
     
  4. dwpg002

    dwpg002 Senior Member

    Joined:
    Dec 29, 2008
    Messages:
    1,041
    Likes Received:
    55
    thanks. I added it in header.php