Robots .txt

Discussion in 'Web Design' started by Cdman, Aug 15, 2011.

  1. Cdman

    Cdman Power Member

    Joined:
    Jul 7, 2011
    Messages:
    608
    Likes Received:
    36
    What's a good example of a robots.txt file for wordpress ?

    For some reason I always have problems with google I indexing sexy bookmark plugin on my site and don't want it happening again on my new site

    Do people use pre made robots.txt files or do it manually ?

    (I'm a noob with this,does WP already start with any robots.txt file ?)

    Thanks
     
  2. sensitiveheart_4

    sensitiveheart_4 Power Member

    Joined:
    Sep 21, 2007
    Messages:
    567
    Likes Received:
    129
    Location:
    New Delhi, India
    All my robot files of wordpress says

    Code:
    User-agent: *
    Disallow:
    
     
  3. blackhit

    blackhit Super Moderator Staff Member Jr. VIP Premium Member

    Joined:
    Jan 28, 2008
    Messages:
    2,411
    Likes Received:
    4,287
    Location:
    Dark Side Of The Moon
    Code:
    User-agent: *
    Disallow: /cgi-bin
    Disallow: /wp-admin
    Disallow: /wp-includes
    Disallow: /wp-content/plugins
    Disallow: /wp-content/cache
    Disallow: /wp-content/themes
    Disallow: /trackback
    Disallow: /feed
    Disallow: /comments
    Disallow: /category/*/*
    Disallow: */trackback
    Disallow: */feed
    Disallow: */comments
    Disallow: /*?*
    Disallow: /*?
    Allow: /wp-content/uploads
    
    # Internet Archiver Wayback Machine
    User-agent: ia_archiver
    Disallow: /
    
    # digg mirror
    User-agent: duggmirror
    Disallow: /
     
  4. Cdman

    Cdman Power Member

    Joined:
    Jul 7, 2011
    Messages:
    608
    Likes Received:
    36
    Might use the one above, thanks