1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Robots .txt

Discussion in 'Web Design' started by Cdman, Aug 15, 2011.

  1. Cdman

    Cdman Power Member

    Joined:
    Jul 7, 2011
    Messages:
    608
    Likes Received:
    36
    What's a good example of a robots.txt file for wordpress ?

    For some reason I always have problems with google I indexing sexy bookmark plugin on my site and don't want it happening again on my new site

    Do people use pre made robots.txt files or do it manually ?

    (I'm a noob with this,does WP already start with any robots.txt file ?)

    Thanks
     
  2. sensitiveheart_4

    sensitiveheart_4 Power Member

    Joined:
    Sep 21, 2007
    Messages:
    563
    Likes Received:
    128
    Location:
    New Delhi, India
    All my robot files of wordpress says

    Code:
    User-agent: *
    Disallow:
    
     
  3. blackhit

    blackhit Super Moderator Staff Member Jr. VIP Premium Member

    Joined:
    Jan 28, 2008
    Messages:
    2,402
    Likes Received:
    4,251
    Location:
    Dark Side Of The Moon
    Code:
    User-agent: *
    Disallow: /cgi-bin
    Disallow: /wp-admin
    Disallow: /wp-includes
    Disallow: /wp-content/plugins
    Disallow: /wp-content/cache
    Disallow: /wp-content/themes
    Disallow: /trackback
    Disallow: /feed
    Disallow: /comments
    Disallow: /category/*/*
    Disallow: */trackback
    Disallow: */feed
    Disallow: */comments
    Disallow: /*?*
    Disallow: /*?
    Allow: /wp-content/uploads
    
    # Internet Archiver Wayback Machine
    User-agent: ia_archiver
    Disallow: /
    
    # digg mirror
    User-agent: duggmirror
    Disallow: /
     
  4. Cdman

    Cdman Power Member

    Joined:
    Jul 7, 2011
    Messages:
    608
    Likes Received:
    36
    Might use the one above, thanks