1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Where is robots.txt in wordpress?

Discussion in 'Web Design' started by loclhero, Nov 20, 2011.

  1. loclhero

    loclhero Supreme Member

    Joined:
    Jun 11, 2007
    Messages:
    1,453
    Likes Received:
    2,413
    Gender:
    Male
    Location:
    Copperhead Road
    I can't find this sucker. G00g is indexing the images on my site which is pulling too much resources and I'm getting bogus site info on my awstats (see tons of searches for a kw but it's coming from g00g images).

    I can't find the file and want to disallow my images file.
     
  2. wannabie

    wannabie Elite Member

    Joined:
    Mar 11, 2009
    Messages:
    3,807
    Likes Received:
    2,954
    Occupation:
    Seo and Marketing Suprisingly
    Location:
    Your bedroom window
    Home Page:
    It might not have one, go into your host and upload a file called robots.txt (same folder as your wordpress) then disallow what you want :)

    You can use noindex,nofollow meta on the pages if you cant get into your host?

    hit me up on skype and ill talk you through it if you want
     
    • Thanks Thanks x 1
    Last edited: Nov 20, 2011
  3. huubert

    huubert Registered Member

    Joined:
    Jul 25, 2011
    Messages:
    67
    Likes Received:
    12
    It isn't there by default AFAIK. Just create one and upload it to your public_html root.
     
    • Thanks Thanks x 1
  4. blackhit

    blackhit Super Moderator Staff Member Jr. VIP Premium Member

    Joined:
    Jan 28, 2008
    Messages:
    2,402
    Likes Received:
    4,251
    Location:
    Dark Side Of The Moon
    It isn't part of the WP install. You'll have to upload it manually to the root of the domain (assuming your blog is in the root).

    This is the one I use. You just add your images folder to it with the disallow command.

    Code:
    User-agent: *
    Disallow: /cgi-bin
    Disallow: /wp-admin
    Disallow: /wp-includes
    Disallow: /wp-content/plugins
    Disallow: /wp-content/cache
    Disallow: /wp-content/themes
    Disallow: /trackback
    Disallow: /feed
    Disallow: /comments
    Disallow: /category/*/*
    Disallow: */trackback
    Disallow: */feed
    Disallow: */comments
    Disallow: /*?*
    Disallow: /*?
    
    # Internet Archiver Wayback Machine
    User-agent: ia_archiver
    Disallow: /
    
    # digg mirror
    User-agent: duggmirror
    Disallow: /

    Here's the info for getting your images out of the index.
    Code:
    http://www.google.com/support/webmasters/bin/answer.py?answer=35308
     
    • Thanks Thanks x 3
  5. jazzc

    jazzc Moderator Staff Member Moderator Jr. VIP

    Joined:
    Jan 27, 2009
    Messages:
    2,468
    Likes Received:
    10,143
    Also, I would recommend Yoast WordPress SEO plugin.

    It has an Indexation section that lets you finetune lots of things one would have easily missed.
     
    • Thanks Thanks x 1