1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Importance of Robots.Txt?

Discussion in 'Black Hat SEO' started by achristopher, Oct 21, 2013.

  1. achristopher

    achristopher Registered Member

    Joined:
    Mar 11, 2011
    Messages:
    98
    Likes Received:
    5
    I get the basic idea of Robots.Txt, but I don't know exactly what to block and what to allow. I run a forum and I use a generic Robots.Txt file. However, when I check my Google webmasters account it appears that its blocking all sorts of pages, some of which I don't think should be blocked. But, at the same time my pages seem to be showing up in the SERPS with the correct Meta descriptions. Is it okay to just not have a Robots.Txt? All, it seems to do is cause problems.
     
  2. dazk2002

    dazk2002 Power Member

    Joined:
    Oct 23, 2012
    Messages:
    706
    Likes Received:
    220
    Location:
    Here and There
    A typical robots.txt file will have this
    User-agent: *
    Disallow: /administrator/
    Disallow: /cache/
    Disallow: /cli/
    Disallow: /components/
    Disallow: /includes/
    Disallow: /installation/
    Disallow: /language/
    Disallow: /libraries/
    Disallow: /logs/
    Disallow: /media/
    Disallow: /modules/
    Disallow: /plugins/
    Disallow: /templates/
    Disallow: /tmp/

    Always allow your content,categories and menus. Another good point is to allow your images as Google will index them and if you've used alt tags you should get some extra traffic from the images alone.
     
    • Thanks Thanks x 1
  3. herefornow

    herefornow Jr. VIP Jr. VIP Premium Member

    Joined:
    Sep 7, 2013
    Messages:
    217
    Likes Received:
    24
    I usually do this for wordpress :

    User-agent: *
    # Sensitive Folder
    Disallow: /cgi-bin
    Disallow: /wp-login.php
    Disallow: /wp-admin
    Disallow: /wp-includes
    Disallow: /wp-content/plugins
    Disallow: /wp-content/cache
    Disallow: /wp-content/themes
    Disallow: /category/*/*
    Disallow: */trackback
    Disallow: */feed
    Disallow: */comments
    Disallow: /*?
    # sensitive files
    User-agent: Googlebot
    Disallow: /*.php$
    Disallow: /*.js$
    Disallow: /*.inc$
    Disallow: /*.css$
    Disallow: /*.gz$
    Disallow: /*.swf$
    Disallow: /*.wmv$
    Disallow: /*.cgi$
    Disallow: /*.xhtml$
    # Google Image
    User-agent: Googlebot-Image
    Disallow:
    Allow: /*
    # Google AdSense
    User-agent: Mediapartners-Google*
    Disallow:
    Allow: /*
    # sitemap
    sitemap: "adress of the sitemap"
     
  4. masterjani

    masterjani Jr. VIP Jr. VIP

    Joined:
    May 10, 2011
    Messages:
    3,183
    Likes Received:
    372
    Occupation:
    seo professional
    Home Page:
    As you are using some kind of forum software, you can find those sample robots.txt file online. Make sure to double check before editing.