Is My Robots File Stuffed?

Discussion in 'Web Design' started by mohnikinsiz, Jun 2, 2012.

  1. mohnikinsiz

    mohnikinsiz Regular Member

    Joined:
    Dec 17, 2011
    Messages:
    334
    Likes Received:
    114
    I'm a bit worried that my robot.txt file is stuffed up. I'm running thesis theme on my site and this is what it looks like.

    Allow: /
    Disallow: ~
    Sitemap: http://www.domain.net/sitemap.xml
    User-Agent: MJ12bot
    User-agent: Googlebot-Mobile
    sitemap:http://cdn.attracta.com/sitemap/12x6x4x.xml.gz (it says that this xml file has no style information associated with it)


    I know that Disallow: / means to block everything but what does Disallow: ~ mean?

    Appreciate any help I can get. Thanks a lot guys!
     
  2. kralcx

    kralcx Junior Member

    Joined:
    May 25, 2010
    Messages:
    178
    Likes Received:
    52
    What do you want to disallow? For example if you don't want the bots to access the cgi-bin folder than put this Disallow: /cgi-bin/ I believe your disallow is incorrect in your robot.txt file.