1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Is My Robots File Stuffed?

Discussion in 'Web Design' started by mohnikinsiz, Jun 2, 2012.

  1. mohnikinsiz

    mohnikinsiz Regular Member

    Joined:
    Dec 17, 2011
    Messages:
    334
    Likes Received:
    114
    I'm a bit worried that my robot.txt file is stuffed up. I'm running thesis theme on my site and this is what it looks like.

    Allow: /
    Disallow: ~
    Sitemap: http://www.domain.net/sitemap.xml
    User-Agent: MJ12bot
    User-agent: Googlebot-Mobile
    sitemap:http://cdn.attracta.com/sitemap/12x6x4x.xml.gz (it says that this xml file has no style information associated with it)


    I know that Disallow: / means to block everything but what does Disallow: ~ mean?

    Appreciate any help I can get. Thanks a lot guys!
     
  2. kralcx

    kralcx Junior Member

    Joined:
    May 25, 2010
    Messages:
    164
    Likes Received:
    49
    What do you want to disallow? For example if you don't want the bots to access the cgi-bin folder than put this Disallow: /cgi-bin/ I believe your disallow is incorrect in your robot.txt file.