1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Robots.txt could cause errors in indexing?

Discussion in 'White Hat SEO' started by azilko, Feb 5, 2011.

  1. azilko

    azilko Newbie

    Joined:
    Aug 24, 2010
    Messages:
    1
    Likes Received:
    0
    Hi, Can someone give me some insite on "this" robots.txt? I have seen this format on a site that ranked very well and am wondering if there is any way that this format will hurt your availability to be indexed or how you are perceived by google? Clearly there a many sites I have found w/ this, so they are indexed, and I know that it is not the best layout, but I just want feedback on this one.

    Thanks!


    <meta name="RATING" content="General">?
    <meta name="ROBOTS" content="index,follow">?
    <meta name="GOOGLEBOT" CONTENT="index, follow">?
    <meta name="REVISIT-AFTER" content="1 days">?
     
  2. Autumn

    Autumn Elite Member

    Joined:
    Nov 18, 2010
    Messages:
    2,197
    Likes Received:
    3,041
    Occupation:
    I figure out ways to make money online and then au
    Location:
    Spamville
    You can leave all of those out, they will have effect on spidering or indexing.

    robots.txt is different from the robots meta tag. Having an incorrect robots.txt will most definitely stop you from getting spidered. Just put up a very basic robots.txt that accepts all spiders and make sure it validates.

    Having a robots.txt isn't strictly necessary but it's good practice and it stops your logs getting filled up with 404s when the bots try to fetch it.
     
    • Thanks Thanks x 1
  3. housemusic

    housemusic Regular Member

    Joined:
    Jan 27, 2009
    Messages:
    389
    Likes Received:
    72
    Occupation:
    Jan 2009
    Location:
    Jan 2009
    Take your time to read http://www.robotstxt.org/
    It will answer all yours robot.txt possible questions.

    One interesting thing is that the robot.txt file is just a note you leave to the spider. In your example it doesnt mean the spider will be forced to visit everyday.

    regards