1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

question about robot.txt file

Discussion in 'White Hat SEO' started by gsmktg, Apr 24, 2010.

  1. gsmktg

    gsmktg Registered Member

    Joined:
    Mar 29, 2010
    Messages:
    52
    Likes Received:
    15
    Location:
    somewhere in Europe
    Hi, i just did a check at http://www.seocentro.com/tools/search-engines/metatag-analyzer.html of my site and at Robots it says:

    Spider robots are not allowed to spider the web page.

    what does this mean and how can i fix it? I didn;t change anything myself to robot.txt (i guess?) but it might bne a plugin of my wordpress who made it? i use all in SEO, seo tag cloud en analytics plugin.

    can somebody help me?
     
  2. lord

    lord Junior Member

    Joined:
    Jan 21, 2009
    Messages:
    176
    Likes Received:
    251
    check that your wordpress is set up to allow indexing

    verify if you have noindex in your source code

    and check your www.domain.com/robots.txt to see in noindex is present for the root folder
     
  3. veni

    veni Junior Member

    Joined:
    Nov 23, 2008
    Messages:
    141
    Likes Received:
    11
    you robot*txt file should contains this, anything extra is really up to you