question about robot.txt file

Discussion in 'White Hat SEO' started by gsmktg, Apr 24, 2010.

  1. gsmktg

    gsmktg Registered Member

    Joined:
    Mar 29, 2010
    Messages:
    52
    Likes Received:
    15
    Location:
    somewhere in Europe
    Hi, i just did a check at http://www.seocentro.com/tools/search-engines/metatag-analyzer.html of my site and at Robots it says:

    Spider robots are not allowed to spider the web page.

    what does this mean and how can i fix it? I didn;t change anything myself to robot.txt (i guess?) but it might bne a plugin of my wordpress who made it? i use all in SEO, seo tag cloud en analytics plugin.

    can somebody help me?
     
  2. lord

    lord Junior Member

    Joined:
    Jan 21, 2009
    Messages:
    176
    Likes Received:
    251
    check that your wordpress is set up to allow indexing

    verify if you have noindex in your source code

    and check your www.domain.com/robots.txt to see in noindex is present for the root folder
     
  3. veni

    veni Junior Member

    Joined:
    Nov 23, 2008
    Messages:
    141
    Likes Received:
    11
    you robot*txt file should contains this, anything extra is really up to you