1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Sitemaps and robot text help.

Discussion in 'Blogging' started by aaronjunited, Aug 4, 2012.

  1. aaronjunited

    aaronjunited Registered Member

    Joined:
    Jul 30, 2012
    Messages:
    56
    Likes Received:
    1
    Location:
    UK
    I'm currently designing my site in WordPress.

    I checked Google to see if had indexed any pages, which I don't want it too just yet until the site is finished.

    I used privacy settings to disallow it from indexing.

    Google has indexed 109 pages and they are all from my WordPress directory, content folder and plugins folder.

    I've tried robots file and sitemaps, sitemaps says robot file is blocking urls but there is nothing in it telling it to block.

    A whole mess tbh.

    I desperately need someone to help me fix this.

    Thanks.
     
  2. Rua999

    Rua999 Power Member

    Joined:
    Jun 25, 2011
    Messages:
    630
    Likes Received:
    407
    Try adding these lines to your robots.txt if they're not their already..

    User-agent: *
    Disallow: /cgi-bin/
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /wp-content/plugins/
    Disallow: /wp-content/cache/
    Disallow: /wp-content/themes/
    Disallow: /wp-login.php
    Disallow: /wp-register.php
    Disallow: /wp-app.php/service
     
  3. aaronjunited

    aaronjunited Registered Member

    Joined:
    Jul 30, 2012
    Messages:
    56
    Likes Received:
    1
    Location:
    UK
    User-agent:
    *Disallow: /cgi-bin/
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /wp-content/
    Disallow: /archives/
    Disallow: /*?*
    Disallow: /category/
    Disallow: /tag/
    Disallow: /search
    Disallow: /index.php
    Disallow: /xmlrpc.php
    Disallow: *?replytocom
    Disallow: /wp-*
    Disallow: /author
    Disallow: /feed/
    Disallow: /trackback/
    Disallow: /comments/feed/
    Disallow: /page/
    Disallow: /comments/
    Disallow: /wp-content/plugins

    User-agent: Mediapartners-Google*
    Allow:/

    User-agent: Googlebot-Image
    Allow:/

    wp-content/uploads/User-agent: Adsbot-Google
    Allow:/

    User-agent: Googlebot-Mobile
    Allow:/


    Thats my file
     
    Last edited: Aug 4, 2012