Sitemaps and robot text help.

aaronjunited

Registered Member
Joined
Jul 30, 2012
Messages
56
Reaction score
1
I'm currently designing my site in WordPress.

I checked Google to see if had indexed any pages, which I don't want it too just yet until the site is finished.

I used privacy settings to disallow it from indexing.

Google has indexed 109 pages and they are all from my WordPress directory, content folder and plugins folder.

I've tried robots file and sitemaps, sitemaps says robot file is blocking urls but there is nothing in it telling it to block.

A whole mess tbh.

I desperately need someone to help me fix this.

Thanks.
 
Try adding these lines to your robots.txt if they're not their already..

User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /wp-login.php
Disallow: /wp-register.php
Disallow: /wp-app.php/service
 
User-agent:
*Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/
Disallow: /archives/
Disallow: /*?*
Disallow: /category/
Disallow: /tag/
Disallow: /search
Disallow: /index.php
Disallow: /xmlrpc.php
Disallow: *?replytocom
Disallow: /wp-*
Disallow: /author
Disallow: /feed/
Disallow: /trackback/
Disallow: /comments/feed/
Disallow: /page/
Disallow: /comments/
Disallow: /wp-content/plugins

User-agent: Mediapartners-Google*
Allow:/

User-agent: Googlebot-Image
Allow:/

wp-content/uploads/User-agent: Adsbot-Google
Allow:/

User-agent: Googlebot-Mobile
Allow:/


Thats my file
 
Last edited:
Back
Top