Robots.txt Allow every thing but three pages blogger?

googlebis · Oct 10, 2017

I want to allow every thing but 3 pages in robots.txt is it correct to do it like this :

User-agent: *
Allow:
Disallow: /p/terms.html
Disallow: /p/privacy.html
Disallow: /p/contact.html
Sitemap: http://www.example.com/atom.xml?redirect=false&start-index=1&max-results=500

i google it and that's what i found but when i test for example /p/terms.html , and run test it says that it's allowed !

CheapAndBest · Oct 10, 2017

Just male your robots.txt like the following:

User-agent: *
Disallow:

This will make every page to index.

Note: Don't forget to edit in WMT robots.txt also.

googlebis · Oct 10, 2017

CheapAndBest said:
Just male your robots.txt like the following:

User-agent: *
Disallow:

This will make every page to index.

Note: Don't forget to edit in WMT robots.txt also.

yes but i want to disallow the privacy for example so how i can do that ?

CheapAndBest · Oct 10, 2017

googlebis said:
yes but i want to disallow the privacy for example so how i can do that ?

To prevent Google web crawlers from indexing a page:
On the <head> section of your page:
<meta name="googlebot" content="noindex">

Simple

googlebis · Oct 10, 2017

CheapAndBest said:
To prevent Google web crawlers from indexing a page:
On the <head> section of your page:
<meta name="googlebot" content="noindex">

Simple

okay thatnks, so my robots.txt will be

User-agent: *
Disallow:

and add the sitemap :

User-agent: *
Disallow:
Sitemap: http://www.example.com/atom.xml?redirect=false&start-index=1&max-results=500

CheapAndBest · Oct 10, 2017

googlebis said:
okay thatnks, so my robots.txt will be

User-agent: *
Disallow:

and add the sitemap :

User-agent: *
Disallow:
Sitemap: http://www.example.com/atom.xml?redirect=false&start-index=1&max-results=500

You can use automatic xml generator on your wordpress by using this plug in: https://wordpress.org/plugins/google-sitemap-generator/

so it always update your new change..

but, don't forget to add sitemap to GWT.

okay

googlebis · Oct 10, 2017

CheapAndBest said:
You can use automatic xml generator on your wordpress by using this plug in: https://wordpress.org/plugins/google-sitemap-generator/

so it always update your new change..

but, don't forget to add sitemap to GWT.

okay

Okay, sorry for disturbing you but i have another problem with sitemap see the image

my sitemap is pending for almost 3 weeks now and for example even when i check the search traffic > internal links i see this :
No data available. (Why not?)
how do i deal with this ?
I'M USING BLOGGER

CheapAndBest · Oct 10, 2017

googlebis said:
Okay, sorry for disturbing you but i have another problem with sitemap see the image
View attachment 96166
my sitemap is pending for almost 3 weeks now and for example even when i check the search traffic > internal links i see this :
No data available. (Why not?)
how do i deal with this ?
I'M USING BLOGGER

okay, wait 1 more week and test what happen.
And why your site map has long url type?

why not it is as: atom.xml

googlebis · Oct 10, 2017

CheapAndBest said:
okay, wait 1 more week and test what happen.
And why your site map has long url type?

why not it is as: atom.xml

that's what this site sais : http://ctrlq.org/blogger/
you put your domain and this site will give you the site map ! is it wrong ?

CheapAndBest · Oct 10, 2017

I have no idea on this

googlebis · Oct 10, 2017

CheapAndBest said:
I have no idea on this

Okay i'm gonna keep seaching about this, i think the problem is with "fetch as google" thing. thanks for helping bro

CheapAndBest · Oct 10, 2017

googlebis said:
Okay i'm gonna keep seaching about this, i think the problem is with "fetch as google" thing. thanks for helping bro

you welcome

Robots.txt Allow every thing but three pages blogger?

googlebis

Power Member

CheapAndBest

Junior Member

googlebis

Power Member

CheapAndBest

Junior Member

googlebis

Power Member

CheapAndBest

Junior Member

googlebis

Power Member

CheapAndBest

Junior Member

googlebis

Power Member

CheapAndBest

Junior Member

googlebis

Power Member

CheapAndBest

Junior Member

Main Menu

Marketplace

Making Money

BlackHat World