xml sitemap in Robot.txt file disallow missing

J

Junoseo

Newbie

Oct 22, 2016

15

0

• Nov 6, 2016

• #1

Hey guys,

an xml sitemap have been link to a robot txt file as following:
user agent:*
sitemap:url.

However normally when a robot txt allow access to all robots it should be written as following
user agent
disallow:
So I am wondering if the robot txt is written properly or if it could cause some issue

 

patriciajohns

Junior Member

May 19, 2015

161

55

• Nov 6, 2016

• #2

I don't understand why anyone will allow or disallow sitemap in robots .txt.
Simple delete the instance of sitemap in robots .txt and search engines will crawl it.
Even if you disallow, search engines can crawl almost any file, but may not index.

 

J

Junoseo

Newbie

Oct 22, 2016

15

0

• Nov 6, 2016
• Thread Starter Thread Starter

• #3

patriciajohns said:

I don't understand why anyone will allow or disallow sitemap in robots .txt.
Simple delete the instance of sitemap in robots .txt and search engines will crawl it.
Even if you disallow, search engines can crawl almost any file, but may not index.

Click to expand...

I am wondering if the robot txt is written properly, shouldn't it be written like this
user agent:*
disallow:

 

W

webspero

Newbie

Sep 6, 2016

36

1

• Nov 7, 2016

• #4

It will not harm your website all if you keep the current code in the robots.txt file for sitemap. As the current code does not mean that crawler has been restricted on sitemap.

 

JKW

Newbie

Nov 7, 2016

39

3

• Nov 7, 2016

• #5

To be indexed properly this code is written in a right manner