1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

xml sitemap in Robot.txt file disallow missing

Discussion in 'White Hat SEO' started by Junoseo, Nov 6, 2016.

  1. Junoseo

    Junoseo Newbie

    Joined:
    Oct 22, 2016
    Messages:
    14
    Likes Received:
    0
    Hey guys,

    an xml sitemap have been link to a robot txt file as following:
    user agent:*
    sitemap:url.

    However normally when a robot txt allow access to all robots it should be written as following
    user agent
    disallow:
    So I am wondering if the robot txt is written properly or if it could cause some issue
     
  2. patriciajohns

    patriciajohns Junior Member

    Joined:
    May 19, 2015
    Messages:
    147
    Likes Received:
    45
    I don't understand why anyone will allow or disallow sitemap in robots .txt.
    Simple delete the instance of sitemap in robots .txt and search engines will crawl it.
    Even if you disallow, search engines can crawl almost any file, but may not index.
     
  3. Junoseo

    Junoseo Newbie

    Joined:
    Oct 22, 2016
    Messages:
    14
    Likes Received:
    0
    I am wondering if the robot txt is written properly, shouldn't it be written like this
    user agent:*
    disallow:
     
  4. webspero

    webspero Newbie

    Joined:
    Sep 6, 2016
    Messages:
    33
    Likes Received:
    1
    Gender:
    Male
    Occupation:
    Development
    Location:
    Chandigarh, India
    Home Page:
    It will not harm your website all if you keep the current code in the robots.txt file for sitemap. As the current code does not mean that crawler has been restricted on sitemap.
     
  5. JKW

    JKW Newbie

    Joined:
    Nov 7, 2016
    Messages:
    39
    Likes Received:
    3
    Gender:
    Male
    To be indexed properly this code is written in a right manner