1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Questions on Robots.txt File

Discussion in 'White Hat SEO' started by Vik Pathania, Feb 2, 2017.

  1. Vik Pathania

    Vik Pathania Regular Member

    Joined:
    Oct 13, 2014
    Messages:
    261
    Likes Received:
    49
    Gender:
    Male
    Occupation:
    Digital Marketing Specialist
    Location:
    New Delhi
    Here is a question for you guys:

    If I add following syntax:

    User-agent: *
    Disallow: /

    This means it will disallow all the bots to crawl any of my page. Right?

    But, would it also stop crawler from crawling my root page? Or crawler can still crawl my root page?


    Looking forward for great views.
     
  2. Emiya

    Emiya Newbie

    Joined:
    Jun 8, 2013
    Messages:
    44
    Likes Received:
    1
    disallow: / will effect to your whole website inclusive your root page.
     
  3. Vik Pathania

    Vik Pathania Regular Member

    Joined:
    Oct 13, 2014
    Messages:
    261
    Likes Received:
    49
    Gender:
    Male
    Occupation:
    Digital Marketing Specialist
    Location:
    New Delhi
    Are you sure? I am bit confused on this
     
  4. Emiya

    Emiya Newbie

    Joined:
    Jun 8, 2013
    Messages:
    44
    Likes Received:
    1
    Yes, because "Disallow: (without slash)" means that robots have access to your entire site.

    You can read here: http://www.robotstxt.org/robotstxt.html
     
  5. gary2

    gary2 Jr. VIP Jr. VIP

    Joined:
    Jan 20, 2013
    Messages:
    1,640
    Likes Received:
    229
    Occupation:
    Inbound Marketer. Blogger. Author.
    Location:
    Near River
    Yeah, It's right. Because "Disallow: /" means that will stop the crawler to crawling your entire site. By adding this it will surely affect your website traffic.