Here is a question for you guys: If I add following syntax: User-agent: * Disallow: / This means it will disallow all the bots to crawl any of my page. Right? But, would it also stop crawler from crawling my root page? Or crawler can still crawl my root page? Looking forward for great views.