sadave cxx
Newbie
- May 27, 2017
- 25
- 1
I don't understand what is the sole purpose of the robot.txt on SEO strategy, can someone explain to me?
robot.txt
- Thread starter sadave cxx
- Start date
drimpalarez
Newbie
- Jun 27, 2017
- 5
- 1
Sometimes you dont want certain pages f your website to be indexed on search engines, so you use robots to allow and disallow pages/directories to crawl and index.
Initial Effort
Elite Member
- Oct 29, 2010
- 1,958
- 279
For you to understand further, please read: https://moz.com/learn/seo/robotstxt
Burglar SEO
Junior Member
- May 7, 2017
- 181
- 40
Robot.txt are used to tell search engines to not crawl this particular pages that is the only purpose
- Aug 21, 2016
- 3,585
- 831
Yes I would accept @Burglar SEO views, Hide particular links in google & maintained your site trusted status 
- Jun 25, 2015
- 4,007
- 859
Robot.txt file helps to reindex your modified page or index your new page at first while submitting it in GWT. Also, it gives the news to the search engine robots. So it's easy to crawl the new or modified site to index them.
cycooperman24
Newbie
- Jul 5, 2017
- 10
- 3
If you only start your web project, robot.txt disallow spy bots to check your not ready website and don't get them ranking your website without content. More often that's mean you will be far from top 100.