Hi 
What if I blocked a directory in a robot.txt but there is a external or internal link to 1 article to this directory? Will the crawler discover all links in this directory or stop with the first article and check robots.txt and respect the configuration?
Thanks
What if I blocked a directory in a robot.txt but there is a external or internal link to 1 article to this directory? Will the crawler discover all links in this directory or stop with the first article and check robots.txt and respect the configuration?
Thanks