Robot.txt and start with Deep link

Carlape

Newbie
Joined
Aug 17, 2017
Messages
9
Reaction score
0
Hi :)

What if I blocked a directory in a robot.txt but there is a external or internal link to 1 article to this directory? Will the crawler discover all links in this directory or stop with the first article and check robots.txt and respect the configuration?

Thanks :)
 
Back
Top