How crawler crawl a web page?

Peter Manson

Newbie
Joined
Jan 3, 2018
Messages
6
Reaction score
0
A question pings me many times If a website disallow bot for specific directory like (
Disallow: /user/* ) but I crawl a URL ( /user/someone) through submit tool so what will happen. Will this page crawl or not?

I forgot to tell that I am getting the do-follow links here. Will I get the link juice or not?
 
If you block google then you wont get any links, block all others, they are shit.
 
Back
Top