How crawler crawl a web page?

Peter Manson

Newbie
Joined
Jan 3, 2018
Messages
6
Reaction score
0
A question pings me many times If a website disallow bot for specific directory like (
Disallow: /user/* ) but I crawl a URL ( /user/someone) through submit tool so what will happen. Will this page crawl or not?

I forgot to tell that I am getting the do-follow links here. Will I get the link juice or not?
 

Nut-Nights

Jr. VIP
Jr. VIP
Joined
Jun 20, 2013
Messages
10,532
Reaction score
7,550
Website
shoppy.gg
If you block google then you wont get any links, block all others, they are shit.
 
Top