how crawler works

  1. Peter Manson

    How crawler crawl a web page?

    A question pings me many times If a website disallow bot for specific directory like ( Disallow: /user/* ) but I crawl a URL ( /user/someone) through submit tool so what will happen. Will this page crawl or not? I forgot to tell that I am getting the do-follow links here. Will I get the link...
Back
Top