how crawler works

  1. Peter Manson

    How crawler crawl a web page?

    A question pings me many times If a website disallow bot for specific directory like ( Disallow: /user/* ) but I crawl a URL ( /user/someone) through submit tool so what will happen. Will this page crawl or not? I forgot to tell that I am getting the do-follow links here. Will I get the link...
Back
Top
AdBlock Detected

We get it, advertisements are annoying!

Sure, ad-blocking software does a great job at blocking ads, but it also blocks useful features and essential functions on BlackHatWorld and other forums. These functions are unrelated to ads, such as internal links and images. For the best site experience please disable your AdBlocker.

I've Disabled AdBlock