DOMAIN HUNTER GATHERER

I am sure @jimbobo2779 can respond to your question(s).

Im hoping he picks this up !

velour said:

Is there a way to scrape subdomains in DHG

when i enter the subdomain in the scrape from website field it seems to only scrape from the root url.


anyone ?

dhg are u there ?

Click to expand...

Yeah, it is really easy.

If you want to stick to a subdomain and not crawl away from that you can simply enter in that subdomain and then also enter it into the "Crawled URLs: Must Contain" input and start the crawl.

Be sure when you start the crawling that you hit "No" button when prompted if you want to crawl off the starting site. If you allow it to crawl off site (clicking Yes when prompted if you want this) then you are instructing the tool to just crawl the internet indiscriminately and it will crawl all pages on that site as well as other sites that website links to.

So as an example, if you want to crawl just the English part of Wikipedia:
Crawl: http://en.wikipedia.org
Crawled URLs: Must Contain: en.wikipedia.org
Hit "No" to crawling off-site

got you !
will give it a shot soon.
Thanks for getting back to me on this