Search results

So, duplicates filtering is on (by hostnames) yet 4/5 of the harvested urls are all duplicates. What gives?

Hi, I've found over 250,000 forums to register on the last couple of days, the thing is though that alot of them don't allow guests to view forum profiles. How can I harvest only those forums that allow public viewing of profiles? Alot of forums will show the profile url even though you can't...

So after scraping for forums I have a list of over 100,000 urls. But not all of them are actual forums. So there are plenty of urls not worth anything to me, what is a good way of filtering the non forum links out of there? Doing it manually is going to cost alot of time, or is this the way to...