The option to filter on duplicates when loading the links database is turned off by default.
Right now it filters while harvesting, and it does filter duplicated hosts. But apparently it doesn't filter good enough.
Does this apply to hrefer as well because they seem to work fine without using that format. Should I change it to the format described in the link or leave it like I have it right now (IP : Port).
Would you recommend SOCKS proxy for hrefer? I'm thinking of trying out 20 Worldwide SOCKS4/5 just to try it out and see how fast I can collect with it and then go for the 100 Worldwide SOCKS4/5 but then I would be paying double. I'm used to having 1500+ transparant/elite/anonymous proxies for...
I have this already in my footprints, and I read topics all the time :). It will harvest profile links, but when you visit those links you will get a message telling you to register to see the profile.
The profile link will be visible in the adress bar though.
I'll keep on searching and also...
This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
By continuing to use this site, you are consenting to our use of cookies.