hrefer duplicate results

S

stealthstorm

Newbie

Nov 2, 2010

21

1

• Apr 9, 2011

• #1

So, duplicates filtering is on (by hostnames) yet 4/5 of the harvested urls are all duplicates. What gives?

 

L

lifco

Regular Member

Apr 5, 2010

206

284

• Apr 9, 2011

• #2

Strange coz when i always use by hostname and pr2+ not seen a dup

Are we talking like the odd one or like a lot ?

 

S

stealthstorm

Newbie

Nov 2, 2010

21

1

• Apr 9, 2011
• Thread Starter Thread Starter

• #3

For example, I have collected 570000 urls and when I filter them with scrapebox I have about 110000 left.

I'm talking about harvested forums btw.

 

Last edited: Apr 9, 2011

J

jb2008

Senior Member

Jul 15, 2010

1,158

976

• Apr 9, 2011

• #4

Did you turn hrefer off (close it) and then open it again? When you open it the duplicate filtering only starts from that open and not from the whole harvest.

 

S

stealthstorm

Newbie

Nov 2, 2010

21

1

• Apr 9, 2011
• Thread Starter Thread Starter

• #5

jb2008 said:

Did you turn hrefer off (close it) and then open it again? When you open it the duplicate filtering only starts from that open and not from the whole harvest.

Click to expand...

The option to filter on duplicates when loading the links database is turned off by default.

Right now it filters while harvesting, and it does filter duplicated hosts. But apparently it doesn't filter good enough.

 

J

jb2008

Senior Member

Jul 15, 2010

1,158

976

• Apr 10, 2011

• #6

Yes, jesus christ, even if you DO have the hostname filtering check, that's what hrefer does.

If you close hrefer, even with duplicates filtering ON, it will only filter duplicate hostnames from that particular session. Try it vs not closing hrefer at all. If you don't close hrefer at all, there will be no duplicates. If you close / reopen many times, more duplicates. But hey, don't take my word for it...