I cannot harvest

hi
I am trying to harvest all the pages of a site. I am an experienced user and I have done this in the past. but now, every time I try to scrape, it scrapes and scrapes and it finds like 6000 links. but then, it says "similar links have been removed" and after that, I am left with 1 or 2 links.

I have tried

site:http://www.domain.com


and I have tried
site:http://www.domain.com John
site:http://www.domain.com card
site:http://www.domain.com real
site:http://www.domain.com brain
site:http://www.domain.com Susan
site:http://www.domain.com (random word)
site:http://www.domain.com (random word 2)
etc.
etc.

By the time same urls are removed, I am left with like 1 or 2 links. I dont know what the hell is going on. I have done this in the past.

I am using private proxies by the way.

Hi guys, so nobody knows why scrapebox is behaving this way?

Do you mean Duplicate domain are removed ? If yes, go to the options drop down (beside, settings) and untick "Automatically Remove Duplicate Domains", it's above "Auto Recovery".

Dude sorry but I don't get your point. I think you should tell us some more about the problem. so that I can help you in this regard.thanks

futurestic06 said:

Dude sorry but I don't get your point. I think you should tell us some more about the problem. so that I can help you in this regard.thanks

Click to expand...

Alright, let me make it simple. How do I harvest all the pages that a domain has?

It's seems like your conflicting what you're looking for. Do you mean you want to download all the clickable links available on the site so you're left with a list of links or do you want to download all the individual pages of a site?

t0.sh said:

It's seems like your conflicting what you're looking for. Do you mean you want to download all the clickable links available on the site so you're left with a list of links or do you want to download all the individual pages of a site?

Click to expand...

I want to harvest all the pages that a domain has.
for example
domain.com/index.html
domain.com/post1.html
domain.com/post2.html
domain.com/about.html

I know for a fact that this domain has hundreds, if not thousands of pages but when I do it in sb, I get like 1 or 2.

Just use site:http://domain.com

Don't put a keyword after it.

And use hrefer, not scrapebox, with good quality tested public proxies, large list. Search some of my posts about hrefer.

I would advice the following:


a) make sure that the proxies are working well and not blacklisted.
b) use the site operator ( You are already using it)
c) Try with a low number of threads.
d) Call support if all these are ok and you still cannot scrape.