Scrapebox Harvester only scraping the first 200 results

[kleinerwauwau]

Hey everyone,

I'm trying to scrape every indexed URL of a specific website. For that purpose, I used "site:website.tld" as harvester keyword. Unfortunately though, it seems, the results are always capped at 200. Said domain though has about 10.000 URLs indexed and I need every single one of them. What could be the issue here? I didn't find any results cap in the Scrapebox settings or anything like that.

I worked a lot with this great tool a few years back and I feel rusty af with it now, so please excuse me if the solution is that obvious.

Cheers!

 

[PHPInjected]

Hey everyone,

I'm trying to scrape every indexed URL of a specific website. For that purpose, I used "site:website.tld" as harvester keyword. Unfortunately though, it seems, the results are always capped at 200. Said domain though has about 10.000 URLs indexed and I need every single one of them. What could be the issue here? I didn't find any results cap in the Scrapebox settings or anything like that.

I worked a lot with this great tool a few years back and I feel rusty af with it now, so please excuse me if the solution is that obvious.

Cheers!

- Are you using proxies or just your IP?
- What version of scrapebox are you using? Update to the latest one, if you're able to.

Try going into the settings, click "harvester engine configuration" and then download default engines.

Outside of that, perhaps @loopline may be able to help you out.

 

[kleinerwauwau]

- Are you using proxies or just your IP?

Thanks for the quick reply; I'm using 10 proxies - back in the days that was enough for my purposes. I could buy more if needed though.

- What version of scrapebox are you using? Update to the latest one, if you're able to.

I'm using the current stable build V2.1.0.38

Try going into the settings, click "harvester engine configuration" and then download default engines.

Outside of that, perhaps @loopline may be able to help you out.

Already did that as well. And yes, summoning the grand master might help alot!

EDIT: Now after another retry, I dont even get 200 results - just nothing lol

 

[PHPInjected]

Thanks for the quick reply; I'm using 10 proxies - back in the days that was enough for my purposes. I could buy more if needed though.

I'm using the current stable build V2.1.0.38

Already did that as well. And yes, summoning the grand master might help alot!

EDIT: Now after another retry, I dont even get 200 results - just nothing lol

Have you tried scraping without the proxies? Try with your normal IP, it may be the proxies themselves.

 

[kleinerwauwau]

Have you tried scraping without the proxies? Try with your normal IP, it may be the proxies themselves.

Completely without proxies I now get 200 again on google and 830 on bing (for reference). So to me with my little knowledge looks like the scrapebox crawler for some reason cannot get (click) past 200 results? Not sure how to circumvent this.

EDIT: 830 on Bing are propably only the actual URLs of the site. On Google there are about 10k indexed because of a recent malware infection and I wanna crawl all of those.

 

[Steptoe]

Google will only ever show a max of around ~350 results for any query, no matter how many results they say there are. Try it yourself manually, I doubt you'll get more than 350-400 before it stops.

To try and get around this, you can add additional keywords to your query, stuff like stopwords (a, this, that, and etc), single letters, negative keywords (-keyword) etc. This can sometimes eke out some more results. But there's no guaranteed way to get all results for a Google query.

 

[kleinerwauwau]

Google will only ever show a max of around ~350 results for any query, no matter how many results they say there are. Try it yourself manually, I doubt you'll get more than 350-400 before it stops.

To try and get around this, you can add additional keywords to your query, stuff like stopwords (a, this, that, and etc), single letters, negative keywords (-keyword) etc. This can sometimes eke out some more results. But there's no guaranteed way to get all results for a Google query.

Hey Steptoe, thanks for clarifying this for me. Sadly, I already feared this would be the case.. Not sure how to workaround it now, but lets see.
Have a good one!

 

[Steptoe]

Hey Steptoe, thanks for clarifying this for me. Sadly, I already feared this would be the case.. Not sure how to workaround it now, but lets see.
Have a good one!

Cramming a load of general keywords along with the types I suggested above usually gets you more urls than a plain site: search, but yeah depending on the domain YMMV unfortunately.

 

[kleinerwauwau]

Cramming a load of general keywords along with the types I suggested above usually gets you more urls than a plain site: search, but yeah depending on the domain YMMV unfortunately.

Thanks, I will try that tomorrow!