anyone there Scrapebox doesn't harvest even a single URL?

Hi, I seem to be having the same issue as the OP and I do not think it's the proxy, as I pay for scraperapi.

Outside of Google (which I know is broken), I can scrape yahoo and bing with the standard harvester without proxies enabled. If I run the same scenario with the scraperapi proxy, the harvesting fails.




That led me to believe it was scraperapi, however, when I test the proxy, it returns a status of "passed" and "completed", which confirms the proxy is work. Furthermore, when I look at the Scrapeapi dashboard, I can confirm the use of one api token.


In regard to using the beta scraper, it's a similar story except bing will not work, with or without proxies, yahoo works. When proxies are enabled both engines produce an error.



I'm really stumped by this; any guidance would greatly be appreciated! Thank you!

I'll also add that scrapebox other tools (e.g. alive check) seems to function through my proxies.

Google scraper is broken in many scrapers die to google js update. Nowadays it's cheaper to buy a. Link list than scrape. What are you scraping for gsa or xrummer?

WhimpyKid said:

Google scraper is broken in many scrapers die to google js update. Nowadays it's cheaper to buy a. Link list than scrape. What are you scraping for gsa or xrummer?

Click to expand...

I'm not link building, I'm just looking to scrape serps and dedupe through scrapebox, then filter to the sites I need and get their contact info.

So in general your issue is likely blocked ips. So its passes the test because the proxies are valid, but no results are scraped because they are blocked by the search engines.

Same reason why it works on alive check, because the proxies work, they are just blocked by the engines for the type of query your doing. scraper api is likley using a huge pool of ips and thus eveyrone else is getting them blocked.

But on the standard scraper you can go to help >> show error log. what are the errors?

Also you can try duplicating that scraper api proxy like 10 times, because in some dialog boxes in scrapebox it sees each line as a unique proxy. so that will help with the proxy retries and turn the proxy retries to the max

loopline said:

So in general your issue is likely blocked ips. So its passes the test because the proxies are valid, but no results are scraped because they are blocked by the search engines.

Same reason why it works on alive check, because the proxies work, they are just blocked by the engines for the type of query your doing. scraper api is likley using a huge pool of ips and thus eveyrone else is getting them blocked.

But on the standard scraper you can go to help >> show error log. what are the errors?

Also you can try duplicating that scraper api proxy like 10 times, because in some dialog boxes in scrapebox it sees each line as a unique proxy. so that will help with the proxy retries and turn the proxy retries to the max

Click to expand...

Hi Loopline, I was hoping the master would respond! Thank you and thank you for the context.

This is what's in the error log.
1/22/2025 4:59:34 PM: HTTP: -1 Server Certificate Invalid or not present, URL: https://www.bing.com/search?q=new+york+city+pizza&sc=12-3&sp=-1&sk=&first=1&FORM=PERE&cnt=20&setlang=en
1/22/2025 4:59:40 PM: HTTP: -1 Server Certificate Invalid or not present, URL: https://search.yahoo.com/search?p=new+york+city+pizza&pz=20&ei=UTF-8&fr=yfp-t-502-sb&b=1&pz=20&xargs=0

For additional context, I tried the same bing url above in the scraperapi api playground and got status 200 along with the search results page html.

dsid2000 said:

Hi Loopline, I was hoping the master would respond! Thank you and thank you for the context.

This is what's in the error log.
1/22/2025 4:59:34 PM: HTTP: -1 Server Certificate Invalid or not present, URL: https://www.bing.com/search?q=new+y...sp=-1&sk=&first=1&FORM=PERE&cnt=20&setlang=en
1/22/2025 4:59:40 PM: HTTP: -1 Server Certificate Invalid or not present, URL: https://search.yahoo.com/search?p=n...20&ei=UTF-8&fr=yfp-t-502-sb&b=1&pz=20&xargs=0

For additional context, I tried the same bing url above in the scraperapi api playground and got status 200 along with the search results page html.
View attachment 414690

Click to expand...

I mean I could keep guessing, but it would be just guesses.

So lets just go back to basics. What do we know?

if we turn off proxies it works.

If we turn on proxies it does not work.

So somewhere, some way its connected to the proxy.

So is it security software blocking the proxy ip?

Is it the proxy altering the data so that scrapebox can not see the urls you want to scrape?

Is there some issue with the proxy funcitonally, like your log shows a -1 server certificate error. So is the proxy somehow messing with the certificate causing the issue?

I could keep guessing but its definitely related in some way to the proxy.

So you could grab a debugger like http debugger - what I use - and they have a free trial. Then run scrapebox and you can see the exact response that scrapebox is seeing and thus you may be able to determine the issue.




Now determining the issue and resolving it are 2 different things. If its that security software is blocking the proxies or something, fine then just add an exception in all security software for the entire scrapebox folder - which you could and probably ideally would do anyway.

But if you find that the certificate or data is altered by the proxy or something like that, then there isn't anything really in scrapebox you can do to offset this. At that point its a case of just getting a different proxy provider or providing that data to scraper api and see if they can do anything.