Scrapebox -test engine Google and Bing wrong.How to fix it？

[William798]

1.Settings-Harvestr Engine Configuration-Google-Test Engine-Next Page Marker：NOT FOUND

2.Bing-Test Engine-Error：No links could be retrieved


The version is latest 2.1.0.35
Import/Update engines-Download default engines from server is applied.

Please help me. Thank you.

 

[William798]

@loopline Test engine Yahoo is ok. But Google and Bing have some problems. I downloaded default engines. It's doesn't work.

 

[loopline]

@loopline Test engine Yahoo is ok. But Google and Bing have some problems. I downloaded default engines. It's doesn't work.

this is weird, it works for me. What country is your IP from?
and/or I see you have some proxies there, can you try the test with a few different proxies?

Its possible google and bing are serving you up different code due to your ip.

also you can save the raw html of the page and go in and look for the next page marker and then put it in the engine file too. Its possible with bing deploying chat gpt and google deploying bard that they are just changing code for some users ips but not others.

 

[William798]

this is weird, it works for me. What country is your IP from?
and/or I see you have some proxies there, can you try the test with a few different proxies?

Its possible google and bing are serving you up different code due to your ip.

also you can save the raw html of the page and go in and look for the next page marker and then put it in the engine file too. Its possible with bing deploying chat gpt and google deploying bard that they are just changing code for some users ips but not others.

My IP is from China. I remote connect and run the scrapebox on a Seattle cloud server.
I upload the raw html in the attach files. Could you help to check ?

 

[William798]

this is weird, it works for me. What country is your IP from?
and/or I see you have some proxies there, can you try the test with a few different proxies?

Its possible google and bing are serving you up different code due to your ip.

also you can save the raw html of the page and go in and look for the next page marker and then put it in the engine file too. Its possible with bing deploying chat gpt and google deploying bard that they are just changing code for some users ips but not others.

I try and harvest with the detailed harvester. Google only has 1 page, the results are much less than Yahoo and Bing.

 

[loopline]

The raw hmtl is the clue. So when you go to settings >> harvester engine configuration and then test the engine, when its done it will ask if you want to save the raw html. Thats the key, because then you can look and see if there is a next page marker. ITs possible google is just serving you different code then the rest of the world for some reason. So would just need to update the marker in scrapebox and then it should work.

 

[William798]

How to update the marker in scrapebox ?
I open the raw html and can't find the next page marker. Theres only "More results" at the bottom of the html.

 

[Steptoe]

Yeah I've noticed that Google have been experimenting with infinite scrolling SERPs rather than pagination on some of my accounts. Might be an issue for scrapers in the future. You'll need to capture what requests are going to Google to display the 'more results' section. Having devtools open (F12), click on the 'more results' button and see what requests get sent - you might be able to get more information from that.

 

[loopline]

Yeah I've noticed that Google have been experimenting with infinite scrolling SERPs rather than pagination on some of my accounts. Might be an issue for scrapers in the future. You'll need to capture what requests are going to Google to display the 'more results' section. Having devtools open (F12), click on the 'more results' button and see what requests get sent - you might be able to get more information from that.

Yeah I saw the same in a small amount of mine too. Im sure its something they will eventually go to, unless bard/ai reshuffles everything, then who knows. support is checking into it though, I did talk with them.

 

[Yzetien]

In first post's screenshots, your "use proxies" were not checked.

The following screenshot might infers that your proxies quality may need improvisation.

For Bing engine, it has different http responses from the http headers you send, try adjusting it around. I forgot if ScrapeBox has the function to adjust Accept-Encoding, if it does change it from Accept-Encoding: gzip, deflate, br to Accept-Encoding:gzip, it might work.

 

[Ranoo]

Yeah I saw the same in a small amount of mine too. Im sure its something they will eventually go to, unless bard/ai reshuffles everything, then who knows. support is checking into it though, I did talk with them.

I also met this problem, but I read your message above, but I am not sure how to solve this problem, could you please ask me for details?Thank you very much

 

[loopline]

Go to settings >> harvester engine configuration



Then click google

Then replace the “Marker For Next Page” box, replace it with this line of code (means delete the existing code and paste this new code in. In this 1 box only):



<span style="display:block;margin-left:53px">|<img border="0" src="nav_next.gif"|src="nav_next_2.gif" width="100"|<br><strong>Next</strong>|filter=0">Next|<span>&gt;</span>|aria-label="More results"



Give the Display Name a new name.



Click add as new engine



Then when you go to harvest select that newly named engine.

 

[autoquick]

Go to settings >> harvester engine configuration



Then click google

Then replace the “Marker For Next Page” box, replace it with this line of code (means delete the existing code and paste this new code in. In this 1 box only):



<span style="display:block;margin-left:53px">|<img border="0" src="nav_next.gif"|src="nav_next_2.gif" width="100"|<br><strong>Next</strong>|filter=0">Next|<span>&gt;</span>|aria-label="More results"



Give the Display Name a new name.



Click add as new engine



Then when you go to harvest select that newly named engine.

Hi,

I also met the same problem with Bing and Google. I've tried this way for Google and it works now.

Please offer a way to solve the problem for Bing.

Thank you in advance.

 

[loopline]

Hi,

I also met the same problem with Bing and Google. I've tried this way for Google and it works now.

Please offer a way to solve the problem for Bing.

Thank you in advance.

well what happens for bing. Does it give you an error?

when you test the engine is there a next page marker?

 

[autoquick]

well what happens for bing. Does it give you an error?

when you test the engine is there a next page marker?

Hi Loopline,
Thank you for your message, it was showing the following error when I test Bing.

By the way, I added the new Google as what you told, only the first page can be found, but the next page still couldn't be found.


Thank you in advance.

 

[loopline]

Hi Loopline,
Thank you for your message, it was showing the following error when I test Bing.
View attachment 260671
By the way, I added the new Google as what you told, only the first page can be found, but the next page still couldn't be found.

View attachment 260673
Thank you in advance.

for GOOGLE

Go to settings >> harvester engine configuration

Then click google

Then replace the “Marker For Next Page” box, replace it with this line of code (means delete the existing code and paste this new code in. In this 1 box only):


<span style="display:block;margin-left:53px">|<img border="0" src="nav_next.gif"|src="nav_next_2.gif" width="100"|<br><strong>Next</strong>|filter=0">Next|<span>&gt;</span>|aria-label="More results"


Give the Display Name a new name.


Click add as new engine


Then when you go to harvest select that newly named engine.





BING
for bing try going to settings >> harvester engine configuration >> import >> download default egines. The default bing is working for me. Else I would need the raw html file from the bing engine to see why its not working.

 

[JBB1026]

H have run into the same problem (Next page marker: NOT FOUND), where I cannot scrape past the second page in Google, with or without proxies.

I have imported the latest search engines, updated to the latest Beta, and with the page marker set to the default:
<span style="display:block;margin-left:53px">|<img border="0" src="nav_next.gif"|src="nav_next_2.gif" width="100"|<br><strong>Next</strong>|filter=0">Next|<span>&gt;</span>
or the suggestion by @loopline
<span style="display:block;margin-left:53px">|<img border="0" src="nav_next.gif"|src="nav_next_2.gif" width="100"|<br><strong>Next</strong>|filter=0">Next|<span>&gt;</span>|aria-label="More results"
it does not work. The second one actually gives the error "No links could be retrieved", so maybe that is just dated by now.

When I export the raw html of the second page when testing, there is actually no "Next Page" marker. I attach the raw html file.

Any help would be appreciated. I followed all the advice offered above.

Thank you very much.

 

[loopline]

Try the beta harvester. Google changed stuff and the whole harvester had to be rebuilt.

ITs a check mark box next to custom harvester.

You can also check my youtube, Ill have a video up on it tomorrow.

 

[proxygo]

Try the beta harvester. Google changed stuff and the whole harvester had to be rebuilt.
ITs a check mark box next to custom harvester.
You can also check my youtube, Ill have a video up on it tomorrow.

you should have been tipped by scrapebox a long time ago

 

[rysioslaw]

For me Google and Bing working properly, but Yahoo is the best.
Very important is your IP geolocation. Just look it's only for you, generally I get about 1000 links/s, today maybe my ISP is angry at me .