Scrapebox not scraping Bing

[lazlopanaflex]

Everything was going great but recently Scrapebox is not scraping data from Bing. Yahoo and other search engines work fine but Bing just shows 0 for all keywords

I have tried multiple proxies and even without proxy, still all 0 results

Any ideas? Please help

 

[Jimoz]

Try with only Bing, unselect other engines. and put few keywords. try without proxies and with proxy. See if it works.

 

[lazlopanaflex]

@Jimoz tried that already, still 0

https://www.dropbox.com/s/n3g4zofl7if8yqy/Screen Shot 2021-01-06 at 9.43.50 PM.png?dl=0

 

[Sweetfunny]

Please go to Settings >> Harvester Engines Configuration >> Select Bing, then click "Test Engine" Do you get search results returned here? Then click the Next button do you also get results for the next page.

If you do then the reason for your issue is your proxies are not working for Bing, because the engine test feature is identical to the harvester except it does not use proxies. You can then add a proxy on this Window where it says "Proxy to test engine" and repeat the test to see if it's the proxy not working or something else.

If not then please go to Settings >> Harvester Engines Configuration >> Import >> Download Default Engines From Server.

This will update the engines file to ensure you have the latest version and all are restored to default.

The latest Bing engine definition is working. You are also free to update or change any engines if they are not working how you like. Also if you go to Help >> Show error log >> Harvester.log you can see what errors are being returned.

 

[lazlopanaflex]

Please go to Settings >> Harvester Engines Configuration >> Select Bing, then click "Test Engine" Do you get search results returned here? Then click the Next button do you also get results for the next page.

I did this and see this
https://www.dropbox.com/s/nv3dvhrukjqs777/Screen Recording 2021-01-07 at 11.33.32 AM.mov?dl=0

If not then please go to Settings >> Harvester Engines Configuration >> Import >> Download Default Engines From Server.

I imported for Bing. Still all 0.

So I tried to troubleshoot.

Now it is failing randomly with 0 results

See this: https://www.dropbox.com/s/ap9ln1mfy3awosa/Screen Shot 2021-01-07 at 11.45.30 AM.png?dl=0

I tested this with multiple use cases ....

Another example:
https://www.dropbox.com/s/pw3m2jrg45w10na/Screen Shot 2021-01-07 at 12.00.36 PM.png?dl=0
The error log is empty

Any ideas?

 

[Deleted member 657006]

Why not scrape public proxys from various places and use them to scrape.

Or get cheap shared proxies to scrape.

Eventually your need to buy proxys to run any bot anyway as times are changing fast

This will be a problem for everyone as so much changing this year with solid updates.

I don't think anyone will survive without buying proxys.

Just a marketing nightmare adding to all marketing problems

It getting really expensive to be a marketer know

 

[lazlopanaflex]

Why not scrape public proxys from various places and use them to scrape.

Or get cheap shared proxies to scrape.

Eventually your need to buy proxys to run any bot anyway as times are changing fast

This will be a problem for everyone as so much changing this year with solid updates.

I don't think anyone will survive without buying proxys.

Just a marketing nightmare adding to all marketing problems

It getting really expensive to be a marketer know

Do you sell proxies? how did you assume I do not use private proxies?
This is not a proxy issue. And I use proxies from instant proxies, never had issues.

 

[loopline]

If your using proxies in your screenshot where it shows the 50 sometimes and not, then the times where its not is a proxies that failed. I just tried it and I get a solid 50 with no proxies.

Try unchecking the use proxies box, then run your word test again on the 50 results. just basic, your words, and bing, set to 50 results, no proxies at all, does it give you 50 results for every query?

Where are you proxies from? Sometimes bing redirects to other countries bings servers and the formatting is different and thus it does not work.

Here is an easy test
https://www.maxmind.com/en/geoip-demo
you can paste your proxies in there, what countries are they from?

basically its either the proxies are from countries where the formatting is different and thats causing the error the proxies are blocked by bing.

Because I assume it works perfectly when you uncheck the use proxies box?

 

[GTRZ]

I got the same issue last week but after few days it came back to normal. Dunno how but now it works fine

 

[lazlopanaflex]

Try unchecking the use proxies box, then run your word test again on the 50 results. just basic, your words, and bing, set to 50 results, no proxies at all, does it give you 50 results for every query?

here are the results without proxies
https://www.dropbox.com/s/ux7ujbh9a278zd7/Screen Shot 2021-01-08 at 12.27.33 AM.png?dl=0

you can paste your proxies in there, what countries are they from?

My IPs are from instantproxies and I specifically ask for California. I just tested them here and it shows California cities for all of them.

I got the same issue last week but after few days it came back to normal. Dunno how but now it works fine

Really? Exact same issue?

 

[loopline]

here are the results without proxies
https://www.dropbox.com/s/ux7ujbh9a278zd7/Screen Shot 2021-01-08 at 12.27.33 AM.png?dl=0


My IPs are from instantproxies and I specifically ask for California. I just tested them here and it shows California cities for all of them.



Really? Exact same issue?

Ok, the non proxies one is a bit odd. If you run it and then take note, even export or screenshot the results. Then wait a few mins and do it again, do it 3 or 4 times. Do you always get results on the same keywords and no results on the other ones?

 

[lazlopanaflex]

Ok, the non proxies one is a bit odd. If you run it and then take note, even export or screenshot the results. Then wait a few mins and do it again, do it 3 or 4 times. Do you always get results on the same keywords and no results on the other ones?

No, its different each time. Check this out - same keywords

https://www.dropbox.com/s/97undu8pxuuiac3/Screen Shot 2021-01-09 at 1.14.33 AM.png?dl=0

 

[Sweetfunny]

No, its different each time. Check this out - same keywords

https://www.dropbox.com/s/97undu8pxuuiac3/Screen Shot 2021-01-09 at 1.14.33 AM.png?dl=0

It seems like Bing is randomly serving you pages with different HTML, so for some queries there's no results because ScrapeBox cant locate any of the required HTML in the page it uses to identify search result URL's. You would have to use a tool like httpdebugger.com which has a free trial so you can log the pages in the background and examine the HTML immediately before and after the search results link which can then be used to modify your definition for that engine.

Or you could just try changing the useragent for that engine and experiment with other useragents, many sites will change the page HTML based on the useragent so with a different agent you could get a more consistent result.

 

[lazlopanaflex]

It seems like Bing is randomly serving you pages with different HTML, so for some queries there's no results because ScrapeBox cant locate any of the required HTML in the page it uses to identify search result URL's. You would have to use a tool like httpdebugger.com which has a free trial so you can log the pages in the background and examine the HTML immediately before and after the search results link which can then be used to modify your definition for that engine.

Or you could just try changing the useragent for that engine and experiment with other useragents, many sites will change the page HTML based on the useragent so with a different agent you could get a more consistent result.

I never had this problem before, Bing used to scrape perfectly.

If user agent is the problem, I am sure others are facing this problem too.

 

[topseoagency2019]

The same thing happened to me. I tried with private and public proxies, as well as direct and through a VPN.

At this time (April 8,2021), with the engine file from the server, there are only 3 engines that work:
Google
IXQuick
ExaLead

Hopefully, the developer will come out with something to make it work... At this time, it is not useful too much for harvesting. Instead of killing private proxies with Google search, there may be other better things to do.

 

[loopline]

You can simply go to settings >> harvester engine configureation >> import >> download default engines

and get all the latest engine files from the server

 

[M4DM4X]

You can simply go to settings >> harvester engine configureation >> import >> download default engines

and get all the latest engine files from the server

Facing the same issue as OP. Did what you & Sweetbunny suggest but still pbl with bing and yahoo, impossible to scrape

 

[loopline]

Facing the same issue as OP. Did what you & Sweetbunny suggest but still pbl with bing and yahoo, impossible to scrape

Try going to settings >> harvester engine configureation

click on bing

Then go to the additional headers settings tab

then delete the user agent that is there, and put in this one

Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:87.0) Gecko/20100101 Firefox/87.0

Then click update engine

Then test the engine, does it work?

~~

You can try the same for yahoo.

 

[M4DM4X]

Try going to settings >> harvester engine configureation

click on bing

Then go to the additional headers settings tab

then delete the user agent that is there, and put in this one

Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:87.0) Gecko/20100101 Firefox/87.0

Then click update engine

Then test the engine, does it work?

~~

You can try the same for yahoo.

Thanks for your help Loopline.
Unfortunately it doesnt work.
Bing :
https://i.gyazo.com/0cd44099f6045c20ffe47ee967abe75f.pngYahoo:
https://i.gyazo.com/717b9047578953cebeb9940b7d49f297.png

 

[steve123_]

having this issue too with bing

edit: thanks loopline your user agent fixed it for me