Scrapebox not scraping Bing

[Pam Danny]

Here's the current engine I use for bing:

[Engine1]
Displayname=New Bing
QueryString=https://www.bing.com/search?q={KEYWORD}&sp=-1&pq={KEYWORD}&sc=1-49&qs=n&sk=&first={PAGENUM}&FORM=PORE
MustBeInLink=http
MustNotBeInLink=microsoft.com|overture.com|msn.com|bing.com|microsofttranslator.com|q=site:http
JustBeforeLink=href="
RightAfterLink="
PageStart=1
PageInc=10
NextPageMarker=">Next</a>|">Next</div></a>|class="sw_next">|title="Next page"
Translation=%3f=?|%3d==|%26=&|%3a=:|&amp;=&|%2F=/
headerData=
Referer=0
SSLVersion=0
Favicon=/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAMCAgMCAgMDAwMEAwMEBQgFBQQEBQoHBwYIDAoMDAsKCwsNDhIQDQ4RDgsLEBYQERMUFRUVDA8XGBYUGBIUFRT/2wBDAQMEBAUEBQkFBQkUDQsNFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBT/wAARCAAQABADASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwD7E1T4Z2sfhnQdVi1G306K6thJcy38xwXIBARVUk9+KwNQ8CXdvpUuqWN3aaxp8JAmlsnJaL/fRgGH5Vqt4j0fxX4X0vStYuLjS7zS1ZILqKLzYnQ44ZQQQcKBken4Utt4g0Twf4f1ay0m5n1e/wBThNvJcSQ+TDFGQQcKTktyf89f5Tr0cpry9pFRjScE+ZT95TUFp7PfWWj93XfmSP26lUx1JcrblPmejj7vLzb83+HXfy5T/9k=
GracePeriod=0
UserAgent=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0
Selected=1
DetailedSelected=0
FollowRelocation=1
AddRelative=
AddFieldValue=
ClearCookies=0
ListMode=0
ReadOnly=0

You can save that as bing.dat then import it via the harvester engine configuration. If that doesnt work then I'd look at proxies being an issue - I've seen bing start throwing out some ip bans where they just refuse to show any results, but still nowhere near as often as Google.

I just did as you said and it created a new Bing. Tried that out, and testing couldn't get any links.
I want to use other scraper software with the same proxies and see what happens

 

[Steptoe]

My scrapebox is currently scraping away nicely using that engine, so I'd say the problem is either proxy related or security/connectivity related on your local machine.

 

[Pam Danny]

My scrapebox is currently scraping away nicely using that engine, so I'd say the problem is either proxy related or security/connectivity related on your local machine.

I am using a VPS with German IP and I am using proxies scrapped with GSA Proxy Scraper. I set it to continually get fresh proxies and then scrapebox feeds from the proxies it gets.

How do you get the proxies? I am scraping 11m keywords, so private proxies may be too expensive for me.

What do you suggest?

 

[Steptoe]

I use a rotating proxy provider. There are many in the marketplace: https://www.blackhatworld.com/searc...&c[child_nodes]=1&c[nodes][0]=112&o=relevance

 

[GoogleAlchemist]

I've tried all the tips referenced in this thread...changing the useragents and downloading the latest engines from the default server etc

I'm not using proxies and am based in the usa. It's not a local ip ban either as its like this fresh out of the gate

When I try the new default engines, or any of the test mods, i get the saem test output "no links could be retrieved. do you want to save the raw html code we received to a file for analysis?"

ive attached 3 of those raw html reports

I found my previous post in this thread from nearly a year ago...and this is still an issue somehow?

 

[loopline]

I've tried all the tips referenced in this thread...changing the useragents and downloading the latest engines from the default server etc

I'm not using proxies and am based in the usa. It's not a local ip ban either as its like this fresh out of the gate

When I try the new default engines, or any of the test mods, i get the saem test output "no links could be retrieved. do you want to save the raw html code we received to a file for analysis?"

ive attached 3 of those raw html reports

I found my previous post in this thread from nearly a year ago...and this is still an issue somehow?

IT works fine for me and I was talking to someone else today and its working for them.

If you load up the page that is the raw html, do you see actual results or do you see something else?

 

[GoogleAlchemist]

I looked at the raw html file in notepad and it looked like a ton of info but when i added that to a wordpress post and previewed it...all it is is just the bing homepage with the search box

 

[loopline]

I looked at the raw html file in notepad and it looked like a ton of info but when i added that to a wordpress post and previewed it...all it is is just the bing homepage with the search box

Then its getting no results Scrapebox uses a specialized bing search engine that doesnt block like normal bing does. I mailed them about it.