Recommended scrapebox tools

[Djphant0m]

Hey @loopline I’m new to scrapebox and my contact form submissions have been getting a really low success rate at first I thought maybe it was my settings(not that I know enough about scrapebox to say if it isn’t) but I just started clicking over to a lot of the failed links and realized that a vast majority have reCaptcha or other captchas.

I know there’s several captcha solving services but I’m personally not super techy so I was wonder if there’s one that’s highly recommend over the others?

 

[loopline]

Hey @loopline I’m new to scrapebox and my contact form submissions have been getting a really low success rate at first I thought maybe it was my settings(not that I know enough about scrapebox to say if it isn’t) but I just started clicking over to a lot of the failed links and realized that a vast majority have reCaptcha or other captchas.

I know there’s several captcha solving services but I’m personally not super techy so I was wonder if there’s one that’s highly recommend over the others?

I use GSA captcha breaker and then set it up to emulate a captcha service scrapebox supports.

Scrapebox does Not support recaptcha, it currently only supports image and text capchas.

Also this may help
http://scrapeboxfaq.com/contact-form-poster-says-failed

 

[710fla]

Use GSA Captcha Breaker or Xevil to emulate a captcha service that is supported by Scrapebox. I believe you can also use 2captcha.com

 

[X-Fact]

+1 for GSA Captcha Breaker

 

[Djphant0m]

Hey @loopline the biggest issues I seem to be having with scrapebox since I’ve started and never even able to fix is duplicate urls. No matter what keywords i use when I go to harvest urls for contact form submission I’ll have 100k+ urls and less than 2k after removing duplicates. For that reason I honestly stopped using it for a while because SB support wasn’t much help. For a literal example “Locksmiths + City name” with 5k different cities for a total of 5k keywords 200k before duplicates 2.6k after removing dupes. :/ I’ve got 10 private proxies from squid proxies. The vps I’m using is pretty slow but would that have an effect on the fact that 95-98% of every harvest I do comes out as duplicates ?

 

[John-S]

Use GSA Captcha Breaker or Xevil to emulate a captcha service.

 

[loopline]

Hey @loopline the biggest issues I seem to be having with scrapebox since I’ve started and never even able to fix is duplicate urls. No matter what keywords i use when I go to harvest urls for contact form submission I’ll have 100k+ urls and less than 2k after removing duplicates. For that reason I honestly stopped using it for a while because SB support wasn’t much help. For a literal example “Locksmiths + City name” with 5k different cities for a total of 5k keywords 200k before duplicates 2.6k after removing dupes. :/ I’ve got 10 private proxies from squid proxies. The vps I’m using is pretty slow but would that have an effect on the fact that 95-98% of every harvest I do comes out as duplicates ?

You could use the yellow pages plugin if you have it.

But you have to bear in mind if lots of your proxies are blocked, a huge percent of your results may be coming from only a handful of keywords.

Also google gives lots of bleed over because they are focusing on their 10 results, after that they throw in everything, including the kitchen sink.

How many locks smiths are there in your cities? you can usually look this up from government data, just a google search. Then you know your scope so you know how much you can get. Maybe there aren't even all that many locks smiths, I have no idea, but first you need to know

what is your scope, what is the potential you are getting

Then look at the report after you harvest, are most cities red with zero results, or are you getting results from like 90% of your keywords?

 

[Djphant0m]

You could use the yellow pages plugin if you have it.

But you have to bear in mind if lots of your proxies are blocked, a huge percent of your results may be coming from only a handful of keywords.

Also google gives lots of bleed over because they are focusing on their 10 results, after that they throw in everything, including the kitchen sink.

How many locks smiths are there in your cities? you can usually look this up from government data, just a google search. Then you know your scope so you know how much you can get. Maybe there aren't even all that many locks smiths, I have no idea, but first you need to know

what is your scope, what is the potential you are getting

Then look at the report after you harvest, are most cities red with zero results, or are you getting results from like 90% of your keywords?

I’ve got a list of locksmiths I scraped from d7 using the same cities (1200 of the cities instead of 5k) and I’ve got well over 100k locksmiths so I know there are plenty in each city.If the proxies are blocked I can just request new ones I’ll give that a try.The vps I’m using is just the free one from Amazon, honestly I’ll probably just switch it over to host windsthis weekend because with the current net speed it takes forever to scrape. Does the yellow pages scraper work for contact for submission? I don’t have it but it’s only like $40 so if it works for contact forms I’ll just get it

 

[romingson]

Where you find the unique keywords ?

 

[loopline]

I’ve got a list of locksmiths I scraped from d7 using the same cities (1200 of the cities instead of 5k) and I’ve got well over 100k locksmiths so I know there are plenty in each city.If the proxies are blocked I can just request new ones I’ll give that a try.The vps I’m using is just the free one from Amazon, honestly I’ll probably just switch it over to host windsthis weekend because with the current net speed it takes forever to scrape. Does the yellow pages scraper work for contact for submission? I don’t have it but it’s only like $40 so if it works for contact forms I’ll just get it

When you get new proxies, first make sure your not going so fast that you get them blocked. If in doubt slow it way down and then you can go up till you get blocked and then you know your sweet spot. Usually in 48 hours or less they are unblocked as well, so your current ones probably will be if you wait, if they are blocked.

That said when you harvest, look at your results after you harvest, even export it and you can look thru it or post it and I can look thru it if you like. Take note if your getting results for most of the keywords or 0 for most of the keywords etc..


Forgot yellow pages part. Yellow pages can scrape the urls, and then you can try and submit to them with the contact form marketing. But the yellow pages plugin will not submit the contact forms, you can use the contact form poster for that.

 

[Djphant0m]

When you get new proxies, first make sure your not going so fast that you get them blocked. If in doubt slow it way down and then you can go up till you get blocked and then you know your sweet spot. Usually in 48 hours or less they are unblocked as well, so your current ones probably will be if you wait, if they are blocked.

That said when you harvest, look at your results after you harvest, even export it and you can look thru it or post it and I can look thru it if you like. Take note if your getting results for most of the keywords or 0 for most of the keywords etc..


Forgot yellow pages part. Yellow pages can scrape the urls, and then you can try and submit to them with the contact form marketing. But the yellow pages plugin will not submit the contact forms, you can use the contact form poster for that.

It takes forever to scrape but here are the results/screenshots of the scrape. The txt file attached here is for 5k keywords only scraping google but not all of the keywords wen through 3960 ended up back in the box. The second textfile was to big to upload so here is a google drive link https://drive.google.com/file/d/17fj4xHclgbTkoKXgImgLW_-R4ONYvW4_/view?usp=sharing for this one i harvested from google,yahoo and bing.

 

[loopline]

It takes forever to scrape but here are the results/screenshots of the scrape. The txt file attached here is for 5k keywords only scraping google but not all of the keywords wen through 3960 ended up back in the box. The second textfile was to big to upload so here is a google drive link https://drive.google.com/file/d/17fj4xHclgbTkoKXgImgLW_-R4ONYvW4_/view?usp=sharing for this one i harvested from google,yahoo and bing.

So if you harvested 5K keywords with no errors but only roughly 1000 completed. So 80% not completing is issue #1.

what country are your proxies from? Test them here
https://www.maxmind.com/en/geoip-demo
Because it sounds like your getting the TOS/privacy type popups with google on some of your proxies. Also I would guess the same for bing since you only got 3 results.

That or your engine file is out of date. But for bing your probably getting redirected to a different country bing that has different html and thats why its not picking up, or the tos/privacy box or both.

The worst case here is you can run httpdebugger pro, it has a 7 day full featured trial and then run scrapebox for a bit, doesn't have to finish, just 100 keywords or 200 or so. Then file >> save session and mail it to me or post it somewhere and both you can I can then see the actual responses scrapebox sees. Else you can use the socket addon tool with your proxies and recreate the queries as well and see what you get back.

Either way, we may need to dig deeper, but first what country are those proxies from?

Also you can try placing each of your proxy in your browser, go to bing for example and then try and search, does it send you to the bing of that country the ip is from or ?

 

[Cartermoro]

for what/.?.

 

[nelida kornilova]

why about the proxy since public proxies affect the result as well

 

[Djphant0m]

So if you harvested 5K keywords with no errors but only roughly 1000 completed. So 80% not completing is issue #1.

what country are your proxies from? Test them here
https://www.maxmind.com/en/geoip-demo
Because it sounds like your getting the TOS/privacy type popups with google on some of your proxies. Also I would guess the same for bing since you only got 3 results.

That or your engine file is out of date. But for bing your probably getting redirected to a different country bing that has different html and thats why its not picking up, or the tos/privacy box or both.

The worst case here is you can run httpdebugger pro, it has a 7 day full featured trial and then run scrapebox for a bit, doesn't have to finish, just 100 keywords or 200 or so. Then file >> save session and mail it to me or post it somewhere and both you can I can then see the actual responses scrapebox sees. Else you can use the socket addon tool with your proxies and recreate the queries as well and see what you get back.

Either way, we may need to dig deeper, but first what country are those proxies from?

Also you can try placing each of your proxy in your browser, go to bing for example and then try and search, does it send you to the bing of that country the ip is from or ?

6 of the proxies are from the USA 4 are from romania but when i use the romanian proxies and go to bin it take me to usa bing. The files were to large to upload here but i did another scrape with debugger pro and put the results in a drive folder here. https://drive.google.com/drive/folders/1znWMMRqA-o8Lo-wUjCS93us8azhsn8Tx?usp=sharing

 

[loopline]

6 of the proxies are from the USA 4 are from romania but when i use the romanian proxies and go to bin it take me to usa bing. The files were to large to upload here but i did another scrape with debugger pro and put the results in a drive folder here. https://drive.google.com/drive/folders/1znWMMRqA-o8Lo-wUjCS93us8azhsn8Tx?usp=sharing

well....

There are no results. Bing says it has no results for your search queries. I went thru over 100 of them and on every one it says

"there are no results for (insert query)"

where the insert query part is your footprint.

Some of them are in a different language, clearly the romanian ones.

Try searching your queries in a browser, do you get results?

see my attached screenshot.

 

[Djphant0m]

well....

There are no results. Bing says it has no results for your search queries. I went thru over 100 of them and on every one it says

"there are no results for (insert query)"

where the insert query part is your footprint.

Some of them are in a different language, clearly the romanian ones.

Try searching your queries in a browser, do you get results?

see my attached screenshot.

If i go to the browser and search bing using the footprints i don't get a result but I do without though, that being said bing isn't working for my scrapebox at all right now. My main issue is still the crazy amount of duplicates. I tried downloading the default engines to fix bing but that didn't work. If it's my proxies I can request new proxies from squidproxies or would it be best to just go with a different provider?

https://drive.google.com/drive/folders/1znWMMRqA-o8Lo-wUjCS93us8azhsn8Tx?usp=sharing(4 of my proxies failed the google test then passed 5 minute later)

 

[loopline]

If i go to the browser and search bing using the footprints i don't get a result but I do without though, that being said bing isn't working for my scrapebox at all right now. My main issue is still the crazy amount of duplicates. I tried downloading the default engines to fix bing but that didn't work. If it's my proxies I can request new proxies from squidproxies or would it be best to just go with a different provider?

https://drive.google.com/drive/folders/1znWMMRqA-o8Lo-wUjCS93us8azhsn8Tx?usp=sharing(4 of my proxies failed the google test then passed 5 minute later)

Its hard to know if its your proxies for sure, sometimes this is a bit of trial and error. If you have proxies you can borrow from another provider or can borrow some from a friend to test that could work.

Bing does give a lot of duplicates in general. Do these footprints work with google?

 

[Djphant0m]

Its hard to know if its your proxies for sure, sometimes this is a bit of trial and error. If you have proxies you can borrow from another provider or can borrow some from a friend to test that could work.

Bing does give a lot of duplicates in general. Do these footprints work with google?

figured id try new proxies, Bought 15 proxies from https://rotatingproxies.com/ still can't seem to scrape google properly or really anything for that matter. Like they'll scrape one ore two keywords for 10-15 results then the rest will just turn up all zeros. disappointing really

 

[loopline]

figured id try new proxies, Bought 15 proxies from https://rotatingproxies.com/ still can't seem to scrape google properly or really anything for that matter. Like they'll scrape one ore two keywords for 10-15 results then the rest will just turn up all zeros. disappointing really

Try the detailed harvester. It will give you endless retries. The thing is that your pulling from a pool of proxies and there is a rotation time limit. So they are rotating every 5 mins. So if you get some scrapes and then run the ips out after 2 mins, your going to have to wait 3 more mins till you get new ips. That assumes the ips were not blocked in the first place, which undoubtedly some are. So you can either slow it down to the point that your staying steady and not getting the ips banned straight away, by lowering connections and possibly using the detailed harvester with a delay, or you can just use the detailed harvester and it will tirelessly retry proxies until it finds one that works. So it will work, but it just may take a while. Let it run while you sleep.

These days google is tight. So you either have to fork over a small fortune for proxies or be smart and patient.