Scrapebox url scrap error 429

[xfreedom]

Hello all,

I wonder if someone can help me with this. Whenever I try to scrape url of google, I only get 100 results before it stops. The logs show me an error 429. Then I have to wait about 8~12 hours before I can try again.
About my configuration, I use 10 private proxies with one thread and max timeout. I try scraping of Google FR and I updated my engine list. Also my keyword contain a "site:" and an "intitle:"
I'm totally clueless on how to fix this so if anyone got an idea, I'm all ears. Thanks !

 

[Gogol]

It probably has to do with the proxies (server response 429 = too many requests). Try using other premium proxy providers/use the proxy harvester and generate Google passed proxies (or whatever it was called.. forgot lol..).
Tagging @loopline in case he has some better solution for you.

 

[xfreedom]

I use private proxies and they pass the google test but I'm still unable to scrape url. For simplest keyword, I also had the issue but a greater number of results like 300, which isn't much. Also for only one keyword.

Anyway, thanks for the answer and tagging @loopline

 

[loopline]

429 is a error Ive started to see popup from google in the past couple months. Its basically an ip block error for all intensive purposes. Google blocks based on all kinds of things and not all blocks are created equal. Even the history of the ips with google comes into play. So what someone did with the ips before you got them affects how you can use them with google.

Basically here is more info

and then also some people just prefer to go this method (which is also what I do)

 

[proxygo]

and just think of all G banned proxies scrapebox
has banned over the years lol, probably millions

 

[xfreedom]

Ok because i've seen lots of your videos, they're of great help but you didn't mention that particular error and you generally speak of scraping different keywords but since I'm scraping only one, I wanted to know if I could try something else before trying rotating proxies like maybe changing the user-agent or things like that.
Anyway, thank you, I'll look into other proxies services to use.

 

[loopline]

and just think of all G banned proxies scrapebox
has banned over the years lol, probably millions

Ok because i've seen lots of your videos, they're of great help but you didn't mention that particular error and you generally speak of scraping different keywords but since I'm scraping only one, I wanted to know if I could try something else before trying rotating proxies like maybe changing the user-agent or things like that.
Anyway, thank you, I'll look into other proxies services to use.

429 is only something Ive started seeing the past couple of months really, so its fairly new. Im not saying you have to have back connect proxies, but just the exact proxies you have may have bad history with google and so may be being banned really quick is all.

 

[xfreedom]

Ok so for the record or if anyone else got the same problem, I tried some rotating proxies and I still had the same error. But I found the problem, since I Was searching through google fr, I changed the fr in the search query but didn't change the marker for the next page...

I planned to try the rotating proxies anyway, and I can see it's better than just dedicated proxies for google scrap after a few tests.


Got another question which is related to the number of results. The difference between the estimation and the actual results is huge, the estimation is around 1k resultats when what I get is 300 results. I tried with the rc=1 parameter (maybe in the wrong way) and estimation is still around 1000 results when actual results are still around 300.
Anyway to improve that ?

 

[loopline]

Ok so for the record or if anyone else got the same problem, I tried some rotating proxies and I still had the same error. But I found the problem, since I Was searching through google fr, I changed the fr in the search query but didn't change the marker for the next page...

I planned to try the rotating proxies anyway, and I can see it's better than just dedicated proxies for google scrap after a few tests.


Got another question which is related to the number of results. The difference between the estimation and the actual results is huge, the estimation is around 1k resultats when what I get is 300 results. I tried with the rc=1 parameter (maybe in the wrong way) and estimation is still around 1000 results when actual results are still around 300.
Anyway to improve that ?

Google often soft caps now between 300 to 600 results. They put so much effort into page 1 that they know that .0001% of people will ever make it past result 300 so they figure if you haven't found it by then your not going to. This is especially true the more advanced of a search query that you use.

Further to be honest googles relevancy can be terrible at result 900 even, like if you are searching for a ski resort in colorado you might get results for lawn care in London UK or ghost hunting equipment (literally). So Id guess they may not even want to show results past 300 anyway, hehe

 

[xfreedom]

Yeah I saw the video where you were explaining about the revelance of results after 300 hundred results. Still it's a shame since for this particular case, I needed all url from a subsection of a forum with a specific keyword.
I guess 300 will have to do !

Thanks again for the answer.

 

[loopline]

Yeah I saw the video where you were explaining about the revelance of results after 300 hundred results. Still it's a shame since for this particular case, I needed all url from a subsection of a forum with a specific keyword.
I guess 300 will have to do !

Thanks again for the answer.

Just tack on keywords. So lets say your query is

site:domain.com inurl:subsection

just do like

site:domain.com inurl:subsection a
site:domain.com inurl:subsection b
site:domain.com inurl:subsection c
site:domain.com inurl:subsection 1
site:domain.com inurl:subsection 2
site:domain.com inurl:subsection car
site:domain.com inurl:subsection purple
etc..

That forces google to return different sets of results from their database and then just remove duplicates when you are done.

That or if you are working in only 1 domain, just use the grab urls by crawling a site function and let scrapebox crawl the site, just keep connections low like 2 or 3 so you dont' get blocked by the site.

You can then filter out any unwanted urls when its done quick and easy.

Cheers!

 

[xfreedom]

Actually my query is site:domain.com intitle:keyword but from what you said, I can just add the inurl:subsection which will indeed provide more results.
Thanks loopline !

 

[loopline]

Actually my query is site:domain.com intitle:keyword but from what you said, I can just add the inurl:subsection which will indeed provide more results.
Thanks loopline !

Your welcome, have a great day!

 

[Paco Adorno]

Hi all, I know this thread is a bit old, but hopefully by now there's more info on the 429 error. I'm using scrapebox and have been getting the error for a few days now. I am using storm proxies and I also tested it with public proxies and even did a quick and short search with no proxies and I always get the same error. Any ideas?

 

[bkenn]

Just tack on keywords. So lets say your query is

site:domain.com inurl:subsection

just do like

site:domain.com inurl:subsection a
site:domain.com inurl:subsection b
site:domain.com inurl:subsection c
site:domain.com inurl:subsection 1
site:domain.com inurl:subsection 2
site:domain.com inurl:subsection car
site:domain.com inurl:subsection purple
etc..

That forces google to return different sets of results from their database and then just remove duplicates when you are done.

That or if you are working in only 1 domain, just use the grab urls by crawling a site function and let scrapebox crawl the site, just keep connections low like 2 or 3 so you dont' get blocked by the site.

You can then filter out any unwanted urls when its done quick and easy.

Cheers!

Thanks . What will be the case if you use "site:domain.com 'Your Query ' " ? Would such queries still be capped ?

 

[710fla]

Hi all, I know this thread is a bit old, but hopefully by now there's more info on the 429 error. I'm using scrapebox and have been getting the error for a few days now. I am using storm proxies and I also tested it with public proxies and even did a quick and short search with no proxies and I always get the same error. Any ideas?

Running into a similar problem @loopline same issue with no proxies, private proxies, rotating proxies etc

Updated Scrapebox engines and installed all updates

 

[bkenn]

What is this scrapebox thing?

Gscrapers elder Brother !

 

[loopline]

Hi all, I know this thread is a bit old, but hopefully by now there's more info on the 429 error. I'm using scrapebox and have been getting the error for a few days now. I am using storm proxies and I also tested it with public proxies and even did a quick and short search with no proxies and I always get the same error. Any ideas?

429, 503 and 302 from google are all ip bans.

Also google made some updates a few days ago so make sure you update your engines file as scrapebox already compensated for it.

Settings >> harvester engine configuration >> import >> download default engines.

Thanks . What will be the case if you use "site:domain.com 'Your Query ' " ? Would such queries still be capped ?

sure, but just keep adding queries. But each query is probably capped at 300 results or less.

Running into a similar problem @loopline same issue with no proxies, private proxies, rotating proxies etc

Updated Scrapebox engines and installed all updates

429, 503 and 302 from google are all ip bans.

Also google made some updates a few days ago so make sure you update your engines file as scrapebox already compensated for it.

Settings >> harvester engine configuration >> import >> download default engines.

What is this scrapebox thing?

Its the swiss army knife of SEO. A must have app for any one doing internet marketing.

https://www.blackhatworld.com/seo/s...auto-blog-commenter-with-prstorm-mode.129096/

 

[Paco Adorno]

429, 503 and 302 from google are all ip bans.

Also google made some updates a few days ago so make sure you update your engines file as scrapebox already compensated for it.

Settings >> harvester engine configuration >> import >> download default engines.



sure, but just keep adding queries. But each query is probably capped at 300 results or less.




429, 503 and 302 from google are all ip bans.

Also google made some updates a few days ago so make sure you update your engines file as scrapebox already compensated for it.

Settings >> harvester engine configuration >> import >> download default engines.



Its the swiss army knife of SEO. A must have app for any one doing internet marketing.

https://www.blackhatworld.com/seo/scrapebox-the-ultimate-serp-scraper-auto-blog-commenter-with-prstorm-mode.129096/

Thanks for the help! It's really appreciated! Would you say that when google made these updates they tightened things up a bit? Like are proxies getting banned quicker than a few weeks ago? in early Jan and in all of 2019 I would typically get 30-60 URL/S (Sometimes even in the hundreds if I had no advanced operators) but now I'm getting 10 on a good day. I'm using stormproxies and have the 80 thread package (20 for scraping). I'm following all of the SB/stormproxies setup guides but it's a lot slower than a few weeks ago. Is this the new normal? Or do you think it might be user error? I've reached out to stormproxies customer support but the issue persists. I appreciate the help!

 

[loopline]

Thanks for the help! It's really appreciated! Would you say that when google made these updates they tightened things up a bit? Like are proxies getting banned quicker than a few weeks ago? in early Jan and in all of 2019 I would typically get 30-60 URL/S (Sometimes even in the hundreds if I had no advanced operators) but now I'm getting 10 on a good day. I'm using stormproxies and have the 80 thread package (20 for scraping). I'm following all of the SB/stormproxies setup guides but it's a lot slower than a few weeks ago. Is this the new normal? Or do you think it might be user error? I've reached out to stormproxies customer support but the issue persists. I appreciate the help!

Yes, it seems they have definitely tightened things up when they made these changes. Proxies are indeed getting banned quicker and I "feel" like (although I don't yet have the data to support this) that its especially true on advanced operator queries.

Storm proxies probably has gotten hammered as it snowballs with the updates and their ip pool is getting banned faster and faster. Ive talked to multiple other people with the same issue with storm proxies, but I sent them a video with a debugger log showing whats happening etc...

Today they said they refreshed their proxy pool. I don't know what that means really, but they are making an effort to make it better at any rate. So I would try again with them now and see how it goes.