Scrapebox url scrap error 429

X

xfreedom

Newbie
Joined
Oct 25, 2018
Messages
16
Reaction score
1
Hello all,

I wonder if someone can help me with this. Whenever I try to scrape url of google, I only get 100 results before it stops. The logs show me an error 429. Then I have to wait about 8~12 hours before I can try again.
About my configuration, I use 10 private proxies with one thread and max timeout. I try scraping of Google FR and I updated my engine list. Also my keyword contain a "site:" and an "intitle:"
I'm totally clueless on how to fix this so if anyone got an idea, I'm all ears. Thanks !
 
Advertise on BHW
It probably has to do with the proxies (server response 429 = too many requests). Try using other premium proxy providers/use the proxy harvester and generate Google passed proxies (or whatever it was called.. forgot lol..).
Tagging @loopline in case he has some better solution for you.
 
Last edited:
I use private proxies and they pass the google test but I'm still unable to scrape url. For simplest keyword, I also had the issue but a greater number of results like 300, which isn't much. Also for only one keyword.

Anyway, thanks for the answer and tagging @loopline
 
429 is a error Ive started to see popup from google in the past couple months. Its basically an ip block error for all intensive purposes. Google blocks based on all kinds of things and not all blocks are created equal. Even the history of the ips with google comes into play. So what someone did with the ips before you got them affects how you can use them with google.

Basically here is more info



and then also some people just prefer to go this method (which is also what I do)

 
Ok because i've seen lots of your videos, they're of great help but you didn't mention that particular error and you generally speak of scraping different keywords but since I'm scraping only one, I wanted to know if I could try something else before trying rotating proxies like maybe changing the user-agent or things like that.
Anyway, thank you, I'll look into other proxies services to use.
 
proxygo said:
and just think of all G banned proxies scrapebox
has banned over the years lol, probably millions
Click to expand...

xfreedom said:
Ok because i've seen lots of your videos, they're of great help but you didn't mention that particular error and you generally speak of scraping different keywords but since I'm scraping only one, I wanted to know if I could try something else before trying rotating proxies like maybe changing the user-agent or things like that.
Anyway, thank you, I'll look into other proxies services to use.
Click to expand...

429 is only something Ive started seeing the past couple of months really, so its fairly new. Im not saying you have to have back connect proxies, but just the exact proxies you have may have bad history with google and so may be being banned really quick is all.
 
Ok so for the record or if anyone else got the same problem, I tried some rotating proxies and I still had the same error. But I found the problem, since I Was searching through google fr, I changed the fr in the search query but didn't change the marker for the next page...

I planned to try the rotating proxies anyway, and I can see it's better than just dedicated proxies for google scrap after a few tests.


Got another question which is related to the number of results. The difference between the estimation and the actual results is huge, the estimation is around 1k resultats when what I get is 300 results. I tried with the rc=1 parameter (maybe in the wrong way) and estimation is still around 1000 results when actual results are still around 300.
Anyway to improve that ?
 
Last edited:
xfreedom said:
Ok so for the record or if anyone else got the same problem, I tried some rotating proxies and I still had the same error. But I found the problem, since I Was searching through google fr, I changed the fr in the search query but didn't change the marker for the next page...

I planned to try the rotating proxies anyway, and I can see it's better than just dedicated proxies for google scrap after a few tests.


Got another question which is related to the number of results. The difference between the estimation and the actual results is huge, the estimation is around 1k resultats when what I get is 300 results. I tried with the rc=1 parameter (maybe in the wrong way) and estimation is still around 1000 results when actual results are still around 300.
Anyway to improve that ?
Click to expand...
Google often soft caps now between 300 to 600 results. They put so much effort into page 1 that they know that .0001% of people will ever make it past result 300 so they figure if you haven't found it by then your not going to. This is especially true the more advanced of a search query that you use.

Further to be honest googles relevancy can be terrible at result 900 even, like if you are searching for a ski resort in colorado you might get results for lawn care in London UK or ghost hunting equipment (literally). So Id guess they may not even want to show results past 300 anyway, hehe
 
Yeah I saw the video where you were explaining about the revelance of results after 300 hundred results. Still it's a shame since for this particular case, I needed all url from a subsection of a forum with a specific keyword.
I guess 300 will have to do !

Thanks again for the answer.
 
xfreedom said:
Yeah I saw the video where you were explaining about the revelance of results after 300 hundred results. Still it's a shame since for this particular case, I needed all url from a subsection of a forum with a specific keyword.
I guess 300 will have to do !

Thanks again for the answer.
Click to expand...
Just tack on keywords. So lets say your query is

site:domain.com inurl:subsection

just do like

site:domain.com inurl:subsection a
site:domain.com inurl:subsection b
site:domain.com inurl:subsection c
site:domain.com inurl:subsection 1
site:domain.com inurl:subsection 2
site:domain.com inurl:subsection car
site:domain.com inurl:subsection purple
etc..

That forces google to return different sets of results from their database and then just remove duplicates when you are done.

That or if you are working in only 1 domain, just use the grab urls by crawling a site function and let scrapebox crawl the site, just keep connections low like 2 or 3 so you dont' get blocked by the site.


You can then filter out any unwanted urls when its done quick and easy.

Cheers!
 
Actually my query is site:domain.com intitle:keyword but from what you said, I can just add the inurl:subsection which will indeed provide more results.
Thanks loopline !
 
xfreedom said:
Actually my query is site:domain.com intitle:keyword but from what you said, I can just add the inurl:subsection which will indeed provide more results.
Thanks loopline !
Click to expand...
Your welcome, have a great day!
 
Hi all, I know this thread is a bit old, but hopefully by now there's more info on the 429 error. I'm using scrapebox and have been getting the error for a few days now. I am using storm proxies and I also tested it with public proxies and even did a quick and short search with no proxies and I always get the same error. Any ideas?
 
loopline said:
Just tack on keywords. So lets say your query is

site:domain.com inurl:subsection

just do like

site:domain.com inurl:subsection a
site:domain.com inurl:subsection b
site:domain.com inurl:subsection c
site:domain.com inurl:subsection 1
site:domain.com inurl:subsection 2
site:domain.com inurl:subsection car
site:domain.com inurl:subsection purple
etc..

That forces google to return different sets of results from their database and then just remove duplicates when you are done.

That or if you are working in only 1 domain, just use the grab urls by crawling a site function and let scrapebox crawl the site, just keep connections low like 2 or 3 so you dont' get blocked by the site.


You can then filter out any unwanted urls when its done quick and easy.

Cheers!
Click to expand...
Thanks . What will be the case if you use "site:domain.com 'Your Query ' " ? Would such queries still be capped ?
 
Paco Adorno said:
Hi all, I know this thread is a bit old, but hopefully by now there's more info on the 429 error. I'm using scrapebox and have been getting the error for a few days now. I am using storm proxies and I also tested it with public proxies and even did a quick and short search with no proxies and I always get the same error. Any ideas?
Click to expand...

Running into a similar problem @loopline same issue with no proxies, private proxies, rotating proxies etc

Updated Scrapebox engines and installed all updates
 
Paco Adorno said:
Hi all, I know this thread is a bit old, but hopefully by now there's more info on the 429 error. I'm using scrapebox and have been getting the error for a few days now. I am using storm proxies and I also tested it with public proxies and even did a quick and short search with no proxies and I always get the same error. Any ideas?
Click to expand...

429, 503 and 302 from google are all ip bans.

Also google made some updates a few days ago so make sure you update your engines file as scrapebox already compensated for it.

Settings >> harvester engine configuration >> import >> download default engines.

bkenn said:
Thanks . What will be the case if you use "site:domain.com 'Your Query ' " ? Would such queries still be capped ?
Click to expand...

sure, but just keep adding queries. But each query is probably capped at 300 results or less.

710fla said:
Running into a similar problem @loopline same issue with no proxies, private proxies, rotating proxies etc

Updated Scrapebox engines and installed all updates
Click to expand...


429, 503 and 302 from google are all ip bans.

Also google made some updates a few days ago so make sure you update your engines file as scrapebox already compensated for it.

Settings >> harvester engine configuration >> import >> download default engines.

Uvula said:
What is this scrapebox thing?
Click to expand...

Its the swiss army knife of SEO. A must have app for any one doing internet marketing.

https://www.blackhatworld.com/seo/s...auto-blog-commenter-with-prstorm-mode.129096/
 
loopline said:
429, 503 and 302 from google are all ip bans.

Also google made some updates a few days ago so make sure you update your engines file as scrapebox already compensated for it.

Settings >> harvester engine configuration >> import >> download default engines.



sure, but just keep adding queries. But each query is probably capped at 300 results or less.




429, 503 and 302 from google are all ip bans.

Also google made some updates a few days ago so make sure you update your engines file as scrapebox already compensated for it.

Settings >> harvester engine configuration >> import >> download default engines.



Its the swiss army knife of SEO. A must have app for any one doing internet marketing.

https://www.blackhatworld.com/seo/scrapebox-the-ultimate-serp-scraper-auto-blog-commenter-with-prstorm-mode.129096/
Click to expand...
Thanks for the help! It's really appreciated! Would you say that when google made these updates they tightened things up a bit? Like are proxies getting banned quicker than a few weeks ago? in early Jan and in all of 2019 I would typically get 30-60 URL/S (Sometimes even in the hundreds if I had no advanced operators) but now I'm getting 10 on a good day. I'm using stormproxies and have the 80 thread package (20 for scraping). I'm following all of the SB/stormproxies setup guides but it's a lot slower than a few weeks ago. Is this the new normal? Or do you think it might be user error? I've reached out to stormproxies customer support but the issue persists. I appreciate the help!
 
Paco Adorno said:
Thanks for the help! It's really appreciated! Would you say that when google made these updates they tightened things up a bit? Like are proxies getting banned quicker than a few weeks ago? in early Jan and in all of 2019 I would typically get 30-60 URL/S (Sometimes even in the hundreds if I had no advanced operators) but now I'm getting 10 on a good day. I'm using stormproxies and have the 80 thread package (20 for scraping). I'm following all of the SB/stormproxies setup guides but it's a lot slower than a few weeks ago. Is this the new normal? Or do you think it might be user error? I've reached out to stormproxies customer support but the issue persists. I appreciate the help!
Click to expand...
Yes, it seems they have definitely tightened things up when they made these changes. Proxies are indeed getting banned quicker and I "feel" like (although I don't yet have the data to support this) that its especially true on advanced operator queries.

Storm proxies probably has gotten hammered as it snowballs with the updates and their ip pool is getting banned faster and faster. Ive talked to multiple other people with the same issue with storm proxies, but I sent them a video with a debugger log showing whats happening etc...

Today they said they refreshed their proxy pool. I don't know what that means really, but they are making an effort to make it better at any rate. So I would try again with them now and see how it goes.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top