bluebottle
Registered Member
- Jun 1, 2022
- 67
- 12
I want to scrape all the urls for a particular search query but without getting blacklisted or banned by Google, how can I do it?
How to scrape google search results?
- Thread starter bluebottle
- Start date
- Sep 17, 2008
- 6,037
- 3,974
If it is the results for a single keyword just doing it manually is an option. If you have loads to scrape you would like be better off using a tool or service dedicated to this such as Scrapebox or one of the online Google scraping tools.
It's not a hard task to create such an application but there are aspects of it that could make it take far more time than the equivalent cost of paying an API to do it.
It's not a hard task to create such an application but there are aspects of it that could make it take far more time than the equivalent cost of paying an API to do it.
bluebottle
Registered Member
- Jun 1, 2022
- 67
- 12
If I use free rotating proxies?
- Apr 23, 2024
- 2,105
- 4,226
From where?? Are this proxies fast talking about internet speed?
- Sep 17, 2008
- 6,037
- 3,974
Unfortunately that is much more added complexity and time cost. You can do it but the last time i was involved in that was years ago and it was only getting harder and harder to find usable proxies then, I can't imagine this side of it has gotten any easier.
For a frame of reference, I have built many tools that scrape Google (Domain Hunter Gatherer, No Hands SEO, No Hands Proxies, The Free Blog Commenter and more that were for private use) scraping of SERPs is not complicated but there are no good sources of free proxies any more as they are all crawled by so many people, the proxies are wrecked the second they hit those websites. No Hands Proxies, from that list was a tool dedicated to scraping public proxies, it would even search for more sources so even that would scrape google to find pages that were new and far less likely to already be scraped.
I stopped selling No Hands Proxies many years ago (about 7 I think) as it was just not nearly as viable as it once was, I'm not in the business of selling software that doesn't work.
I would say it is going to save you so much work to use a service or software for this
- Oct 5, 2023
- 452
- 171
You've already been given the correct advice about proxies, but I'll add a couple more points. Not only is the "clean IP" and the IP rotation are important, but so is the proxy's geolocation. This is because search results for the same query will differ depending on the search conducted in different countries and locations.
For not super massive scraping, free proxies might work, but you'll have much less headache with reliable paid ones. There are many proxy options available on the BHW marketplace: https://www.blackhatworld.com/forums/proxies-for-sale.112/
For not super massive scraping, free proxies might work, but you'll have much less headache with reliable paid ones. There are many proxy options available on the BHW marketplace: https://www.blackhatworld.com/forums/proxies-for-sale.112/
lucinda54gv
Junior Member
- Sep 12, 2024
- 101
- 18
Instead of scraping, you can use the Google Custom Search API. This is a legal way to get search results without the risk of getting banned.
- Feb 28, 2010
- 5,497
- 4,419
You'll want to use rotating residential proxies which will give you a huge pool of IP's which will pretty much make you unlockable
gamball
BANNED
- May 25, 2024
- 123
- 72
i use my own python script for that
- Feb 14, 2024
- 516
- 127
You only have 3 options:
1. Use Search API Services like SerpAPI, Zenserp, or ScrapingBee/
2. Scraping Directly with Rotating Proxies, but free proxies are not reliable, it's better to use a paid one
Recommendations for this method:
Change the user-agent header with each request to make your requests look like they're coming from different browsers.
Adding random delays between requests to prevent Google from detecting bots (e.g., 1-10 seconds).
Use CAPTCHA Bypass Service like 2Captcha or AntiCaptcha.
Take breaks between requests.
3. Using a python script.
gamball
BANNED
- May 25, 2024
- 123
- 72
i wrote my own script for that, getting around 1000 urls per minute.
- Apr 23, 2012
- 2,853
- 5,869
OP, if you still need this - i wrote a script that will scrape the SERPs and even summarize them using GPT for you
https://www.blackhatworld.com/seo/free-script-google-serp-scraper-gpt-summarizer.1602000/
@blogmen thanks for the shout.
https://www.blackhatworld.com/seo/free-script-google-serp-scraper-gpt-summarizer.1602000/
@blogmen thanks for the shout.
