How does someone scrape Google Search?

Yzetien

Yzetien

Junior Member
Joined
Jul 23, 2018
Messages
198
Reaction score
110
Hi, for some reason I've been failing for the past three months to effectively scrape Google search results. Posting this in hope of some advice in the right redirection.

Attempted Scraping methods.
- Scrapebox
- Python Requests+Scrapy/Bs4 (I didn't try Selenium because I am a complete novice in coding)
- Various other tools that are advertised for search engine scraping.
- Attempted to emulate legit browsing headers with it's user-agents respectively across desktop/android and iOS sent in the HTTP requests(Not cookies, as I don't understand them at all especially when it comes to the endless confusion of Google).

The common issue I came across, is that regardless whatever configuration I tried the IPs get blacklisted about 1,000 requests later even if I am using approximately 5threads / 100 IPv4 or 5 thread / 1000IPv6. Each proxy costs are about $1-2/IPv4 address, and $0.05/IPv6. It's not cost effective to constantly purchase new proxies or to burn through Captcha solving solutions as far as I know of the costs associated. Is there any other solutions excluding services such as Luminati packages as I work as an individual and can't really use the amount of traffic required for their minimum spending as I don't have that scale of operational usage. Thanks in advance to anyone that can shred some light to this.
 
Advertise on BHW
Their JavaScript code can detect almost any scraping job. Have you tried their API? Or you need other things?
 
TomTheCat said:
Their JavaScript code can detect almost any scraping job. Have you tried their API? Or you need other things?
Click to expand...

Do you mean Google's CSE? Honestly, I completely forgot about it! Will that require a lot of proxies too, or it'd be less restrictive in the usage?
 
google notices scraping pretty quickly, also a bit iffy results
 
Yzetien said:
Hi, for some reason I've been failing for the past three months to effectively scrape Google search results. Posting this in hope of some advice in the right redirection.

Attempted Scraping methods.
- Scrapebox
- Python Requests+Scrapy/Bs4 (I didn't try Selenium because I am a complete novice in coding)
- Various other tools that are advertised for search engine scraping.
- Attempted to emulate legit browsing headers with it's user-agents respectively across desktop/android and iOS sent in the HTTP requests(Not cookies, as I don't understand them at all especially when it comes to the endless confusion of Google).

The common issue I came across, is that regardless whatever configuration I tried the IPs get blacklisted about 1,000 requests later even if I am using approximately 5threads / 100 IPv4 or 5 thread / 1000IPv6. Each proxy costs are about $1-2/IPv4 address, and $0.05/IPv6. It's not cost effective to constantly purchase new proxies or to burn through Captcha solving solutions as far as I know of the costs associated. Is there any other solutions excluding services such as Luminati packages as I work as an individual and can't really use the amount of traffic required for their minimum spending as I don't have that scale of operational usage. Thanks in advance to anyone that can shred some light to this.
Click to expand...

I have coded one personally my self, google is not detecting the selenium. I have used some bypass methods. But I have give more deal to make sure its not asking for captcha.
 
Yzetien said:
Hi, for some reason I've been failing for the past three months to effectively scrape Google search results. Posting this in hope of some advice in the right redirection.

Attempted Scraping methods.
- Scrapebox
- Python Requests+Scrapy/Bs4 (I didn't try Selenium because I am a complete novice in coding)
- Various other tools that are advertised for search engine scraping.
- Attempted to emulate legit browsing headers with it's user-agents respectively across desktop/android and iOS sent in the HTTP requests(Not cookies, as I don't understand them at all especially when it comes to the endless confusion of Google).

The common issue I came across, is that regardless whatever configuration I tried the IPs get blacklisted about 1,000 requests later even if I am using approximately 5threads / 100 IPv4 or 5 thread / 1000IPv6. Each proxy costs are about $1-2/IPv4 address, and $0.05/IPv6. It's not cost effective to constantly purchase new proxies or to burn through Captcha solving solutions as far as I know of the costs associated. Is there any other solutions excluding services such as Luminati packages as I work as an individual and can't really use the amount of traffic required for their minimum spending as I don't have that scale of operational usage. Thanks in advance to anyone that can shred some light to this.
Click to expand...

Why don't you try rotating proxies than dedicated? If you did j guess you'd never opened a thread like this.

@loopline


May be able to help you
 
I collect the search results with Python and selenium.
I allow considerable time on each scrape so it goes undetected.
 
I have been using Google CSE API for over 10 years, its just like Google Search and the results are in JSON.

Obviously the downside is costs adds up to hundreds and thousands if your search volume gets high, and you can only get first 100 results (10 pages) for any given keyword.
 
Yzetien said:
Hi, for some reason I've been failing for the past three months to effectively scrape Google search results. Posting this in hope of some advice in the right redirection.

Attempted Scraping methods.
- Scrapebox
- Python Requests+Scrapy/Bs4 (I didn't try Selenium because I am a complete novice in coding)
- Various other tools that are advertised for search engine scraping.
- Attempted to emulate legit browsing headers with it's user-agents respectively across desktop/android and iOS sent in the HTTP requests(Not cookies, as I don't understand them at all especially when it comes to the endless confusion of Google).

The common issue I came across, is that regardless whatever configuration I tried the IPs get blacklisted about 1,000 requests later even if I am using approximately 5threads / 100 IPv4 or 5 thread / 1000IPv6. Each proxy costs are about $1-2/IPv4 address, and $0.05/IPv6. It's not cost effective to constantly purchase new proxies or to burn through Captcha solving solutions as far as I know of the costs associated. Is there any other solutions excluding services such as Luminati packages as I work as an individual and can't really use the amount of traffic required for their minimum spending as I don't have that scale of operational usage. Thanks in advance to anyone that can shred some light to this.
Click to expand...
At the end of the day its about speed. So if you go slow enough you won't get ips blocked at all. IPV6 proxies get blocked faster then IPV4, like WAY faster, most generally from what I have seen. I only use IPV4. The method of bot you is less of a concern then is the actual speed of how many requests your making to google in X time from a given ip. So you can slow it absurdly down, like 1 connection per 100 IPv4 proxies plus a delay of 120 seconds. Then dial it back and make it faster and faster till you get blocked, then you know where your sweet spot is for your query type. I said that was absurdly slow, but really these days for google 1 connection per 100 proxies with a 120 delay might actually be right on par.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top