Scraping Google about to get even harder?

Steptoe

Steptoe

Elite Member
Jr. Executive VIP
Jr. VIP
Joined
Aug 9, 2017
Messages
5,624
Reaction score
6,924
Using Google SERPs as a data source has been increasingly difficult over the past few years. First they made socket-based scraping difficult unless you can emulate a Javascript-capable environment, then they did away with the num=100 parameter, increasing our needed scraping requests x10 for the same data, and now they are going back several years to giving us obfuscated urls (google.com/goto?url=kjsgkwkegnwknwuegi) instead of plain links in the SERP html. This means that compared to a few years ago, we now need to use a heavier scraping environment, 10 times as many requests as before (with each request increasing the chance of triggering antibot measures), and now need to resolve the destination of each redirect link with a GET request (HEAD seems to be blocked).

https://www.seroundtable.com/google-search-goto-tracking-41957.html

Again this will have a knock-on effect for all our favourite data-driven apps like Ahrefs.
 
Advertise on BHW
Steptoe said:
Using Google SERPs as a data source has been increasingly difficult over the past few years. First they made socket-based scraping difficult unless you can emulate a Javascript-capable environment, then they did away with the num=100 parameter, increasing our needed scraping requests x10 for the same data, and now they are going back several years to giving us obfuscated urls (google.com/goto?url=kjsgkwkegnwknwuegi) instead of plain links in the SERP html. This means that compared to a few years ago, we now need to use a heavier scraping environment, 10 times as many requests as before (with each request increasing the chance of triggering antibot measures), and now need to resolve the destination of each redirect link with a GET request (HEAD seems to be blocked).

https://www.seroundtable.com/google-search-goto-tracking-41957.html

Again this will have a knock-on effect for all our favourite data-driven apps like Ahrefs.
Click to expand...
Good point. The changes may look small individually, but together they make Google serp scraping much harder.

More requests, Javascript based scraping, and now having to resolve those hidden urls all mean more time, cost, and chances of getting blocked.

I think the bigger concern is how this will affect tools like ahrefs. If getting reliable serp data becomes harder and more expensive, will these tools have to update data less often or increase their prices?

It will be interesting to see how the major Seo tools deal with this.
 
Steptoe said:
Using Google SERPs as a data source has been increasingly difficult over the past few years. First they made socket-based scraping difficult unless you can emulate a Javascript-capable environment, then they did away with the num=100 parameter, increasing our needed scraping requests x10 for the same data, and now they are going back several years to giving us obfuscated urls (google.com/goto?url=kjsgkwkegnwknwuegi) instead of plain links in the SERP html. This means that compared to a few years ago, we now need to use a heavier scraping environment, 10 times as many requests as before (with each request increasing the chance of triggering antibot measures), and now need to resolve the destination of each redirect link with a GET request (HEAD seems to be blocked).

https://www.seroundtable.com/google-search-goto-tracking-41957.html

Again this will have a knock-on effect for all our favourite data-driven apps like Ahrefs.
Click to expand...
That means SEO tools would become more unreliable. Since they made the last major change, Semrush's organic keyword data become late. You don't get as much recent info as possuble.

This would severely impact small tools with little budgets.
 
The num=100 change was already making scraping more difficult and if every result now needs another request it could make things even slower
 
It really does feel like an endless cat and mouse game. Every time the community finds a stable way to pull SERP data, Google goes and changes the rules again.

Losing the 100 results parameter was already a huge hit to efficiency, and now this new redirect tracking just adds unnecessary overhead and slows everything down even more. You are totally right about the ripple effect too. It is bound to drive up infrastructure costs for tools like Ahrefs, and those costs will eventually get passed down to the rest of us.

We will just have to see how the big data providers adapt, but it is definitely getting exhausting trying to keep up with all of it.
 
HenryObi said:
That means SEO tools would become more unreliable. Since they made the last major change, Semrush's organic keyword data become late. You don't get as much recent info as possuble.

This would severely impact small tools with little budgets.
Click to expand...
More unreliable, and even more expensive.
 
ahrefs and semrush and other will now increase pricing of their plans as more resources will need to pull up the results. else all these companies will be out-of-market if google data can't be included.
 
Is this one of the reasons ahref becomes so costly?
 
I wonder if this was always on the table or the loss in the serpapi court case forced its creation.
 
Vynith said:
I wonder if this was always on the table or the loss in the serpapi court case forced its creation.
Click to expand...
I think the SERPAPI case definitely shat in their cornflakes. But several years ago Google SERPs urls were obfuscated too and we adapted, though it looks like there may be javascript shenanigans going on with them now too. TBH my first route when I see people scraping any of my url-based data sources is to obfuscate them.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top